2020-11-20T11:42:47.899 INFO:root:teuthology version: 1.0.0-357fd4a 2020-11-20T11:42:47.901 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi/5640307 --verbose --block --name pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi --owner scheduled_pdonnell@teuthology --lock --description fs/thrash/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/filestore-xfs overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/ffsb}} -- /tmp/teuthology-worker.6dhiipl5.tmp 2020-11-20T11:42:47.939 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-20T11:42:48.007 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi/5640307 branch: wip-pdonnell-testing-20201120.053228 description: fs/thrash/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/filestore-xfs overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/ffsb}} email: pdonnell@redhat.com first_in_suite: false job_id: '5640307' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '18.04' overrides: admin_socket: branch: wip-pdonnell-testing-20201120.053228 ceph: cephfs: session_timeout: 300 conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 15m rados osd op timeout: 15m mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 15m rados osd op timeout: 15m mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 filestore flush min: 0 osd max markdown count: 1000 osd objectstore: filestore osd op complaint time: 180 osd sloppy crc: true fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - Replacing daemon mds - SLOW_OPS - slow request log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: b86e691bf191566922a8f08e2a5abf64fde8cfcb ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: osd objectstore: filestore osd sloppy crc: true filestore: true fs: xfs install: ceph: sha1: b86e691bf191566922a8f08e2a5abf64fde8cfcb workunit: branch: wip-pdonnell-testing-20201120.053228 sha1: b86e691bf191566922a8f08e2a5abf64fde8cfcb owner: scheduled_pdonnell@teuthology priority: 95 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - osd.4 - osd.5 - osd.6 - osd.7 sha1: b86e691bf191566922a8f08e2a5abf64fde8cfcb sleep_before_teardown: 0 suite: fs suite_branch: wip-pdonnell-testing-20201120.053228 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201120.053228/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: b86e691bf191566922a8f08e2a5abf64fde8cfcb tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - ceph-fuse: null - mds_thrash: null - workunit: clients: all: - suites/ffsb.sh teuthology_branch: master timestamp: 2020-11-20_08:39:42 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.21182 2020-11-20T11:42:48.008 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201120.053228/qa; will attempt to use it 2020-11-20T11:42:48.009 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201120.053228/qa/tasks 2020-11-20T11:42:48.009 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-11-20T11:42:48.011 INFO:teuthology.task.internal:Checking packages... 2020-11-20T11:42:48.033 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash 'b86e691bf191566922a8f08e2a5abf64fde8cfcb' 2020-11-20T11:42:48.033 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-11-20T11:42:48.034 INFO:teuthology.packaging:ref: None 2020-11-20T11:42:48.034 INFO:teuthology.packaging:tag: None 2020-11-20T11:42:48.035 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20201120.053228 2020-11-20T11:42:48.035 INFO:teuthology.packaging:sha1: b86e691bf191566922a8f08e2a5abf64fde8cfcb 2020-11-20T11:42:48.035 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-pdonnell-testing-20201120.053228 2020-11-20T11:42:48.202 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-7397-gb86e691b-1bionic 2020-11-20T11:42:48.203 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-11-20T11:42:48.214 INFO:teuthology.task.internal:no buildpackages task found 2020-11-20T11:42:48.214 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-11-20T11:42:48.232 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-11-20T11:42:48.233 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-20T11:42:48.292 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:42:58.309 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:43:08.327 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:43:18.336 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:43:28.372 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:43:38.390 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:43:48.418 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:43:58.443 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:44:08.458 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:44:18.484 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:44:28.502 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:44:38.519 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:44:48.539 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:44:58.559 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:45:08.579 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:45:18.594 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:45:28.610 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:45:38.628 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:45:48.648 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:45:58.673 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:46:08.690 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:46:18.706 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:46:28.730 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:46:38.748 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:46:48.774 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:46:58.795 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:47:08.821 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:47:18.837 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:47:28.858 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:47:38.869 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:47:48.886 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:47:58.902 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:48:08.928 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:48:18.952 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:48:28.968 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:48:38.985 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_pdonnell@teuthology', 'count': 2, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi/5640307'} 2020-11-20T11:48:38.995 ERROR:teuthology.lock.ops:Insufficient nodes available to lock 2 smithi nodes. 2020-11-20T11:48:38.996 ERROR:teuthology.lock.ops:{"message": "only 0 nodes available"} 2020-11-20T11:48:38.996 INFO:teuthology.task.internal.lock_machines:0 smithi machines locked this try, 0/2 locked so far 2020-11-20T11:48:38.996 INFO:teuthology.task.internal.lock_machines:0 machines locked (0 new); need 2 more 2020-11-20T11:48:38.997 WARNING:teuthology.task.internal.lock_machines:Could not lock enough machines, waiting... 2020-11-20T11:48:49.014 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:48:59.029 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:49:09.049 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:49:19.063 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:49:29.088 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:49:39.106 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:49:49.122 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:49:59.145 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:50:09.174 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:50:19.185 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:50:29.233 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:50:39.249 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_pdonnell@teuthology', 'count': 2, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi/5640307'} 2020-11-20T11:50:39.273 DEBUG:teuthology.lock.ops:locked ubuntu@smithi189.front.sepia.ceph.com, ubuntu@smithi047.front.sepia.ceph.com 2020-11-20T11:50:39.331 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f89b1e610d0>, signals=[15]) 2020-11-20T11:50:39.540 INFO:teuthology.provision.fog.smithi047:Scheduling deploy of ubuntu 18.04 2020-11-20T11:50:39.543 INFO:teuthology.provision.fog.smithi189:Scheduling deploy of ubuntu 18.04 2020-11-20T11:50:39.918 INFO:teuthology.orchestra.console:Power off smithi047 2020-11-20T11:50:39.920 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-20T11:50:39.943 INFO:teuthology.orchestra.console:Power off smithi189 2020-11-20T11:50:39.944 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-20T11:50:39.973 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:50:39.989 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:50:44.015 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:50:44.024 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:50:48.126 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:50:48.136 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:50:52.254 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:50:52.280 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:50:52.467 INFO:teuthology.orchestra.console:Power off for smithi189 completed 2020-11-20T11:50:52.497 INFO:teuthology.orchestra.console:Power off for smithi047 completed 2020-11-20T11:50:52.569 INFO:teuthology.orchestra.console:Power on smithi189 2020-11-20T11:50:52.569 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-20T11:50:52.592 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:50:52.603 INFO:teuthology.orchestra.console:Power on smithi047 2020-11-20T11:50:52.604 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-20T11:50:52.626 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:50:56.619 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:50:56.647 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:50:56.830 INFO:teuthology.orchestra.console:Power on for smithi189 completed 2020-11-20T11:50:56.860 INFO:teuthology.orchestra.console:Power on for smithi047 completed 2020-11-20T11:50:56.932 INFO:teuthology.provision.fog.smithi189:Waiting for deploy to finish 2020-11-20T11:50:56.962 INFO:teuthology.provision.fog.smithi047:Waiting for deploy to finish 2020-11-20T11:53:43.529 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:53:53.545 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:53:58.650 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:54:02.598 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:54:09.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:54:11.660 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:54:18.473 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:54:20.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:54:27.531 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:54:36.589 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:54:45.670 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:54:54.726 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:54:58.286 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:54:59.737 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-20T11:55:03.789 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:55:05.745 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:55:05.755 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-20T11:55:11.758 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:55:11.769 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-20T11:55:17.770 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:55:17.781 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-11-20T11:55:23.784 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:55:23.906 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-20T11:55:23.908 INFO:teuthology.orchestra.run.smithi047:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-20T11:55:24.287 INFO:teuthology.provision.fog.smithi047:Node is ready 2020-11-20T11:55:24.390 INFO:teuthology.orchestra.run.smithi047:> hostname 2020-11-20T11:55:24.420 INFO:teuthology.orchestra.run.smithi047.stdout:smithi047.front.sepia.ceph.com 2020-11-20T11:55:24.421 INFO:teuthology.orchestra.run.smithi047:> grep smithi047.front.sepia.ceph.com /etc/hosts 2020-11-20T11:55:24.494 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47 smithi047.front.sepia.ceph.com smithi047 2020-11-20T11:55:24.494 INFO:teuthology.orchestra.run.smithi047:> sudo hostname smithi047 2020-11-20T11:55:24.583 INFO:teuthology.orchestra.run.smithi047:> sudo sed -i -e 's/smithi047.front.sepia.ceph.com/smithi047/g' /etc/hosts 2020-11-20T11:55:24.661 INFO:teuthology.orchestra.run.smithi047:> sudo sed -i -e 's/172.21.15.47/172.21.15.47/g' /etc/hosts 2020-11-20T11:55:24.727 INFO:teuthology.orchestra.run.smithi047:> sudo sed -i -e 's/smithi047.front.sepia.ceph.com/smithi047/g' /etc/hostname 2020-11-20T11:55:24.788 INFO:teuthology.orchestra.run.smithi047:> sudo hostname smithi047 2020-11-20T11:55:24.846 INFO:teuthology.provision.fog.smithi047:Deploy complete! 2020-11-20T11:55:24.847 DEBUG:teuthology.parallel:result is None 2020-11-20T11:56:09.827 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:56:10.011 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-20T11:56:10.012 INFO:teuthology.orchestra.run.smithi189:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-20T11:56:10.382 INFO:teuthology.provision.fog.smithi189:Node is ready 2020-11-20T11:56:10.382 INFO:teuthology.orchestra.run.smithi189:> hostname 2020-11-20T11:56:10.428 INFO:teuthology.orchestra.run.smithi189.stdout:smithi189.front.sepia.ceph.com 2020-11-20T11:56:10.429 INFO:teuthology.orchestra.run.smithi189:> grep smithi189.front.sepia.ceph.com /etc/hosts 2020-11-20T11:56:10.477 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.15.189 smithi189.front.sepia.ceph.com smithi189 2020-11-20T11:56:10.477 INFO:teuthology.orchestra.run.smithi189:> sudo hostname smithi189 2020-11-20T11:56:10.608 INFO:teuthology.orchestra.run.smithi189:> sudo sed -i -e 's/smithi189.front.sepia.ceph.com/smithi189/g' /etc/hosts 2020-11-20T11:56:10.620 INFO:teuthology.orchestra.run.smithi189:> sudo sed -i -e 's/172.21.15.189/172.21.15.189/g' /etc/hosts 2020-11-20T11:56:10.677 INFO:teuthology.orchestra.run.smithi189:> sudo sed -i -e 's/smithi189.front.sepia.ceph.com/smithi189/g' /etc/hostname 2020-11-20T11:56:10.755 INFO:teuthology.orchestra.run.smithi189:> sudo hostname smithi189 2020-11-20T11:56:10.840 INFO:teuthology.provision.fog.smithi189:Deploy complete! 2020-11-20T11:56:10.841 DEBUG:teuthology.parallel:result is None 2020-11-20T11:56:11.238 INFO:teuthology.lock.ops:Checking smithi189.front.sepia.ceph.com 2020-11-20T11:56:11.240 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-20T11:56:11.263 INFO:teuthology.lock.ops:Checking smithi047.front.sepia.ceph.com 2020-11-20T11:56:11.263 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-20T11:56:11.281 INFO:teuthology.lock.ops:Updating [smithi189.front.sepia.ceph.com]: set os type and version on server 2020-11-20T11:56:11.281 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-20T11:56:11.282 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:56:11.414 INFO:teuthology.orchestra.run.smithi189:> true 2020-11-20T11:56:11.661 INFO:teuthology.orchestra.run.smithi189:> uname -m 2020-11-20T11:56:11.721 INFO:teuthology.orchestra.run.smithi189.stdout:x86_64 2020-11-20T11:56:11.722 INFO:teuthology.orchestra.run.smithi189:> cat /etc/os-release 2020-11-20T11:56:11.778 INFO:teuthology.orchestra.run.smithi189.stdout:NAME="Ubuntu" 2020-11-20T11:56:11.779 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-20T11:56:11.779 INFO:teuthology.orchestra.run.smithi189.stdout:ID=ubuntu 2020-11-20T11:56:11.779 INFO:teuthology.orchestra.run.smithi189.stdout:ID_LIKE=debian 2020-11-20T11:56:11.780 INFO:teuthology.orchestra.run.smithi189.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-20T11:56:11.780 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_ID="18.04" 2020-11-20T11:56:11.781 INFO:teuthology.orchestra.run.smithi189.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-20T11:56:11.782 INFO:teuthology.orchestra.run.smithi189.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-20T11:56:11.782 INFO:teuthology.orchestra.run.smithi189.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-20T11:56:11.782 INFO:teuthology.orchestra.run.smithi189.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-20T11:56:11.782 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_CODENAME=bionic 2020-11-20T11:56:11.783 INFO:teuthology.orchestra.run.smithi189.stdout:UBUNTU_CODENAME=bionic 2020-11-20T11:56:11.784 INFO:teuthology.lock.ops:Updating smithi189.front.sepia.ceph.com on lock server 2020-11-20T11:56:11.804 INFO:teuthology.lock.ops:Updating [smithi047.front.sepia.ceph.com]: set os type and version on server 2020-11-20T11:56:11.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-20T11:56:11.805 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:56:11.933 INFO:teuthology.orchestra.run.smithi047:> true 2020-11-20T11:56:12.176 INFO:teuthology.orchestra.run.smithi047:> uname -m 2020-11-20T11:56:12.228 INFO:teuthology.orchestra.run.smithi047.stdout:x86_64 2020-11-20T11:56:12.229 INFO:teuthology.orchestra.run.smithi047:> cat /etc/os-release 2020-11-20T11:56:12.280 INFO:teuthology.orchestra.run.smithi047.stdout:NAME="Ubuntu" 2020-11-20T11:56:12.280 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-20T11:56:12.280 INFO:teuthology.orchestra.run.smithi047.stdout:ID=ubuntu 2020-11-20T11:56:12.280 INFO:teuthology.orchestra.run.smithi047.stdout:ID_LIKE=debian 2020-11-20T11:56:12.281 INFO:teuthology.orchestra.run.smithi047.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-20T11:56:12.281 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION_ID="18.04" 2020-11-20T11:56:12.281 INFO:teuthology.orchestra.run.smithi047.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-20T11:56:12.281 INFO:teuthology.orchestra.run.smithi047.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-20T11:56:12.282 INFO:teuthology.orchestra.run.smithi047.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-20T11:56:12.282 INFO:teuthology.orchestra.run.smithi047.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-20T11:56:12.282 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION_CODENAME=bionic 2020-11-20T11:56:12.282 INFO:teuthology.orchestra.run.smithi047.stdout:UBUNTU_CODENAME=bionic 2020-11-20T11:56:12.283 INFO:teuthology.lock.ops:Updating smithi047.front.sepia.ceph.com on lock server 2020-11-20T11:56:12.307 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-11-20T11:56:12.351 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi047.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFX3BDFEwaObOOpCQyvJwob5VQzCMqTMEbhuS6IulSBMirctkmFxZC7hPAuD0bK7UueGYa4ZrAcOGRq1Ua8S+lQQG8co0lx8/LsrTZmOsnqsSkMn+cG337gLxDLFGzUPAfs6l7ebIdq3ls/l56RVUwEDTkxeDjC6VXLvqoqWypI51W/SUqcXr+cdc/p+o76WLdTa0s6VtxOjUDZYwCI1IIwFd7c2BJbooBfBZd5pEfWiAwHz75DORHbyP02zSagtElOa+oetEXCbzek1BobGpRJG4MDYyhLVYx77xhJgq8+Ox7s4pdMbGAKIiccLRxucgYJHeEaCbqVZWxQVV+xEIp smithi189.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7fiCLM94ZgBmaFMAXibiC0SEL9yXQUymMFlmebFjXuaZfc+uNILlmQpqiaXYFvmv894Yoz9Br8CoB65eJH3F8CCkocG/yO9D0oz/yJysH6rXiWhs7wNxKjzM7a6Vy+io00a5WDEP5+hpAci2C5d6K3TGgn3ldmi+u7FOkBfZATNjoHVIBXbixp9TaPEUCbAmbnm7JNdsISCvhy7aKPY56qeUklZ1CLAzmU3ZOY0HVPosKQduqA5g2z32K5ltuJNtUZWbjod64YjNG1rTKI+wSOR1Fk936drVyMPzN65wxfSTtg+6Rc+dbRX/CjnWsvOAncF751H7mivQ6WfasgAPz 2020-11-20T11:56:12.351 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-20T11:56:12.411 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-11-20T11:56:12.420 INFO:teuthology.task.internal:Saving configuration 2020-11-20T11:56:12.440 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-11-20T11:56:12.866 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-11-20T11:56:12.885 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-20 11:50:39.256195', 'description': '/home/teuthworker/archive/pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi/5640307', 'locked_by': 'scheduled_pdonnell@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIZg7si3Q9tgrQbQvacDclPPJb0ekfmK5iUp2HY4NNyl', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi189.front.sepia.ceph.com'} 2020-11-20T11:56:12.901 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-20 11:50:39.256213', 'description': '/home/teuthworker/archive/pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi/5640307', 'locked_by': 'scheduled_pdonnell@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJBPXrDc7er9WGLL8fV8dzzifm5dvCClqbCiBcJybP2E', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi047.front.sepia.ceph.com'} 2020-11-20T11:56:12.902 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-11-20T11:56:12.980 INFO:teuthology.task.internal:roles: ubuntu@smithi189.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2020-11-20T11:56:12.981 INFO:teuthology.task.internal:roles: ubuntu@smithi047.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2020-11-20T11:56:12.981 INFO:teuthology.run_tasks:Running task console_log... 2020-11-20T11:56:13.171 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f89b59b3048>, signals=[15]) 2020-11-20T11:56:13.171 INFO:teuthology.run_tasks:Running task internal.connect... 2020-11-20T11:56:13.179 INFO:teuthology.task.internal:Opening connections... 2020-11-20T11:56:13.179 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi189.front.sepia.ceph.com 2020-11-20T11:56:13.181 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:56:13.258 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi047.front.sepia.ceph.com 2020-11-20T11:56:13.260 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:56:13.380 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-11-20T11:56:13.391 INFO:teuthology.orchestra.run.smithi189:> uname -m 2020-11-20T11:56:13.455 INFO:teuthology.orchestra.run.smithi189.stdout:x86_64 2020-11-20T11:56:13.456 INFO:teuthology.orchestra.run.smithi189:> cat /etc/os-release 2020-11-20T11:56:13.504 INFO:teuthology.orchestra.run.smithi189.stdout:NAME="Ubuntu" 2020-11-20T11:56:13.504 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-20T11:56:13.504 INFO:teuthology.orchestra.run.smithi189.stdout:ID=ubuntu 2020-11-20T11:56:13.505 INFO:teuthology.orchestra.run.smithi189.stdout:ID_LIKE=debian 2020-11-20T11:56:13.505 INFO:teuthology.orchestra.run.smithi189.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-20T11:56:13.505 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_ID="18.04" 2020-11-20T11:56:13.505 INFO:teuthology.orchestra.run.smithi189.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-20T11:56:13.505 INFO:teuthology.orchestra.run.smithi189.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-20T11:56:13.506 INFO:teuthology.orchestra.run.smithi189.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-20T11:56:13.506 INFO:teuthology.orchestra.run.smithi189.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-20T11:56:13.506 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_CODENAME=bionic 2020-11-20T11:56:13.506 INFO:teuthology.orchestra.run.smithi189.stdout:UBUNTU_CODENAME=bionic 2020-11-20T11:56:13.507 INFO:teuthology.lock.ops:Updating smithi189.front.sepia.ceph.com on lock server 2020-11-20T11:56:13.525 INFO:teuthology.orchestra.run.smithi047:> uname -m 2020-11-20T11:56:13.586 INFO:teuthology.orchestra.run.smithi047.stdout:x86_64 2020-11-20T11:56:13.587 INFO:teuthology.orchestra.run.smithi047:> cat /etc/os-release 2020-11-20T11:56:13.635 INFO:teuthology.orchestra.run.smithi047.stdout:NAME="Ubuntu" 2020-11-20T11:56:13.636 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-11-20T11:56:13.636 INFO:teuthology.orchestra.run.smithi047.stdout:ID=ubuntu 2020-11-20T11:56:13.636 INFO:teuthology.orchestra.run.smithi047.stdout:ID_LIKE=debian 2020-11-20T11:56:13.637 INFO:teuthology.orchestra.run.smithi047.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-11-20T11:56:13.637 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION_ID="18.04" 2020-11-20T11:56:13.637 INFO:teuthology.orchestra.run.smithi047.stdout:HOME_URL="https://www.ubuntu.com/" 2020-11-20T11:56:13.637 INFO:teuthology.orchestra.run.smithi047.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-11-20T11:56:13.638 INFO:teuthology.orchestra.run.smithi047.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-11-20T11:56:13.638 INFO:teuthology.orchestra.run.smithi047.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-11-20T11:56:13.638 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION_CODENAME=bionic 2020-11-20T11:56:13.639 INFO:teuthology.orchestra.run.smithi047.stdout:UBUNTU_CODENAME=bionic 2020-11-20T11:56:13.639 INFO:teuthology.lock.ops:Updating smithi047.front.sepia.ceph.com on lock server 2020-11-20T11:56:13.655 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-11-20T11:56:13.734 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-11-20T11:56:13.742 INFO:teuthology.task.internal:Checking for old test directory... 2020-11-20T11:56:13.743 INFO:teuthology.orchestra.run.smithi047:> test '!' -e /home/ubuntu/cephtest 2020-11-20T11:56:13.746 INFO:teuthology.orchestra.run.smithi189:> test '!' -e /home/ubuntu/cephtest 2020-11-20T11:56:13.752 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-11-20T11:56:13.762 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-11-20T11:56:13.762 INFO:teuthology.orchestra.run.smithi047:> test -z $(ls -A /var/lib/ceph) 2020-11-20T11:56:13.792 INFO:teuthology.orchestra.run.smithi189:> test -z $(ls -A /var/lib/ceph) 2020-11-20T11:56:13.804 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-11-20T11:56:13.853 INFO:teuthology.run_tasks:Running task kernel... 2020-11-20T11:56:13.876 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}, 'mds.a': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'mds.b': {'kdb': True, 'sha1': 'distro'}, 'osd.4': {'kdb': True, 'sha1': 'distro'}, 'osd.5': {'kdb': True, 'sha1': 'distro'}, 'osd.6': {'kdb': True, 'sha1': 'distro'}, 'osd.7': {'kdb': True, 'sha1': 'distro'}} 2020-11-20T11:56:13.877 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-11-20T11:56:13.878 INFO:teuthology.orchestra.run.smithi189:> uname -r 2020-11-20T11:56:13.884 INFO:teuthology.orchestra.run.smithi189.stdout:4.15.0-123-generic 2020-11-20T11:56:13.884 INFO:teuthology.task.kernel:Running kernel on smithi189: 4.15.0-123-generic 2020-11-20T11:56:13.885 INFO:teuthology.orchestra.run.smithi189:> sudo apt-get clean 2020-11-20T11:56:13.988 INFO:teuthology.orchestra.run.smithi189:> sudo apt-get update 2020-11-20T11:56:14.248 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-20T11:56:14.248 INFO:teuthology.orchestra.run.smithi189.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-11-20T11:56:14.313 INFO:teuthology.orchestra.run.smithi189.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-11-20T11:56:14.588 INFO:teuthology.orchestra.run.smithi189.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-11-20T11:56:14.639 INFO:teuthology.orchestra.run.smithi189.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,426 kB] 2020-11-20T11:56:14.769 INFO:teuthology.orchestra.run.smithi189.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,761 kB] 2020-11-20T11:56:15.066 INFO:teuthology.orchestra.run.smithi189.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [861 kB] 2020-11-20T11:56:15.101 INFO:teuthology.orchestra.run.smithi189.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,089 kB] 2020-11-20T11:56:15.161 INFO:teuthology.orchestra.run.smithi189.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,162 kB] 2020-11-20T11:56:15.163 INFO:teuthology.orchestra.run.smithi189.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [967 kB] 2020-11-20T11:56:15.194 INFO:teuthology.orchestra.run.smithi189.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [373 kB] 2020-11-20T11:56:15.199 INFO:teuthology.orchestra.run.smithi189.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [243 kB] 2020-11-20T11:56:15.206 INFO:teuthology.orchestra.run.smithi189.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [17.3 kB] 2020-11-20T11:56:15.206 INFO:teuthology.orchestra.run.smithi189.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [196 kB] 2020-11-20T11:56:15.212 INFO:teuthology.orchestra.run.smithi189.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [26.3 kB] 2020-11-20T11:56:15.213 INFO:teuthology.orchestra.run.smithi189.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,547 kB] 2020-11-20T11:56:15.280 INFO:teuthology.orchestra.run.smithi189.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,692 kB] 2020-11-20T11:56:15.329 INFO:teuthology.orchestra.run.smithi189.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [356 kB] 2020-11-20T11:56:15.333 INFO:teuthology.orchestra.run.smithi189.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [36.1 kB] 2020-11-20T11:56:15.333 INFO:teuthology.orchestra.run.smithi189.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [14.4 kB] 2020-11-20T11:56:15.334 INFO:teuthology.orchestra.run.smithi189.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [7,308 B] 2020-11-20T11:56:16.555 INFO:teuthology.orchestra.run.smithi189.stdout:Fetched 12.0 MB in 2s (5,099 kB/s) 2020-11-20T11:56:17.919 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2020-11-20T11:56:17.967 INFO:teuthology.orchestra.run.smithi189:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-11-20T11:56:18.047 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2020-11-20T11:56:18.298 INFO:teuthology.orchestra.run.smithi189.stdout:Building dependency tree... 2020-11-20T11:56:18.298 INFO:teuthology.orchestra.run.smithi189.stdout:Reading state information... 2020-11-20T11:56:18.380 INFO:teuthology.orchestra.run.smithi189.stderr:E: Unable to locate package linux-image-current-generic 2020-11-20T11:56:18.381 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-11-20T11:56:18.382 INFO:teuthology.orchestra.run.smithi189:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-11-20T11:56:18.505 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2020-11-20T11:56:18.755 INFO:teuthology.orchestra.run.smithi189.stdout:Building dependency tree... 2020-11-20T11:56:18.756 INFO:teuthology.orchestra.run.smithi189.stdout:Reading state information... 2020-11-20T11:56:18.928 INFO:teuthology.orchestra.run.smithi189.stdout:The following additional packages will be installed: 2020-11-20T11:56:18.929 INFO:teuthology.orchestra.run.smithi189.stdout: linux-generic linux-headers-4.15.0-124 linux-headers-4.15.0-124-generic 2020-11-20T11:56:18.929 INFO:teuthology.orchestra.run.smithi189.stdout: linux-headers-generic linux-image-4.15.0-124-generic 2020-11-20T11:56:18.929 INFO:teuthology.orchestra.run.smithi189.stdout: linux-modules-4.15.0-124-generic linux-modules-extra-4.15.0-124-generic 2020-11-20T11:56:18.930 INFO:teuthology.orchestra.run.smithi189.stdout:Suggested packages: 2020-11-20T11:56:18.931 INFO:teuthology.orchestra.run.smithi189.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2020-11-20T11:56:18.931 INFO:teuthology.orchestra.run.smithi189.stdout:Recommended packages: 2020-11-20T11:56:18.931 INFO:teuthology.orchestra.run.smithi189.stdout: thermald 2020-11-20T11:56:18.958 INFO:teuthology.orchestra.run.smithi189.stdout:The following NEW packages will be installed: 2020-11-20T11:56:18.958 INFO:teuthology.orchestra.run.smithi189.stdout: linux-headers-4.15.0-124 linux-headers-4.15.0-124-generic 2020-11-20T11:56:18.959 INFO:teuthology.orchestra.run.smithi189.stdout: linux-image-4.15.0-124-generic linux-modules-4.15.0-124-generic 2020-11-20T11:56:18.959 INFO:teuthology.orchestra.run.smithi189.stdout: linux-modules-extra-4.15.0-124-generic 2020-11-20T11:56:18.961 INFO:teuthology.orchestra.run.smithi189.stdout:The following packages will be upgraded: 2020-11-20T11:56:18.961 INFO:teuthology.orchestra.run.smithi189.stdout: linux-generic linux-headers-generic linux-image-generic 2020-11-20T11:56:19.155 INFO:teuthology.orchestra.run.smithi189.stdout:3 upgraded, 5 newly installed, 0 to remove and 22 not upgraded. 2020-11-20T11:56:19.155 INFO:teuthology.orchestra.run.smithi189.stdout:Need to get 65.9 MB of archives. 2020-11-20T11:56:19.156 INFO:teuthology.orchestra.run.smithi189.stdout:After this operation, 333 MB of additional disk space will be used. 2020-11-20T11:56:19.156 INFO:teuthology.orchestra.run.smithi189.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-124-generic amd64 4.15.0-124.127 [13.1 MB] 2020-11-20T11:56:20.146 INFO:teuthology.orchestra.run.smithi189.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-124-generic amd64 4.15.0-124.127 [8,012 kB] 2020-11-20T11:56:20.384 INFO:teuthology.orchestra.run.smithi189.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-124-generic amd64 4.15.0-124.127 [32.8 MB] 2020-11-20T11:56:21.327 INFO:teuthology.orchestra.run.smithi189.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.124.111 [1,868 B] 2020-11-20T11:56:21.327 INFO:teuthology.orchestra.run.smithi189.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.124.111 [2,624 B] 2020-11-20T11:56:21.328 INFO:teuthology.orchestra.run.smithi189.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-124 all 4.15.0-124.127 [10.9 MB] 2020-11-20T11:56:21.618 INFO:teuthology.orchestra.run.smithi189.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-124-generic amd64 4.15.0-124.127 [1,084 kB] 2020-11-20T11:56:21.666 INFO:teuthology.orchestra.run.smithi189.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.124.111 [2,520 B] 2020-11-20T11:56:22.089 INFO:teuthology.orchestra.run.smithi189.stdout:Fetched 65.9 MB in 3s (24.5 MB/s) 2020-11-20T11:56:22.228 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package linux-modules-4.15.0-124-generic. 2020-11-20T11:56:22.264 INFO:teuthology.orchestra.run.smithi189.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102077 files and directories currently installed.) 2020-11-20T11:56:22.267 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../0-linux-modules-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-20T11:56:22.312 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-modules-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-20T11:56:23.922 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package linux-image-4.15.0-124-generic. 2020-11-20T11:56:23.934 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../1-linux-image-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-20T11:56:24.031 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-image-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-20T11:56:24.566 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-124-generic. 2020-11-20T11:56:24.579 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-20T11:56:24.616 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-modules-extra-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-20T11:56:28.299 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../3-linux-generic_4.15.0.124.111_amd64.deb ... 2020-11-20T11:56:28.420 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-generic (4.15.0.124.111) over (4.15.0.123.110) ... 2020-11-20T11:56:28.735 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.124.111_amd64.deb ... 2020-11-20T11:56:28.841 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-image-generic (4.15.0.124.111) over (4.15.0.123.110) ... 2020-11-20T11:56:29.143 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package linux-headers-4.15.0-124. 2020-11-20T11:56:29.160 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../5-linux-headers-4.15.0-124_4.15.0-124.127_all.deb ... 2020-11-20T11:56:29.201 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-headers-4.15.0-124 (4.15.0-124.127) ... 2020-11-20T11:56:34.526 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package linux-headers-4.15.0-124-generic. 2020-11-20T11:56:34.547 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../6-linux-headers-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-20T11:56:34.584 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-headers-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-20T11:56:37.731 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.124.111_amd64.deb ... 2020-11-20T11:56:37.853 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking linux-headers-generic (4.15.0.124.111) over (4.15.0.123.110) ... 2020-11-20T11:56:38.194 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-modules-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-20T11:56:38.809 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-image-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-20T11:56:39.227 INFO:teuthology.orchestra.run.smithi189.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-124-generic 2020-11-20T11:56:39.227 INFO:teuthology.orchestra.run.smithi189.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-124-generic 2020-11-20T11:56:39.342 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-modules-extra-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-20T11:56:39.802 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-headers-4.15.0-124 (4.15.0-124.127) ... 2020-11-20T11:56:39.928 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-image-generic (4.15.0.124.111) ... 2020-11-20T11:56:40.054 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-headers-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-20T11:56:40.180 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-headers-generic (4.15.0.124.111) ... 2020-11-20T11:56:40.332 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up linux-generic (4.15.0.124.111) ... 2020-11-20T11:56:40.457 INFO:teuthology.orchestra.run.smithi189.stdout:Processing triggers for linux-image-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-20T11:56:40.672 INFO:teuthology.orchestra.run.smithi189.stdout:/etc/kernel/postinst.d/initramfs-tools: 2020-11-20T11:56:40.672 INFO:teuthology.orchestra.run.smithi189.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-124-generic 2020-11-20T11:56:55.731 INFO:teuthology.orchestra.run.smithi189.stdout:/etc/kernel/postinst.d/zz-update-grub: 2020-11-20T11:56:55.732 INFO:teuthology.orchestra.run.smithi189.stdout:Sourcing file `/etc/default/grub' 2020-11-20T11:56:55.760 INFO:teuthology.orchestra.run.smithi189.stdout:Generating grub configuration file ... 2020-11-20T11:56:56.129 INFO:teuthology.orchestra.run.smithi189.stdout:Found linux image: /boot/vmlinuz-4.15.0-124-generic 2020-11-20T11:56:56.157 INFO:teuthology.orchestra.run.smithi189.stdout:Found initrd image: /boot/initrd.img-4.15.0-124-generic 2020-11-20T11:56:56.483 INFO:teuthology.orchestra.run.smithi189.stdout:Found linux image: /boot/vmlinuz-4.15.0-123-generic 2020-11-20T11:56:56.509 INFO:teuthology.orchestra.run.smithi189.stdout:Found initrd image: /boot/initrd.img-4.15.0-123-generic 2020-11-20T11:56:57.955 INFO:teuthology.orchestra.run.smithi189.stdout:done 2020-11-20T11:56:58.326 INFO:teuthology.orchestra.run.smithi189:> dpkg -s linux-image-generic 2020-11-20T11:56:58.349 INFO:teuthology.orchestra.run.smithi189.stdout:Package: linux-image-generic 2020-11-20T11:56:58.349 INFO:teuthology.orchestra.run.smithi189.stdout:Status: install ok installed 2020-11-20T11:56:58.349 INFO:teuthology.orchestra.run.smithi189.stdout:Priority: optional 2020-11-20T11:56:58.349 INFO:teuthology.orchestra.run.smithi189.stdout:Section: kernel 2020-11-20T11:56:58.350 INFO:teuthology.orchestra.run.smithi189.stdout:Installed-Size: 16 2020-11-20T11:56:58.350 INFO:teuthology.orchestra.run.smithi189.stdout:Maintainer: Ubuntu Kernel Team 2020-11-20T11:56:58.350 INFO:teuthology.orchestra.run.smithi189.stdout:Architecture: amd64 2020-11-20T11:56:58.350 INFO:teuthology.orchestra.run.smithi189.stdout:Source: linux-meta 2020-11-20T11:56:58.351 INFO:teuthology.orchestra.run.smithi189.stdout:Version: 4.15.0.124.111 2020-11-20T11:56:58.351 INFO:teuthology.orchestra.run.smithi189.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-11-20T11:56:58.351 INFO:teuthology.orchestra.run.smithi189.stdout:Depends: linux-image-4.15.0-124-generic, linux-modules-extra-4.15.0-124-generic, linux-firmware, intel-microcode, amd64-microcode 2020-11-20T11:56:58.351 INFO:teuthology.orchestra.run.smithi189.stdout:Recommends: thermald 2020-11-20T11:56:58.351 INFO:teuthology.orchestra.run.smithi189.stdout:Description: Generic Linux kernel image 2020-11-20T11:56:58.352 INFO:teuthology.orchestra.run.smithi189.stdout: This package will always depend on the latest generic kernel image 2020-11-20T11:56:58.352 INFO:teuthology.orchestra.run.smithi189.stdout: available. 2020-11-20T11:56:58.352 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-123-generic Expected: 4.15.0-124-generic 2020-11-20T11:56:58.353 INFO:teuthology.orchestra.run.smithi047:> uname -r 2020-11-20T11:56:58.371 INFO:teuthology.orchestra.run.smithi047.stdout:4.15.0-123-generic 2020-11-20T11:56:58.372 INFO:teuthology.task.kernel:Running kernel on smithi047: 4.15.0-123-generic 2020-11-20T11:56:58.373 INFO:teuthology.orchestra.run.smithi047:> sudo apt-get clean 2020-11-20T11:56:58.482 INFO:teuthology.orchestra.run.smithi047:> sudo apt-get update 2020-11-20T11:56:58.618 INFO:teuthology.orchestra.run.smithi047.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-11-20T11:56:58.740 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-20T11:56:58.819 INFO:teuthology.orchestra.run.smithi047.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-11-20T11:56:58.866 INFO:teuthology.orchestra.run.smithi047.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,426 kB] 2020-11-20T11:56:58.970 INFO:teuthology.orchestra.run.smithi047.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [861 kB] 2020-11-20T11:56:58.995 INFO:teuthology.orchestra.run.smithi047.stdout:Get:6 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,089 kB] 2020-11-20T11:56:58.998 INFO:teuthology.orchestra.run.smithi047.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [967 kB] 2020-11-20T11:56:59.013 INFO:teuthology.orchestra.run.smithi047.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [243 kB] 2020-11-20T11:56:59.092 INFO:teuthology.orchestra.run.smithi047.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-11-20T11:56:59.214 INFO:teuthology.orchestra.run.smithi047.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,761 kB] 2020-11-20T11:56:59.597 INFO:teuthology.orchestra.run.smithi047.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,162 kB] 2020-11-20T11:56:59.620 INFO:teuthology.orchestra.run.smithi047.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [373 kB] 2020-11-20T11:56:59.627 INFO:teuthology.orchestra.run.smithi047.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [17.3 kB] 2020-11-20T11:56:59.627 INFO:teuthology.orchestra.run.smithi047.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [196 kB] 2020-11-20T11:56:59.630 INFO:teuthology.orchestra.run.smithi047.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [26.3 kB] 2020-11-20T11:56:59.631 INFO:teuthology.orchestra.run.smithi047.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,692 kB] 2020-11-20T11:56:59.709 INFO:teuthology.orchestra.run.smithi047.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,547 kB] 2020-11-20T11:56:59.768 INFO:teuthology.orchestra.run.smithi047.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [356 kB] 2020-11-20T11:56:59.772 INFO:teuthology.orchestra.run.smithi047.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [14.4 kB] 2020-11-20T11:56:59.772 INFO:teuthology.orchestra.run.smithi047.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [36.1 kB] 2020-11-20T11:56:59.772 INFO:teuthology.orchestra.run.smithi047.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [7,308 B] 2020-11-20T11:57:00.619 INFO:teuthology.orchestra.run.smithi047.stdout:Fetched 12.0 MB in 2s (6,272 kB/s) 2020-11-20T11:57:02.118 INFO:teuthology.orchestra.run.smithi047.stdout:Reading package lists... 2020-11-20T11:57:02.166 INFO:teuthology.orchestra.run.smithi047:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-11-20T11:57:02.252 INFO:teuthology.orchestra.run.smithi047.stdout:Reading package lists... 2020-11-20T11:57:02.516 INFO:teuthology.orchestra.run.smithi047.stdout:Building dependency tree... 2020-11-20T11:57:02.517 INFO:teuthology.orchestra.run.smithi047.stdout:Reading state information... 2020-11-20T11:57:02.602 INFO:teuthology.orchestra.run.smithi047.stderr:E: Unable to locate package linux-image-current-generic 2020-11-20T11:57:02.603 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-11-20T11:57:02.603 INFO:teuthology.orchestra.run.smithi047:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-11-20T11:57:02.684 INFO:teuthology.orchestra.run.smithi047.stdout:Reading package lists... 2020-11-20T11:57:02.947 INFO:teuthology.orchestra.run.smithi047.stdout:Building dependency tree... 2020-11-20T11:57:02.948 INFO:teuthology.orchestra.run.smithi047.stdout:Reading state information... 2020-11-20T11:57:03.132 INFO:teuthology.orchestra.run.smithi047.stdout:The following additional packages will be installed: 2020-11-20T11:57:03.132 INFO:teuthology.orchestra.run.smithi047.stdout: linux-generic linux-headers-4.15.0-124 linux-headers-4.15.0-124-generic 2020-11-20T11:57:03.132 INFO:teuthology.orchestra.run.smithi047.stdout: linux-headers-generic linux-image-4.15.0-124-generic 2020-11-20T11:57:03.133 INFO:teuthology.orchestra.run.smithi047.stdout: linux-modules-4.15.0-124-generic linux-modules-extra-4.15.0-124-generic 2020-11-20T11:57:03.135 INFO:teuthology.orchestra.run.smithi047.stdout:Suggested packages: 2020-11-20T11:57:03.135 INFO:teuthology.orchestra.run.smithi047.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2020-11-20T11:57:03.135 INFO:teuthology.orchestra.run.smithi047.stdout:Recommended packages: 2020-11-20T11:57:03.136 INFO:teuthology.orchestra.run.smithi047.stdout: thermald 2020-11-20T11:57:03.162 INFO:teuthology.orchestra.run.smithi047.stdout:The following NEW packages will be installed: 2020-11-20T11:57:03.163 INFO:teuthology.orchestra.run.smithi047.stdout: linux-headers-4.15.0-124 linux-headers-4.15.0-124-generic 2020-11-20T11:57:03.163 INFO:teuthology.orchestra.run.smithi047.stdout: linux-image-4.15.0-124-generic linux-modules-4.15.0-124-generic 2020-11-20T11:57:03.164 INFO:teuthology.orchestra.run.smithi047.stdout: linux-modules-extra-4.15.0-124-generic 2020-11-20T11:57:03.165 INFO:teuthology.orchestra.run.smithi047.stdout:The following packages will be upgraded: 2020-11-20T11:57:03.165 INFO:teuthology.orchestra.run.smithi047.stdout: linux-generic linux-headers-generic linux-image-generic 2020-11-20T11:57:03.346 INFO:teuthology.orchestra.run.smithi047.stdout:3 upgraded, 5 newly installed, 0 to remove and 22 not upgraded. 2020-11-20T11:57:03.346 INFO:teuthology.orchestra.run.smithi047.stdout:Need to get 65.9 MB of archives. 2020-11-20T11:57:03.346 INFO:teuthology.orchestra.run.smithi047.stdout:After this operation, 333 MB of additional disk space will be used. 2020-11-20T11:57:03.346 INFO:teuthology.orchestra.run.smithi047.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-124-generic amd64 4.15.0-124.127 [13.1 MB] 2020-11-20T11:57:04.349 INFO:teuthology.orchestra.run.smithi047.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-124-generic amd64 4.15.0-124.127 [8,012 kB] 2020-11-20T11:57:04.594 INFO:teuthology.orchestra.run.smithi047.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-124-generic amd64 4.15.0-124.127 [32.8 MB] 2020-11-20T11:57:05.581 INFO:teuthology.orchestra.run.smithi047.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.124.111 [1,868 B] 2020-11-20T11:57:05.581 INFO:teuthology.orchestra.run.smithi047.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.124.111 [2,624 B] 2020-11-20T11:57:05.582 INFO:teuthology.orchestra.run.smithi047.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-124 all 4.15.0-124.127 [10.9 MB] 2020-11-20T11:57:05.881 INFO:teuthology.orchestra.run.smithi047.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-124-generic amd64 4.15.0-124.127 [1,084 kB] 2020-11-20T11:57:05.914 INFO:teuthology.orchestra.run.smithi047.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.124.111 [2,520 B] 2020-11-20T11:57:06.430 INFO:teuthology.orchestra.run.smithi047.stdout:Fetched 65.9 MB in 3s (24.1 MB/s) 2020-11-20T11:57:06.588 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package linux-modules-4.15.0-124-generic. 2020-11-20T11:57:06.626 INFO:teuthology.orchestra.run.smithi047.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102077 files and directories currently installed.) 2020-11-20T11:57:06.629 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../0-linux-modules-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-20T11:57:06.817 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking linux-modules-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-20T11:57:08.853 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package linux-image-4.15.0-124-generic. 2020-11-20T11:57:08.854 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../1-linux-image-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-20T11:57:08.854 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking linux-image-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-20T11:57:09.175 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-124-generic. 2020-11-20T11:57:09.188 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-20T11:57:09.225 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking linux-modules-extra-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-20T11:57:12.985 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../3-linux-generic_4.15.0.124.111_amd64.deb ... 2020-11-20T11:57:13.263 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking linux-generic (4.15.0.124.111) over (4.15.0.123.110) ... 2020-11-20T11:57:13.405 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.124.111_amd64.deb ... 2020-11-20T11:57:13.525 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking linux-image-generic (4.15.0.124.111) over (4.15.0.123.110) ... 2020-11-20T11:57:13.889 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package linux-headers-4.15.0-124. 2020-11-20T11:57:13.898 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../5-linux-headers-4.15.0-124_4.15.0-124.127_all.deb ... 2020-11-20T11:57:13.936 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking linux-headers-4.15.0-124 (4.15.0-124.127) ... 2020-11-20T11:57:18.725 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package linux-headers-4.15.0-124-generic. 2020-11-20T11:57:18.746 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../6-linux-headers-4.15.0-124-generic_4.15.0-124.127_amd64.deb ... 2020-11-20T11:57:18.785 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking linux-headers-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-20T11:57:21.530 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.124.111_amd64.deb ... 2020-11-20T11:57:21.661 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking linux-headers-generic (4.15.0.124.111) over (4.15.0.123.110) ... 2020-11-20T11:57:22.012 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up linux-modules-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-20T11:57:22.598 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up linux-image-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-20T11:57:23.046 INFO:teuthology.orchestra.run.smithi047.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-124-generic 2020-11-20T11:57:23.046 INFO:teuthology.orchestra.run.smithi047.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-124-generic 2020-11-20T11:57:23.217 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up linux-modules-extra-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-20T11:57:23.727 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up linux-headers-4.15.0-124 (4.15.0-124.127) ... 2020-11-20T11:57:23.861 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up linux-image-generic (4.15.0.124.111) ... 2020-11-20T11:57:23.987 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up linux-headers-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-20T11:57:24.106 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up linux-headers-generic (4.15.0.124.111) ... 2020-11-20T11:57:24.206 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up linux-generic (4.15.0.124.111) ... 2020-11-20T11:57:24.342 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for linux-image-4.15.0-124-generic (4.15.0-124.127) ... 2020-11-20T11:57:24.565 INFO:teuthology.orchestra.run.smithi047.stdout:/etc/kernel/postinst.d/initramfs-tools: 2020-11-20T11:57:24.565 INFO:teuthology.orchestra.run.smithi047.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-124-generic 2020-11-20T11:57:39.846 INFO:teuthology.orchestra.run.smithi047.stdout:/etc/kernel/postinst.d/zz-update-grub: 2020-11-20T11:57:39.848 INFO:teuthology.orchestra.run.smithi047.stdout:Sourcing file `/etc/default/grub' 2020-11-20T11:57:39.869 INFO:teuthology.orchestra.run.smithi047.stdout:Generating grub configuration file ... 2020-11-20T11:57:40.197 INFO:teuthology.orchestra.run.smithi047.stdout:Found linux image: /boot/vmlinuz-4.15.0-124-generic 2020-11-20T11:57:40.224 INFO:teuthology.orchestra.run.smithi047.stdout:Found initrd image: /boot/initrd.img-4.15.0-124-generic 2020-11-20T11:57:40.591 INFO:teuthology.orchestra.run.smithi047.stdout:Found linux image: /boot/vmlinuz-4.15.0-123-generic 2020-11-20T11:57:40.624 INFO:teuthology.orchestra.run.smithi047.stdout:Found initrd image: /boot/initrd.img-4.15.0-123-generic 2020-11-20T11:57:41.990 INFO:teuthology.orchestra.run.smithi047.stdout:done 2020-11-20T11:57:42.543 INFO:teuthology.orchestra.run.smithi047:> dpkg -s linux-image-generic 2020-11-20T11:57:42.568 INFO:teuthology.orchestra.run.smithi047.stdout:Package: linux-image-generic 2020-11-20T11:57:42.569 INFO:teuthology.orchestra.run.smithi047.stdout:Status: install ok installed 2020-11-20T11:57:42.569 INFO:teuthology.orchestra.run.smithi047.stdout:Priority: optional 2020-11-20T11:57:42.569 INFO:teuthology.orchestra.run.smithi047.stdout:Section: kernel 2020-11-20T11:57:42.569 INFO:teuthology.orchestra.run.smithi047.stdout:Installed-Size: 16 2020-11-20T11:57:42.570 INFO:teuthology.orchestra.run.smithi047.stdout:Maintainer: Ubuntu Kernel Team 2020-11-20T11:57:42.570 INFO:teuthology.orchestra.run.smithi047.stdout:Architecture: amd64 2020-11-20T11:57:42.570 INFO:teuthology.orchestra.run.smithi047.stdout:Source: linux-meta 2020-11-20T11:57:42.570 INFO:teuthology.orchestra.run.smithi047.stdout:Version: 4.15.0.124.111 2020-11-20T11:57:42.571 INFO:teuthology.orchestra.run.smithi047.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-11-20T11:57:42.571 INFO:teuthology.orchestra.run.smithi047.stdout:Depends: linux-image-4.15.0-124-generic, linux-modules-extra-4.15.0-124-generic, linux-firmware, intel-microcode, amd64-microcode 2020-11-20T11:57:42.571 INFO:teuthology.orchestra.run.smithi047.stdout:Recommends: thermald 2020-11-20T11:57:42.571 INFO:teuthology.orchestra.run.smithi047.stdout:Description: Generic Linux kernel image 2020-11-20T11:57:42.571 INFO:teuthology.orchestra.run.smithi047.stdout: This package will always depend on the latest generic kernel image 2020-11-20T11:57:42.572 INFO:teuthology.orchestra.run.smithi047.stdout: available. 2020-11-20T11:57:42.572 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-123-generic Expected: 4.15.0-124-generic 2020-11-20T11:57:42.572 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2020-11-20T11:57:42.572 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-11-20T11:57:42.574 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-11-20T11:57:42.574 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2020-11-20T11:57:42.574 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi189.front.sepia.ceph.com, path=None, version=distro) 2020-11-20T11:57:42.575 INFO:teuthology.orchestra.run.smithi189:> sudo apt-get clean 2020-11-20T11:57:42.720 INFO:teuthology.orchestra.run.smithi189:> sudo apt-get update 2020-11-20T11:57:42.967 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-11-20T11:57:42.967 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-20T11:57:43.034 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-11-20T11:57:43.112 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-11-20T11:57:44.849 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2020-11-20T11:57:44.896 INFO:teuthology.orchestra.run.smithi189:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-11-20T11:57:45.026 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2020-11-20T11:57:45.287 INFO:teuthology.orchestra.run.smithi189.stdout:Building dependency tree... 2020-11-20T11:57:45.287 INFO:teuthology.orchestra.run.smithi189.stdout:Reading state information... 2020-11-20T11:57:45.369 INFO:teuthology.orchestra.run.smithi189.stderr:E: Unable to locate package linux-image-current-generic 2020-11-20T11:57:45.371 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-11-20T11:57:45.372 INFO:teuthology.orchestra.run.smithi189:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-11-20T11:57:45.461 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2020-11-20T11:57:45.714 INFO:teuthology.orchestra.run.smithi189.stdout:Building dependency tree... 2020-11-20T11:57:45.716 INFO:teuthology.orchestra.run.smithi189.stdout:Reading state information... 2020-11-20T11:57:45.910 INFO:teuthology.orchestra.run.smithi189.stdout:linux-image-generic is already the newest version (4.15.0.124.111). 2020-11-20T11:57:45.911 INFO:teuthology.orchestra.run.smithi189.stdout:0 upgraded, 0 newly installed, 0 to remove and 22 not upgraded. 2020-11-20T11:57:45.911 INFO:teuthology.orchestra.run.smithi189:> dpkg -s linux-image-generic 2020-11-20T11:57:45.941 INFO:teuthology.orchestra.run.smithi189.stdout:Package: linux-image-generic 2020-11-20T11:57:45.942 INFO:teuthology.orchestra.run.smithi189.stdout:Status: install ok installed 2020-11-20T11:57:45.942 INFO:teuthology.orchestra.run.smithi189.stdout:Priority: optional 2020-11-20T11:57:45.942 INFO:teuthology.orchestra.run.smithi189.stdout:Section: kernel 2020-11-20T11:57:45.942 INFO:teuthology.orchestra.run.smithi189.stdout:Installed-Size: 16 2020-11-20T11:57:45.942 INFO:teuthology.orchestra.run.smithi189.stdout:Maintainer: Ubuntu Kernel Team 2020-11-20T11:57:45.943 INFO:teuthology.orchestra.run.smithi189.stdout:Architecture: amd64 2020-11-20T11:57:45.943 INFO:teuthology.orchestra.run.smithi189.stdout:Source: linux-meta 2020-11-20T11:57:45.943 INFO:teuthology.orchestra.run.smithi189.stdout:Version: 4.15.0.124.111 2020-11-20T11:57:45.944 INFO:teuthology.orchestra.run.smithi189.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-11-20T11:57:45.944 INFO:teuthology.orchestra.run.smithi189.stdout:Depends: linux-image-4.15.0-124-generic, linux-modules-extra-4.15.0-124-generic, linux-firmware, intel-microcode, amd64-microcode 2020-11-20T11:57:45.944 INFO:teuthology.orchestra.run.smithi189.stdout:Recommends: thermald 2020-11-20T11:57:45.944 INFO:teuthology.orchestra.run.smithi189.stdout:Description: Generic Linux kernel image 2020-11-20T11:57:45.945 INFO:teuthology.orchestra.run.smithi189.stdout: This package will always depend on the latest generic kernel image 2020-11-20T11:57:45.945 INFO:teuthology.orchestra.run.smithi189.stdout: available. 2020-11-20T11:57:45.945 INFO:teuthology.orchestra.run.smithi189:> mktemp 2020-11-20T11:57:45.995 INFO:teuthology.orchestra.run.smithi189.stdout:/tmp/tmp.ruTcLIHVc8 2020-11-20T11:57:45.996 INFO:teuthology.orchestra.run.smithi189:> sudo cp /boot/grub/grub.cfg /tmp/tmp.ruTcLIHVc8 2020-11-20T11:57:46.016 INFO:teuthology.orchestra.run.smithi189:> sudo chmod 0666 /tmp/tmp.ruTcLIHVc8 2020-11-20T11:57:46.147 DEBUG:teuthology.orchestra.remote:smithi189:/tmp/tmp.ruTcLIHVc8 is 8KB 2020-11-20T11:57:46.172 INFO:teuthology.orchestra.run.smithi189:> rm -fr /tmp/tmp.ruTcLIHVc8 2020-11-20T11:57:46.184 INFO:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2020-11-20T11:57:46.243 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-20T11:57:46.244 INFO:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/grub.d/01_ceph_kernel 2020-11-20T11:57:46.244 INFO:teuthology.orchestra.run.smithi189:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-11-20T11:57:46.299 INFO:teuthology.orchestra.run.smithi189.stderr:+ sudo dd of=/etc/grub.d/01_ceph_kernel 2020-11-20T11:57:46.347 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2020-11-20T11:57:46.348 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2020-11-20T11:57:46.348 INFO:teuthology.orchestra.run.smithi189.stderr:93 bytes copied, 0.0317807 s, 2.9 kB/s 2020-11-20T11:57:46.348 INFO:teuthology.orchestra.run.smithi189.stderr:+ sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-11-20T11:57:46.360 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-124-generic 2020-11-20T11:57:46.362 INFO:teuthology.orchestra.run.smithi189:> sudo update-grub 2020-11-20T11:57:47.588 INFO:teuthology.orchestra.run.smithi189.stderr:Sourcing file `/etc/default/grub' 2020-11-20T11:57:47.609 INFO:teuthology.orchestra.run.smithi189.stderr:Generating grub configuration file ... 2020-11-20T11:57:47.843 INFO:teuthology.orchestra.run.smithi189.stderr:Found linux image: /boot/vmlinuz-4.15.0-124-generic 2020-11-20T11:57:47.855 INFO:teuthology.orchestra.run.smithi189.stderr:Found initrd image: /boot/initrd.img-4.15.0-124-generic 2020-11-20T11:57:48.166 INFO:teuthology.orchestra.run.smithi189.stderr:Found linux image: /boot/vmlinuz-4.15.0-123-generic 2020-11-20T11:57:48.179 INFO:teuthology.orchestra.run.smithi189.stderr:Found initrd image: /boot/initrd.img-4.15.0-123-generic 2020-11-20T11:57:49.361 INFO:teuthology.orchestra.run.smithi189.stderr:done 2020-11-20T11:57:49.362 INFO:teuthology.orchestra.run.smithi189:> sudo shutdown -r now 2020-11-20T11:57:49.371 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2020-11-20T11:57:49.371 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi047.front.sepia.ceph.com, path=None, version=distro) 2020-11-20T11:57:49.372 INFO:teuthology.orchestra.run.smithi047:> sudo apt-get clean 2020-11-20T11:57:49.424 INFO:teuthology.orchestra.run.smithi047:> sudo apt-get update 2020-11-20T11:57:49.600 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-11-20T11:57:49.716 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-20T11:57:49.794 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-11-20T11:57:49.876 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-11-20T11:57:51.508 INFO:teuthology.orchestra.run.smithi047.stdout:Reading package lists... 2020-11-20T11:57:51.555 INFO:teuthology.orchestra.run.smithi047:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-11-20T11:57:51.639 INFO:teuthology.orchestra.run.smithi047.stdout:Reading package lists... 2020-11-20T11:57:51.908 INFO:teuthology.orchestra.run.smithi047.stdout:Building dependency tree... 2020-11-20T11:57:51.908 INFO:teuthology.orchestra.run.smithi047.stdout:Reading state information... 2020-11-20T11:57:51.997 INFO:teuthology.orchestra.run.smithi047.stderr:E: Unable to locate package linux-image-current-generic 2020-11-20T11:57:52.002 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-11-20T11:57:52.002 INFO:teuthology.orchestra.run.smithi047:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-11-20T11:57:52.093 INFO:teuthology.orchestra.run.smithi047.stdout:Reading package lists... 2020-11-20T11:57:52.333 INFO:teuthology.orchestra.run.smithi047.stdout:Building dependency tree... 2020-11-20T11:57:52.334 INFO:teuthology.orchestra.run.smithi047.stdout:Reading state information... 2020-11-20T11:57:52.540 INFO:teuthology.orchestra.run.smithi047.stdout:linux-image-generic is already the newest version (4.15.0.124.111). 2020-11-20T11:57:52.541 INFO:teuthology.orchestra.run.smithi047.stdout:0 upgraded, 0 newly installed, 0 to remove and 22 not upgraded. 2020-11-20T11:57:52.543 INFO:teuthology.orchestra.run.smithi047:> dpkg -s linux-image-generic 2020-11-20T11:57:52.577 INFO:teuthology.orchestra.run.smithi047.stdout:Package: linux-image-generic 2020-11-20T11:57:52.578 INFO:teuthology.orchestra.run.smithi047.stdout:Status: install ok installed 2020-11-20T11:57:52.578 INFO:teuthology.orchestra.run.smithi047.stdout:Priority: optional 2020-11-20T11:57:52.578 INFO:teuthology.orchestra.run.smithi047.stdout:Section: kernel 2020-11-20T11:57:52.579 INFO:teuthology.orchestra.run.smithi047.stdout:Installed-Size: 16 2020-11-20T11:57:52.579 INFO:teuthology.orchestra.run.smithi047.stdout:Maintainer: Ubuntu Kernel Team 2020-11-20T11:57:52.579 INFO:teuthology.orchestra.run.smithi047.stdout:Architecture: amd64 2020-11-20T11:57:52.579 INFO:teuthology.orchestra.run.smithi047.stdout:Source: linux-meta 2020-11-20T11:57:52.579 INFO:teuthology.orchestra.run.smithi047.stdout:Version: 4.15.0.124.111 2020-11-20T11:57:52.580 INFO:teuthology.orchestra.run.smithi047.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-11-20T11:57:52.580 INFO:teuthology.orchestra.run.smithi047.stdout:Depends: linux-image-4.15.0-124-generic, linux-modules-extra-4.15.0-124-generic, linux-firmware, intel-microcode, amd64-microcode 2020-11-20T11:57:52.580 INFO:teuthology.orchestra.run.smithi047.stdout:Recommends: thermald 2020-11-20T11:57:52.580 INFO:teuthology.orchestra.run.smithi047.stdout:Description: Generic Linux kernel image 2020-11-20T11:57:52.581 INFO:teuthology.orchestra.run.smithi047.stdout: This package will always depend on the latest generic kernel image 2020-11-20T11:57:52.581 INFO:teuthology.orchestra.run.smithi047.stdout: available. 2020-11-20T11:57:52.581 INFO:teuthology.orchestra.run.smithi047:> mktemp 2020-11-20T11:57:52.626 INFO:teuthology.orchestra.run.smithi047.stdout:/tmp/tmp.JIf2UVx3NE 2020-11-20T11:57:52.627 INFO:teuthology.orchestra.run.smithi047:> sudo cp /boot/grub/grub.cfg /tmp/tmp.JIf2UVx3NE 2020-11-20T11:57:52.698 INFO:teuthology.orchestra.run.smithi047:> sudo chmod 0666 /tmp/tmp.JIf2UVx3NE 2020-11-20T11:57:52.819 DEBUG:teuthology.orchestra.remote:smithi047:/tmp/tmp.JIf2UVx3NE is 8KB 2020-11-20T11:57:52.885 INFO:teuthology.orchestra.run.smithi047:> rm -fr /tmp/tmp.JIf2UVx3NE 2020-11-20T11:57:52.894 INFO:teuthology.orchestra.run.smithi047:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2020-11-20T11:57:52.954 INFO:teuthology.orchestra.run.smithi047:> set -ex 2020-11-20T11:57:52.954 INFO:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/grub.d/01_ceph_kernel 2020-11-20T11:57:52.954 INFO:teuthology.orchestra.run.smithi047:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-11-20T11:57:53.002 INFO:teuthology.orchestra.run.smithi047.stderr:+ sudo dd of=/etc/grub.d/01_ceph_kernel 2020-11-20T11:57:53.043 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records in 2020-11-20T11:57:53.044 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records out 2020-11-20T11:57:53.044 INFO:teuthology.orchestra.run.smithi047.stderr:93 bytes copied, 0.0260938 s, 3.6 kB/s 2020-11-20T11:57:53.045 INFO:teuthology.orchestra.run.smithi047.stderr:+ sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-11-20T11:57:53.057 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-124-generic 2020-11-20T11:57:53.057 INFO:teuthology.orchestra.run.smithi047:> sudo update-grub 2020-11-20T11:57:54.472 INFO:teuthology.orchestra.run.smithi047.stderr:Sourcing file `/etc/default/grub' 2020-11-20T11:57:54.482 INFO:teuthology.orchestra.run.smithi047.stderr:Generating grub configuration file ... 2020-11-20T11:57:54.706 INFO:teuthology.orchestra.run.smithi047.stderr:Found linux image: /boot/vmlinuz-4.15.0-124-generic 2020-11-20T11:57:54.718 INFO:teuthology.orchestra.run.smithi047.stderr:Found initrd image: /boot/initrd.img-4.15.0-124-generic 2020-11-20T11:57:55.035 INFO:teuthology.orchestra.run.smithi047.stderr:Found linux image: /boot/vmlinuz-4.15.0-123-generic 2020-11-20T11:57:55.048 INFO:teuthology.orchestra.run.smithi047.stderr:Found initrd image: /boot/initrd.img-4.15.0-123-generic 2020-11-20T11:57:56.194 INFO:teuthology.orchestra.run.smithi047.stderr:done 2020-11-20T11:57:56.195 INFO:teuthology.orchestra.run.smithi047:> sudo shutdown -r now 2020-11-20T11:58:26.273 INFO:teuthology.misc:Re-opening connections... 2020-11-20T11:58:26.273 INFO:teuthology.misc:trying to connect to ubuntu@smithi189.front.sepia.ceph.com 2020-11-20T11:58:26.274 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-20T11:58:26.274 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:59:26.276 DEBUG:teuthology.orchestra.remote:timed out 2020-11-20T11:59:27.836 INFO:teuthology.misc:trying to connect to ubuntu@smithi047.front.sepia.ceph.com 2020-11-20T11:59:27.837 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-20T11:59:27.838 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:59:27.919 INFO:teuthology.orchestra.run.smithi047:> true 2020-11-20T11:59:28.326 DEBUG:teuthology.misc:waited 62.05268383026123 2020-11-20T11:59:29.327 INFO:teuthology.misc:trying to connect to ubuntu@smithi189.front.sepia.ceph.com 2020-11-20T11:59:29.328 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-20T11:59:29.328 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:59:44.711 INFO:teuthology.orchestra.run.smithi189:> true 2020-11-20T11:59:45.070 DEBUG:teuthology.misc:waited 78.79662466049194 2020-11-20T11:59:46.071 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2020-11-20T11:59:46.072 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.15.0-124-generic"... 2020-11-20T11:59:46.072 INFO:teuthology.orchestra.run.smithi189:> uname -r 2020-11-20T11:59:46.079 INFO:teuthology.orchestra.run.smithi189.stdout:4.15.0-124-generic 2020-11-20T11:59:46.079 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-124-generic vs 4.15.0-124-generic 2020-11-20T11:59:46.079 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-11-20T11:59:46.080 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2020-11-20T11:59:46.080 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2020-11-20T11:59:46.080 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.15.0-124-generic"... 2020-11-20T11:59:46.080 INFO:teuthology.orchestra.run.smithi047:> uname -r 2020-11-20T11:59:46.087 INFO:teuthology.orchestra.run.smithi047.stdout:4.15.0-124-generic 2020-11-20T11:59:46.087 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-124-generic vs 4.15.0-124-generic 2020-11-20T11:59:46.088 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-11-20T11:59:46.088 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2020-11-20T11:59:47.088 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2020-11-20T11:59:47.089 INFO:teuthology.orchestra.run.smithi189:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-11-20T11:59:47.116 INFO:teuthology.orchestra.run.smithi189.stdout:ttyS1 2020-11-20T11:59:47.138 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2020-11-20T11:59:47.138 INFO:teuthology.orchestra.run.smithi047:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-11-20T11:59:47.165 INFO:teuthology.orchestra.run.smithi047.stdout:ttyS1 2020-11-20T11:59:47.194 INFO:teuthology.run_tasks:Running task internal.base... 2020-11-20T11:59:47.205 INFO:teuthology.task.internal:Creating test directory... 2020-11-20T11:59:47.206 INFO:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-11-20T11:59:47.237 INFO:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-11-20T11:59:47.244 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-11-20T11:59:47.259 INFO:teuthology.run_tasks:Running task internal.archive... 2020-11-20T11:59:47.276 INFO:teuthology.task.internal:Creating archive directory... 2020-11-20T11:59:47.276 INFO:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-11-20T11:59:47.285 INFO:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-11-20T11:59:47.300 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-11-20T11:59:47.311 INFO:teuthology.task.internal:Enabling coredump saving... 2020-11-20T11:59:47.311 INFO:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-11-20T11:59:47.341 INFO:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-11-20T11:59:47.363 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-20T11:59:47.369 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-20T11:59:47.370 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-20T11:59:47.376 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-20T11:59:47.379 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-11-20T11:59:47.390 INFO:teuthology.task.internal:Configuring sudo... 2020-11-20T11:59:47.391 INFO:teuthology.orchestra.run.smithi047:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-11-20T11:59:47.419 INFO:teuthology.orchestra.run.smithi189:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-11-20T11:59:47.435 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-11-20T11:59:47.455 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-11-20T11:59:47.456 INFO:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-11-20T11:59:47.473 INFO:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-11-20T11:59:47.481 INFO:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-20T11:59:47.530 INFO:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-20T11:59:47.578 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-20T11:59:47.579 INFO:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-20T11:59:47.624 INFO:teuthology.orchestra.run.smithi189.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-20T11:59:47.638 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2020-11-20T11:59:47.639 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2020-11-20T11:59:47.639 INFO:teuthology.orchestra.run.smithi189.stderr:152 bytes copied, 3.5613e-05 s, 4.3 MB/s 2020-11-20T11:59:47.640 INFO:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-20T11:59:47.647 INFO:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-20T11:59:47.694 INFO:teuthology.orchestra.run.smithi047:> set -ex 2020-11-20T11:59:47.695 INFO:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-20T11:59:47.739 INFO:teuthology.orchestra.run.smithi047.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-20T11:59:47.761 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records in 2020-11-20T11:59:47.761 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records out 2020-11-20T11:59:47.761 INFO:teuthology.orchestra.run.smithi047.stderr:152 bytes copied, 3.4378e-05 s, 4.4 MB/s 2020-11-20T11:59:47.762 INFO:teuthology.orchestra.run.smithi047:> sudo service rsyslog restart 2020-11-20T11:59:47.805 INFO:teuthology.orchestra.run.smithi189:> sudo service rsyslog restart 2020-11-20T11:59:47.851 INFO:teuthology.run_tasks:Running task internal.timer... 2020-11-20T11:59:47.868 INFO:teuthology.task.internal:Starting timer... 2020-11-20T11:59:47.869 INFO:teuthology.run_tasks:Running task pcp... 2020-11-20T11:59:47.892 INFO:teuthology.run_tasks:Running task selinux... 2020-11-20T11:59:47.910 DEBUG:teuthology.task.selinux:Excluding smithi189: OS 'ubuntu' does not support SELinux 2020-11-20T11:59:47.911 DEBUG:teuthology.task.selinux:Excluding smithi047: OS 'ubuntu' does not support SELinux 2020-11-20T11:59:47.911 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-11-20T11:59:47.911 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2020-11-20T11:59:47.911 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-11-20T11:59:47.912 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-11-20T11:59:47.933 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated; assuming it is current 2020-11-20T11:59:47.935 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-11-20T11:59:47.947 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-11-20T11:59:47.949 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi189.front.sepia.ceph.com,smithi047.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-11-20T11:59:48.300 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-11-20T11:59:49.285 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-11-20T11:59:51.895 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-11-20T11:59:51.915 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:51 +0000 (0:00:00.070) 0:00:00.070 ******* 2020-11-20T11:59:51.928 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:51 +0000 (0:00:00.012) 0:00:00.083 ******* 2020-11-20T11:59:53.607 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2020-11-20T11:59:53.960 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] Friday 20 November 2020 11:59:53 +0000 (0:00:01.766) 0:00:01.849 ******* ok: [smithi189.front.sepia.ceph.com] Friday 20 November 2020 11:59:53 +0000 (0:00:00.172) 0:00:02.021 ******* 2020-11-20T11:59:53.971 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-11-20T11:59:54.011 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:54 +0000 (0:00:00.144) 0:00:02.166 ******* 2020-11-20T11:59:54.019 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-11-20T11:59:54.060 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:54 +0000 (0:00:00.049) 0:00:02.215 ******* 2020-11-20T11:59:54.466 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-11-20T11:59:54.468 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-11-20T11:59:54.503 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:54 +0000 (0:00:00.442) 0:00:02.657 ******* 2020-11-20T11:59:54.510 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-11-20T11:59:54.549 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:54 +0000 (0:00:00.046) 0:00:02.704 ******* 2020-11-20T11:59:54.831 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-11-20T11:59:54.833 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.078394", "end": "2020-11-20 11:59:54.813122", "rc": 0, "start": "2020-11-20 11:59:54.734728", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-11-20T11:59:54.868 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:54 +0000 (0:00:00.318) 0:00:03.022 ******* 2020-11-20T11:59:54.875 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.095821", "end": "2020-11-20 11:59:54.681455", "rc": 0, "start": "2020-11-20 11:59:54.585634", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-11-20T11:59:54.914 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:54 +0000 (0:00:00.046) 0:00:03.068 ******* 2020-11-20T11:59:55.123 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [smithi189.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-20T11:59:55.124 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-20T11:59:55.158 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:55 +0000 (0:00:00.244) 0:00:03.313 ******* 2020-11-20T11:59:55.193 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:55 +0000 (0:00:00.032) 0:00:03.345 ******* 2020-11-20T11:59:55.922 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-11-20T11:59:55.924 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1605873595.171314-8940916213913/source", "state": "file", "uid": 0} 2020-11-20T11:59:55.962 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:55 +0000 (0:00:00.770) 0:00:04.116 ******* 2020-11-20T11:59:55.970 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1605873595.2087204-92051386341445/source", "state": "file", "uid": 0} 2020-11-20T11:59:56.008 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:56 +0000 (0:00:00.045) 0:00:04.162 ******* 2020-11-20T11:59:56.256 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-11-20T11:59:56.259 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-11-20T11:59:56.261 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail 2020-11-20T11:59:56.263 INFO:teuthology.task.ansible.out:.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@ 2020-11-20T11:59:56.264 INFO:teuthology.task.ansible.out:aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-11-20T11:59:56.403 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-11-20T11:59:56.411 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-11-20T11:59:56.557 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:56 +0000 (0:00:00.549) 0:00:04.711 ******* 2020-11-20T11:59:56.589 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:56 +0000 (0:00:00.032) 0:00:04.744 ******* 2020-11-20T11:59:57.317 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi047.front.sepia.ceph.com] 2020-11-20T11:59:57.339 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] 2020-11-20T11:59:57.446 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-11-20T11:59:57.448 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:57 +0000 (0:00:00.857) 0:00:05.601 ******* 2020-11-20T11:59:57.486 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:57 +0000 (0:00:00.039) 0:00:05.641 ******* 2020-11-20T11:59:57.811 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** changed: [smithi047.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.175640", "end": "2020-11-20 11:59:57.788524", "rc": 0, "start": "2020-11-20 11:59:57.612884", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-20T11:59:57.852 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:57 +0000 (0:00:00.364) 0:00:06.005 ******* 2020-11-20T11:59:57.868 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.160363", "end": "2020-11-20 11:59:57.834375", "rc": 0, "start": "2020-11-20 11:59:57.674012", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-20T11:59:57.912 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:57 +0000 (0:00:00.061) 0:00:06.067 ******* 2020-11-20T11:59:57.993 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** changed: [smithi047.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.013233", "end": "2020-11-20 11:59:57.817710", "failed_when_result": false, "rc": 0, "start": "2020-11-20 11:59:57.804477", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-20T11:59:58.037 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:58 +0000 (0:00:00.123) 0:00:06.190 ******* 2020-11-20T11:59:58.067 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-11-20T11:59:58.069 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-11-20T11:59:58.133 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:58 +0000 (0:00:00.087) 0:00:06.277 ******* 2020-11-20T11:59:58.153 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.207 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:58 +0000 (0:00:00.082) 0:00:06.360 ******* 2020-11-20T11:59:58.242 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.278 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:58 +0000 (0:00:00.073) 0:00:06.433 ******* 2020-11-20T11:59:58.307 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.350 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:58 +0000 (0:00:00.071) 0:00:06.505 ******* 2020-11-20T11:59:58.358 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.272260", "end": "2020-11-20 11:59:58.287360", "failed_when_result": false, "rc": 0, "start": "2020-11-20 11:59:58.015100", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-20T11:59:58.414 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:58 +0000 (0:00:00.064) 0:00:06.569 ******* 2020-11-20T11:59:58.424 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-11-20T11:59:58.427 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.474 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:58 +0000 (0:00:00.050) 0:00:06.620 ******* TASK [users : Determine if we can skip users and pubkeys updates] ************** ok: [smithi189.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-11-20T11:59:58.512 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:58 +0000 (0:00:00.047) 0:00:06.667 ******* 2020-11-20T11:59:58.519 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-11-20T11:59:58.521 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.558 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:58 +0000 (0:00:00.045) 0:00:06.712 ******* 2020-11-20T11:59:58.565 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.602 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:58 +0000 (0:00:00.043) 0:00:06.756 ******* 2020-11-20T11:59:58.620 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.632 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.634 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} TASK [users : Remove managed_admin_users from managed_users] ******************* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.682 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:58 +0000 (0:00:00.080) 0:00:06.837 ******* 2020-11-20T11:59:58.691 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.692 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.693 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.697 INFO:teuthology.task.ansible.out: skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.699 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.701 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f30 2020-11-20T11:59:58.705 INFO:teuthology.task.ansible.out:09adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.712 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVN 2020-11-20T11:59:58.715 INFO:teuthology.task.ansible.out:TUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} TASK [users : Filter the managed_users list] *********************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru 2020-11-20T11:59:58.718 INFO:teuthology.task.ansible.out:8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.770 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:58 +0000 (0:00:00.086) 0:00:06.923 ******* 2020-11-20T11:59:58.779 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.780 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.782 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.784 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.784 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d8 2020-11-20T11:59:58.787 INFO:teuthology.task.ansible.out:9bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.787 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ov 2020-11-20T11:59:58.789 INFO:teuthology.task.ansible.out:pn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.805 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} TASK [users : Filter the managed_admin_users list] ***************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.855 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:58 +0000 (0:00:00.086) 0:00:07.009 ******* 2020-11-20T11:59:58.862 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.865 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4d 2020-11-20T11:59:58.871 INFO:teuthology.task.ansible.out:c798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.879 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.890 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-11-20T11:59:58.892 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.934 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:58 +0000 (0:00:00.078) 0:00:07.088 ******* 2020-11-20T11:59:58.943 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "v 2020-11-20T11:59:58.945 INFO:teuthology.task.ansible.out:asu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.959 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.971 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.982 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.992 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.994 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:58.998 INFO:teuthology.task.ansible.out: skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.009 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8 2020-11-20T11:59:59.011 INFO:teuthology.task.ansible.out:ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.020 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.021 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.026 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.028 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.049 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.049 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.050 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.051 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.052 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.053 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.054 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.059 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.062 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.064 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.080 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.086 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.088 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.099 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.104 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ub 2020-11-20T11:59:59.106 INFO:teuthology.task.ansible.out:untu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.121 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.125 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.135 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.150 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinste 2020-11-20T11:59:59.152 INFO:teuthology.task.ansible.out:in@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f 2020-11-20T11:59:59.156 INFO:teuthology.task.ansible.out:730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504 2020-11-20T11:59:59.164 INFO:teuthology.task.ansible.out:df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.171 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.176 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "c 2020-11-20T11:59:59.181 INFO:teuthology.task.ansible.out:hanged": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.195 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.202 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.204 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.216 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.219 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika 2020-11-20T11:59:59.222 INFO:teuthology.task.ansible.out:@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.240 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.250 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkur 2020-11-20T11:59:59.255 INFO:teuthology.task.ansible.out:xfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.263 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.283 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'ofrie 2020-11-20T11:59:59.285 INFO:teuthology.task.ansible.out:dma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.289 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.291 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.302 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.317 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh 2020-11-20T11:59:59.322 INFO:teuthology.task.ansible.out:", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.328 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.330 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.349 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.359 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.374 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rE 2020-11-20T11:59:59.380 INFO:teuthology.task.ansible.out:hQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.390 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner 2020-11-20T11:59:59.392 INFO:teuthology.task.ansible.out:@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.406 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.419 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.434 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.447 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.449 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.461 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": " 2020-11-20T11:59:59.464 INFO:teuthology.task.ansible.out:admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.475 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.486 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.505 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84c 2020-11-20T11:59:59.510 INFO:teuthology.task.ansible.out:a78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.518 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.521 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.532 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.551 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d 2020-11-20T11:59:59.554 INFO:teuthology.task.ansible.out:4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.565 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.578 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.581 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.591 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.593 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.603 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.605 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.625 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.628 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.666 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:59 +0000 (0:00:00.731) 0:00:07.820 ******* 2020-11-20T11:59:59.672 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.674 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.686 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.688 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} TASK [users : Create all users without sudo access.] *************************** 2020-11-20T11:59:59.728 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:59 +0000 (0:00:00.063) 0:00:07.883 ******* 2020-11-20T11:59:59.736 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-20T11:59:59.737 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.738 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.739 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.751 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.763 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.766 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.811 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:59 +0000 (0:00:00.082) 0:00:07.966 ******* 2020-11-20T11:59:59.821 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.824 INFO:teuthology.task.ansible.out: skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.825 INFO:teuthology.task.ansible.out: skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.827 INFO:teuthology.task.ansible.out: skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.832 INFO:teuthology.task.ansible.out: skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.841 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.855 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.899 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:59 +0000 (0:00:00.087) 0:00:08.054 ******* 2020-11-20T11:59:59.902 INFO:teuthology.task.ansible.out: 2020-11-20T11:59:59.907 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.909 INFO:teuthology.task.ansible.out: skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.911 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.913 INFO:teuthology.task.ansible.out: skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.915 INFO:teuthology.task.ansible.out: skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.928 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.946 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-11-20T11:59:59.949 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:59:59.989 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:59:59 +0000 (0:00:00.086) 0:00:08.140 ******* 2020-11-20T12:00:00.029 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:00 +0000 (0:00:00.042) 0:00:08.183 ******* 2020-11-20T12:00:00.034 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-11-20T12:00:00.036 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.073 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:00 +0000 (0:00:00.044) 0:00:08.228 ******* 2020-11-20T12:00:00.082 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-11-20T12:00:00.119 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:00 +0000 (0:00:00.044) 0:00:08.272 ******* 2020-11-20T12:00:00.125 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.163 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:00 +0000 (0:00:00.044) 0:00:08.317 ******* 2020-11-20T12:00:00.169 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-11-20T12:00:00.172 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.225 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:00 +0000 (0:00:00.057) 0:00:08.374 ******* 2020-11-20T12:00:00.228 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** skipping: [smithi047.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.250 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-11-20T12:00:00.251 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.296 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:00 +0000 (0:00:00.076) 0:00:08.451 ******* 2020-11-20T12:00:00.303 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-20T12:00:00.304 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.307 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.317 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.320 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.323 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.323 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.335 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.341 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi047.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.385 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:00 +0000 (0:00:00.088) 0:00:08.540 ******* 2020-11-20T12:00:00.396 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.411 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.424 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi047.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.463 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:00 +0000 (0:00:00.073) 0:00:08.613 ******* 2020-11-20T12:00:00.467 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.470 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.510 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:00 +0000 (0:00:00.050) 0:00:08.663 ******* 2020-11-20T12:00:00.516 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.551 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:00 +0000 (0:00:00.042) 0:00:08.706 ******* 2020-11-20T12:00:00.593 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** skipping: [smithi189.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.623 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.626 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.647 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.654 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.658 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.660 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.677 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.685 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.697 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.698 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.709 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.710 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.725 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.734 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.764 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.771 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.782 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.795 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.823 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.834 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.846 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.847 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.861 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.873 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.883 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.885 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.894 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.896 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.906 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.916 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:00.960 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:00 +0000 (0:00:00.408) 0:00:09.115 ******* 2020-11-20T12:00:01.141 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "checksum": "eefb312a9cf82af5b9a46fb88fa640babe5fe20f", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "size": 40, "state": "file", "uid": 0} 2020-11-20T12:00:01.183 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:01 +0000 (0:00:00.222) 0:00:09.338 ******* 2020-11-20T12:00:01.210 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-11-20T12:00:01.213 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => { "msg": "Host smithi047.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-11-20T12:00:01.252 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:01 +0000 (0:00:00.068) 0:00:09.406 ******* 2020-11-20T12:00:01.279 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-11-20T12:00:01.281 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "eefb312a9cf82af5b9a46fb88fa640babe5fe20f", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "size": 40, "state": "file", "uid": 0} 2020-11-20T12:00:01.325 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:01 +0000 (0:00:00.073) 0:00:09.480 ******* 2020-11-20T12:00:01.333 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** ok: [smithi047.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-11-20T12:00:01.373 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:01 +0000 (0:00:00.047) 0:00:09.528 ******* 2020-11-20T12:00:01.379 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-11-20T12:00:01.381 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => { "msg": "Host smithi189.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-11-20T12:00:01.419 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:01 +0000 (0:00:00.045) 0:00:09.573 ******* 2020-11-20T12:00:01.425 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-11-20T12:00:01.428 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:01.465 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:01 +0000 (0:00:00.046) 0:00:09.620 ******* 2020-11-20T12:00:01.472 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-11-20T12:00:01.473 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-11-20T12:00:01.517 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:01 +0000 (0:00:00.051) 0:00:09.672 ******* 2020-11-20T12:00:01.523 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-11-20T12:00:01.524 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:01.580 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:01 +0000 (0:00:00.063) 0:00:09.735 ******* 2020-11-20T12:00:01.682 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:01 +0000 (0:00:00.077) 0:00:09.812 ******* TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:01.710 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:01 +0000 (0:00:00.052) 0:00:09.865 ******* 2020-11-20T12:00:01.717 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:01.761 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:01 +0000 (0:00:00.049) 0:00:09.914 ******* 2020-11-20T12:00:01.770 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-11-20T12:00:01.772 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:01.821 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:01 +0000 (0:00:00.061) 0:00:09.976 ******* 2020-11-20T12:00:01.828 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-11-20T12:00:01.835 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:01.867 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:01 +0000 (0:00:00.045) 0:00:10.021 ******* 2020-11-20T12:00:01.873 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-11-20T12:00:01.875 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:01.925 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:01 +0000 (0:00:00.058) 0:00:10.079 ******* 2020-11-20T12:00:01.933 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-11-20T12:00:01.935 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:01.992 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:01 +0000 (0:00:00.067) 0:00:10.147 ******* 2020-11-20T12:00:02.002 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-11-20T12:00:02.004 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:02.052 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:02 +0000 (0:00:00.059) 0:00:10.207 ******* 2020-11-20T12:00:02.060 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-11-20T12:00:02.062 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:02.103 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:02 +0000 (0:00:00.051) 0:00:10.258 ******* 2020-11-20T12:00:02.109 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-11-20T12:00:02.110 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:02.169 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:02 +0000 (0:00:00.066) 0:00:10.324 ******* 2020-11-20T12:00:02.177 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-11-20T12:00:02.178 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:02.225 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:02 +0000 (0:00:00.055) 0:00:10.379 ******* 2020-11-20T12:00:02.234 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-11-20T12:00:02.236 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:02.275 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:02 +0000 (0:00:00.050) 0:00:10.430 ******* 2020-11-20T12:00:02.282 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:02.319 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:02 +0000 (0:00:00.043) 0:00:10.474 ******* 2020-11-20T12:00:02.329 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** skipping: [smithi047.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi047.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T12:00:02.366 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:02 +0000 (0:00:00.046) 0:00:10.521 ******* 2020-11-20T12:00:02.375 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:02.419 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:02 +0000 (0:00:00.051) 0:00:10.572 ******* 2020-11-20T12:00:02.424 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-11-20T12:00:02.427 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:02.480 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:02 +0000 (0:00:00.063) 0:00:10.635 ******* 2020-11-20T12:00:02.493 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** skipping: [smithi189.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi189.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T12:00:02.529 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:02 +0000 (0:00:00.049) 0:00:10.684 ******* 2020-11-20T12:00:02.537 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-11-20T12:00:02.539 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:02.586 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:02 +0000 (0:00:00.055) 0:00:10.740 ******* 2020-11-20T12:00:02.620 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-11-20T12:00:02.622 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Determine if node is registered with subscription-manager.] ***** skipping: [smithi047.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T12:00:02.673 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:02 +0000 (0:00:00.086) 0:00:10.826 ******* 2020-11-20T12:00:02.726 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:02 +0000 (0:00:00.054) 0:00:10.881 ******* 2020-11-20T12:00:02.738 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:02.797 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:02 +0000 (0:00:00.070) 0:00:10.951 ******* 2020-11-20T12:00:02.809 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:02.885 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:02 +0000 (0:00:00.083) 0:00:11.034 ******* 2020-11-20T12:00:02.887 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi047.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T12:00:02.946 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:02 +0000 (0:00:00.066) 0:00:11.101 ******* 2020-11-20T12:00:02.955 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-11-20T12:00:02.957 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T12:00:03.001 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:02 +0000 (0:00:00.055) 0:00:11.156 ******* 2020-11-20T12:00:03.009 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-11-20T12:00:03.011 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:03.071 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:03 +0000 (0:00:00.069) 0:00:11.225 ******* 2020-11-20T12:00:03.077 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-11-20T12:00:03.079 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:03.114 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:03 +0000 (0:00:00.043) 0:00:11.269 ******* 2020-11-20T12:00:03.121 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-11-20T12:00:03.123 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:03.164 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:03 +0000 (0:00:00.045) 0:00:11.315 ******* 2020-11-20T12:00:03.171 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi189.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T12:00:03.209 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:03 +0000 (0:00:00.047) 0:00:11.362 ******* 2020-11-20T12:00:03.215 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-11-20T12:00:03.217 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:03.254 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:03 +0000 (0:00:00.045) 0:00:11.407 ******* 2020-11-20T12:00:03.260 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-11-20T12:00:03.263 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:03.301 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:03 +0000 (0:00:00.048) 0:00:11.456 ******* 2020-11-20T12:00:03.311 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:03.348 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:03 +0000 (0:00:00.046) 0:00:11.503 ******* 2020-11-20T12:00:03.356 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:03.392 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:03 +0000 (0:00:00.043) 0:00:11.546 ******* 2020-11-20T12:00:03.397 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-11-20T12:00:03.399 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T12:00:03.438 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:03 +0000 (0:00:00.046) 0:00:11.593 ******* 2020-11-20T12:00:03.444 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-11-20T12:00:03.446 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:03.485 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:03 +0000 (0:00:00.046) 0:00:11.640 ******* 2020-11-20T12:00:03.491 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-11-20T12:00:03.493 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:03.528 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:03 +0000 (0:00:00.043) 0:00:11.683 ******* 2020-11-20T12:00:03.540 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:03.573 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:03 +0000 (0:00:00.045) 0:00:11.728 ******* 2020-11-20T12:00:03.580 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-11-20T12:00:03.582 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:03.618 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:03 +0000 (0:00:00.044) 0:00:11.773 ******* 2020-11-20T12:00:03.624 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-11-20T12:00:03.626 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T12:00:03.662 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:03 +0000 (0:00:00.044) 0:00:11.817 ******* 2020-11-20T12:00:03.669 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:03.708 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:03 +0000 (0:00:00.045) 0:00:11.863 ******* 2020-11-20T12:00:03.715 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-11-20T12:00:03.716 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:03.754 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:03 +0000 (0:00:00.046) 0:00:11.909 ******* 2020-11-20T12:00:03.761 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-11-20T12:00:03.763 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:03.799 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:03 +0000 (0:00:00.045) 0:00:11.954 ******* 2020-11-20T12:00:03.806 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-11-20T12:00:03.809 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:03.848 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:03 +0000 (0:00:00.048) 0:00:12.003 ******* 2020-11-20T12:00:03.861 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:03.898 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:03 +0000 (0:00:00.049) 0:00:12.053 ******* 2020-11-20T12:00:03.913 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:03.948 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:03 +0000 (0:00:00.049) 0:00:12.103 ******* 2020-11-20T12:00:03.954 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-11-20T12:00:03.956 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:04.008 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:03 +0000 (0:00:00.054) 0:00:12.157 ******* 2020-11-20T12:00:04.015 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:04.073 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:04 +0000 (0:00:00.070) 0:00:12.227 ******* 2020-11-20T12:00:04.082 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-11-20T12:00:04.082 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T12:00:04.138 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:04 +0000 (0:00:00.065) 0:00:12.292 ******* 2020-11-20T12:00:04.145 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:04.203 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:04 +0000 (0:00:00.065) 0:00:12.358 ******* 2020-11-20T12:00:04.211 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:04.259 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:04 +0000 (0:00:00.055) 0:00:12.414 ******* 2020-11-20T12:00:04.281 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:04.336 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:04 +0000 (0:00:00.077) 0:00:12.491 ******* 2020-11-20T12:00:04.350 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:04.396 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:04 +0000 (0:00:00.058) 0:00:12.550 ******* 2020-11-20T12:00:04.402 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi189.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T12:00:04.466 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:04 +0000 (0:00:00.070) 0:00:12.621 ******* 2020-11-20T12:00:04.473 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-11-20T12:00:04.475 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:04.511 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:04 +0000 (0:00:00.044) 0:00:12.665 ******* 2020-11-20T12:00:04.517 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-11-20T12:00:04.519 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:04.569 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:04 +0000 (0:00:00.057) 0:00:12.723 ******* 2020-11-20T12:00:04.576 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-11-20T12:00:04.581 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:04.626 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:04 +0000 (0:00:00.057) 0:00:12.781 ******* 2020-11-20T12:00:04.641 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:04.688 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:04 +0000 (0:00:00.061) 0:00:12.843 ******* 2020-11-20T12:00:04.705 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* skipping: [smithi047.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:04.746 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:04 +0000 (0:00:00.057) 0:00:12.900 ******* 2020-11-20T12:00:04.753 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-11-20T12:00:04.756 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:04.791 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:04 +0000 (0:00:00.043) 0:00:12.944 ******* 2020-11-20T12:00:04.798 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:04.841 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:04 +0000 (0:00:00.047) 0:00:12.992 ******* 2020-11-20T12:00:04.856 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-11-20T12:00:04.857 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:04.895 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:04 +0000 (0:00:00.057) 0:00:13.050 ******* 2020-11-20T12:00:04.948 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* skipping: [smithi189.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:05.011 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:05 +0000 (0:00:00.116) 0:00:13.166 ******* 2020-11-20T12:00:05.021 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-20T12:00:05.023 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.013394", "end": "2020-11-20 12:00:04.826177", "rc": 0, "start": "2020-11-20 12:00:04.812783", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-11-20T12:00:05.086 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:05 +0000 (0:00:00.074) 0:00:13.240 ******* 2020-11-20T12:00:05.100 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:05.157 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:05 +0000 (0:00:00.069) 0:00:13.310 ******* 2020-11-20T12:00:05.168 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:05.220 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:05 +0000 (0:00:00.063) 0:00:13.373 ******* 2020-11-20T12:00:05.261 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:05.303 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:05 +0000 (0:00:00.083) 0:00:13.457 ******* 2020-11-20T12:00:05.331 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:05.375 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:05 +0000 (0:00:00.072) 0:00:13.530 ******* 2020-11-20T12:00:05.402 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:05.454 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:05 +0000 (0:00:00.078) 0:00:13.608 ******* 2020-11-20T12:00:05.464 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-20T12:00:05.468 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.135177", "end": "2020-11-20 12:00:05.413488", "rc": 0, "start": "2020-11-20 12:00:05.278311", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-11-20T12:00:05.509 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:05 +0000 (0:00:00.055) 0:00:13.663 ******* 2020-11-20T12:00:05.515 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:05.561 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:05 +0000 (0:00:00.048) 0:00:13.711 ******* 2020-11-20T12:00:05.565 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:05.630 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:05 +0000 (0:00:00.073) 0:00:13.785 ******* 2020-11-20T12:00:05.637 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:05.690 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:05 +0000 (0:00:00.059) 0:00:13.845 ******* 2020-11-20T12:00:05.696 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-11-20T12:00:05.699 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:05.741 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:05 +0000 (0:00:00.049) 0:00:13.894 ******* 2020-11-20T12:00:05.749 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:05.784 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:05 +0000 (0:00:00.044) 0:00:13.939 ******* 2020-11-20T12:00:05.796 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:05.837 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:05 +0000 (0:00:00.052) 0:00:13.991 ******* 2020-11-20T12:00:05.881 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:05.919 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:05 +0000 (0:00:00.081) 0:00:14.073 ******* 2020-11-20T12:00:05.946 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-11-20T12:00:05.949 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:05.990 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:05 +0000 (0:00:00.071) 0:00:14.145 ******* 2020-11-20T12:00:06.017 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-11-20T12:00:06.019 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:06.055 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:06 +0000 (0:00:00.065) 0:00:14.210 ******* 2020-11-20T12:00:06.082 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-11-20T12:00:06.084 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:06.137 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:06 +0000 (0:00:00.081) 0:00:14.291 ******* 2020-11-20T12:00:06.520 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-11-20T12:00:06.522 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.264426", "end": "2020-11-20 12:00:06.500318", "rc": 0, "start": "2020-11-20 12:00:06.235892", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:00:06.557 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:06 +0000 (0:00:00.420) 0:00:14.712 ******* 2020-11-20T12:00:06.584 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-11-20T12:00:06.586 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:06.620 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:06 +0000 (0:00:00.063) 0:00:14.775 ******* 2020-11-20T12:00:06.682 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-11-20T12:00:06.684 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.763954", "end": "2020-11-20 12:00:06.500468", "rc": 0, "start": "2020-11-20 12:00:05.736514", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:00:06.744 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:06 +0000 (0:00:00.123) 0:00:14.899 ******* 2020-11-20T12:00:06.773 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-11-20T12:00:06.775 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:06.818 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:06 +0000 (0:00:00.073) 0:00:14.973 ******* 2020-11-20T12:00:06.826 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-11-20T12:00:06.828 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.025330", "end": "2020-11-20 12:00:06.768769", "rc": 0, "start": "2020-11-20 12:00:06.743439", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:00:06.879 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:06 +0000 (0:00:00.054) 0:00:15.027 ******* 2020-11-20T12:00:07.008 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.040693", "end": "2020-11-20 12:00:06.828220", "rc": 0, "start": "2020-11-20 12:00:06.787527", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:00:07.052 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:07 +0000 (0:00:00.179) 0:00:15.207 ******* 2020-11-20T12:00:10.576 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-11-20T12:00:10.578 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1605873376, "cache_updated": false, "changed": false} 2020-11-20T12:00:10.616 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:10 +0000 (0:00:03.563) 0:00:18.771 ******* 2020-11-20T12:00:10.624 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1605873420, "cache_updated": false, "changed": false} 2020-11-20T12:00:10.674 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:10 +0000 (0:00:00.058) 0:00:18.829 ******* 2020-11-20T12:00:11.508 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-11-20T12:00:11.509 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"cache_update_time": 1605873376, "cache_updated": false, "changed": false} 2020-11-20T12:00:11.552 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:11 +0000 (0:00:00.877) 0:00:19.706 ******* 2020-11-20T12:00:11.559 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"cache_update_time": 1605873420, "cache_updated": false, "changed": false} 2020-11-20T12:00:11.608 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:11 +0000 (0:00:00.055) 0:00:19.762 ******* 2020-11-20T12:00:11.616 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-11-20T12:00:11.618 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:11.659 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:11 +0000 (0:00:00.051) 0:00:19.813 ******* 2020-11-20T12:00:11.670 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:11.709 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:11 +0000 (0:00:00.049) 0:00:19.863 ******* 2020-11-20T12:00:11.987 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-11-20T12:00:11.989 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-11-20T12:00:12.024 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:12 +0000 (0:00:00.315) 0:00:20.179 ******* 2020-11-20T12:00:12.031 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-11-20T12:00:12.069 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:12 +0000 (0:00:00.045) 0:00:20.224 ******* 2020-11-20T12:00:12.890 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-11-20T12:00:12.892 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"cache_update_time": 1605873376, "cache_updated": false, "changed": false} 2020-11-20T12:00:12.939 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:12 +0000 (0:00:00.869) 0:00:21.094 ******* 2020-11-20T12:00:12.959 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"cache_update_time": 1605873420, "cache_updated": false, "changed": false} 2020-11-20T12:00:12.997 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:12 +0000 (0:00:00.057) 0:00:21.151 ******* 2020-11-20T12:00:13.326 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-11-20T12:00:13.327 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-11-20T12:00:13.349 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-11-20T12:00:13.627 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-11-20T12:00:13.645 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-11-20T12:00:13.895 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-11-20T12:00:13.932 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:13 +0000 (0:00:00.935) 0:00:22.087 ******* 2020-11-20T12:00:13.938 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-11-20T12:00:13.979 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:13 +0000 (0:00:00.046) 0:00:22.133 ******* 2020-11-20T12:00:14.062 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-11-20T12:00:14.065 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-11-20T12:00:14.124 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:14 +0000 (0:00:00.145) 0:00:22.279 ******* 2020-11-20T12:00:14.135 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-11-20T12:00:14.173 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:14 +0000 (0:00:00.048) 0:00:22.328 ******* 2020-11-20T12:00:14.433 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-11-20T12:00:14.435 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-11-20T12:00:14.488 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-11-20T12:00:14.750 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-11-20T12:00:14.794 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-11-20T12:00:15.057 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-11-20T12:00:15.099 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:15 +0000 (0:00:00.925) 0:00:23.253 ******* 2020-11-20T12:00:15.123 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-11-20T12:00:15.158 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:15 +0000 (0:00:00.059) 0:00:23.313 ******* 2020-11-20T12:00:15.166 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** ok: [smithi047.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-11-20T12:00:15.204 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:15 +0000 (0:00:00.046) 0:00:23.359 ******* 2020-11-20T12:00:15.214 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-11-20T12:00:15.255 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:15 +0000 (0:00:00.049) 0:00:23.409 ******* 2020-11-20T12:00:15.261 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-11-20T12:00:15.262 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:15.301 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:15 +0000 (0:00:00.046) 0:00:23.456 ******* 2020-11-20T12:00:15.308 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:15.348 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:15 +0000 (0:00:00.047) 0:00:23.503 ******* 2020-11-20T12:00:15.354 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-11-20T12:00:15.356 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T12:00:15.392 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:15 +0000 (0:00:00.042) 0:00:23.545 ******* 2020-11-20T12:00:15.397 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T12:00:15.434 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:15 +0000 (0:00:00.043) 0:00:23.589 ******* 2020-11-20T12:00:15.439 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-11-20T12:00:15.441 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:15.452 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:15.466 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:15.512 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:15 +0000 (0:00:00.077) 0:00:23.666 ******* 2020-11-20T12:00:15.519 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:15.523 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:15.560 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:15 +0000 (0:00:00.048) 0:00:23.715 ******* 2020-11-20T12:00:15.569 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-11-20T12:00:15.571 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:15.608 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:15 +0000 (0:00:00.047) 0:00:23.763 ******* 2020-11-20T12:00:15.615 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:15.654 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:15 +0000 (0:00:00.046) 0:00:23.809 ******* 2020-11-20T12:00:16.530 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** ok: [smithi047.front.sepia.ceph.com] => {"cache_update_time": 1605873420, "cache_updated": false, "changed": false} ok: [smithi189.front.sepia.ceph.com] => {"cache_update_time": 1605873376, "cache_updated": false, "changed": false} 2020-11-20T12:00:17.340 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:16 +0000 (0:00:00.910) 0:00:24.720 ******* Friday 20 November 2020 12:00:16 +0000 (0:00:00.033) 0:00:24.753 ******* TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} Friday 20 November 2020 12:00:16 +0000 (0:00:00.367) 0:00:25.120 ******* ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} Friday 20 November 2020 12:00:17 +0000 (0:00:00.047) 0:00:25.168 ******* TASK [common : Upload nagios check_mem script] ********************************* ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} Friday 20 November 2020 12:00:17 +0000 (0:00:00.317) 0:00:25.485 ******* 2020-11-20T12:00:17.370 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-11-20T12:00:17.425 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:17 +0000 (0:00:00.094) 0:00:25.580 ******* 2020-11-20T12:00:17.487 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-11-20T12:00:17.489 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-11-20T12:00:17.586 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-11-20T12:00:17.622 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-11-20T12:00:17.668 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:17 +0000 (0:00:00.240) 0:00:25.820 ******* 2020-11-20T12:00:17.693 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-11-20T12:00:17.695 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ ok: [smithi189.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-11-20T12:00:17.733 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:17 +0000 (0:00:00.066) 0:00:25.886 ******* 2020-11-20T12:00:17.788 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:17 +0000 (0:00:00.052) 0:00:25.938 ******* 2020-11-20T12:00:17.790 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-11-20T12:00:17.792 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T12:00:17.828 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:17 +0000 (0:00:00.044) 0:00:25.983 ******* 2020-11-20T12:00:17.835 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-11-20T12:00:17.837 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:17.872 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:17 +0000 (0:00:00.043) 0:00:26.026 ******* 2020-11-20T12:00:17.878 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-11-20T12:00:17.880 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:17.919 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:17 +0000 (0:00:00.047) 0:00:26.074 ******* 2020-11-20T12:00:17.926 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-11-20T12:00:17.931 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T12:00:17.966 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:17 +0000 (0:00:00.046) 0:00:26.120 ******* 2020-11-20T12:00:17.993 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-11-20T12:00:17.994 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:18.029 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:18 +0000 (0:00:00.063) 0:00:26.184 ******* 2020-11-20T12:00:18.246 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-11-20T12:00:18.248 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2020-11-20T12:00:18.289 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:18 +0000 (0:00:00.259) 0:00:26.444 ******* 2020-11-20T12:00:18.360 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2020-11-20T12:00:18.397 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:18 +0000 (0:00:00.108) 0:00:26.552 ******* 2020-11-20T12:00:18.908 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-11-20T12:00:18.910 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2020-11-20 11:59:36 UTC", "ActiveEnterTimestampMonotonic": "20185366", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "time-sync.target -.mount nss-lookup.target systemd-tmpfiles-setup.service sysinit.target system.slice local-fs.target remote-fs.target systemd-journald.socket var-run.mount network.target basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2020-11-20 11:59:36 UTC", "AssertTimestampMonotonic": "20183502", "Before": "getty@tty1.service multi-user.target xdm.service shutdown.target plymouth-quit.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2020-11-20 11:59:36 UTC", "ConditionTimestampMonotonic": "20183502", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-11-20T12:00:18.912 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1153", "ExecMainStartTimestamp": "Fri 2020-11-20 11:59:36 UTC", "ExecMainStartTimestampMonotonic": "20185302", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Fri 2020-11-20 11:59:36 UTC] ; stop_time=[n/a] ; pid=1153 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2020-11-20 11:59:36 UTC", "InactiveExitTimestampMonotonic": "20185366", "InvocationID": "fa7dc04acf894f5bbbe8fd31e91c4f4f", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2020-11-20T12:00:18.913 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1153", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2020-11-20T12:00:18.915 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2020-11-20 11:59:36 UTC", "StateChangeTimestampMonotonic": "20185366", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2020-11-20 11:59:36 UTC", "WatchdogTimestampMonotonic": "20185365", "WatchdogUSec": "0"}} ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2020-11-20 11:59:22 UTC", "ActiveEnterTimestampMonotonic": "18157934", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target basic.target -.mount network.target var-run.mount systemd-tmpfiles-setup.service nss-lookup.target time-sync.target system.slice 2020-11-20T12:00:18.917 INFO:teuthology.task.ansible.out: local-fs.target systemd-journald.socket remote-fs.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2020-11-20 11:59:22 UTC", "AssertTimestampMonotonic": "18157197", "Before": "xdm.service getty@tty1.service plymouth-quit.service shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2020-11-20 11:59:22 UTC", "ConditionTimestampMonotonic": "18157197", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-server.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1218", "ExecMainStartTimestamp": "Fri 2020-11-20 11:59:22 UTC", "ExecMai 2020-11-20T12:00:18.918 INFO:teuthology.task.ansible.out:nStartTimestampMonotonic": "18157907", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Fri 2020-11-20 11:59:22 UTC] ; stop_time=[n/a] ; pid=1218 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2020-11-20 11:59:22 UTC", "InactiveExitTimestampMonotonic": "18157934", "InvocationID": "41f851c4e1bb424fa235213523930888", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft 2020-11-20T12:00:18.920 INFO:teuthology.task.ansible.out:": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1218", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "St 2020-11-20T12:00:18.922 INFO:teuthology.task.ansible.out:artLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2020-11-20 11:59:22 UTC", "StateChangeTimestampMonotonic": "18157934", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2020-11-20 11:59:22 UTC", "WatchdogTimestampMonotonic": "18157934", "WatchdogUSec": "0"}} 2020-11-20T12:00:18.950 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:18 +0000 (0:00:00.552) 0:00:27.105 ******* 2020-11-20T12:00:18.985 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:18 +0000 (0:00:00.033) 0:00:27.139 ******* 2020-11-20T12:00:18.990 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-11-20T12:00:18.992 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:19.027 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:19 +0000 (0:00:00.043) 0:00:27.182 ******* 2020-11-20T12:00:19.035 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:19.071 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:19 +0000 (0:00:00.043) 0:00:27.225 ******* 2020-11-20T12:00:19.082 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-11-20T12:00:19.084 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:19.132 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:19 +0000 (0:00:00.061) 0:00:27.287 ******* 2020-11-20T12:00:19.141 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:19.181 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:19 +0000 (0:00:00.048) 0:00:27.335 ******* 2020-11-20T12:00:19.186 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-11-20T12:00:19.188 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:19.223 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:19 +0000 (0:00:00.042) 0:00:27.378 ******* 2020-11-20T12:00:19.230 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:19.270 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:19 +0000 (0:00:00.047) 0:00:27.425 ******* 2020-11-20T12:00:19.276 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-11-20T12:00:19.278 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:19.313 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:19 +0000 (0:00:00.042) 0:00:27.468 ******* 2020-11-20T12:00:19.320 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:19.355 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:19 +0000 (0:00:00.042) 0:00:27.510 ******* 2020-11-20T12:00:19.362 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-11-20T12:00:19.364 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:19.397 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:19 +0000 (0:00:00.042) 0:00:27.552 ******* 2020-11-20T12:00:19.404 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:19.441 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:19 +0000 (0:00:00.043) 0:00:27.596 ******* 2020-11-20T12:00:19.447 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-11-20T12:00:19.449 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:19.483 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:19 +0000 (0:00:00.042) 0:00:27.638 ******* 2020-11-20T12:00:19.490 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:19.530 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:19 +0000 (0:00:00.046) 0:00:27.685 ******* 2020-11-20T12:00:19.537 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-11-20T12:00:19.539 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:19.574 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:19 +0000 (0:00:00.043) 0:00:27.729 ******* 2020-11-20T12:00:19.583 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:19.626 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:19 +0000 (0:00:00.052) 0:00:27.781 ******* 2020-11-20T12:00:19.634 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-11-20T12:00:19.636 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:19.672 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:19 +0000 (0:00:00.045) 0:00:27.826 ******* 2020-11-20T12:00:19.678 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:19.715 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:19 +0000 (0:00:00.043) 0:00:27.870 ******* 2020-11-20T12:00:19.722 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-11-20T12:00:19.724 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:19.763 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:19 +0000 (0:00:00.048) 0:00:27.918 ******* 2020-11-20T12:00:19.770 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:19.807 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:19 +0000 (0:00:00.044) 0:00:27.962 ******* 2020-11-20T12:00:19.813 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-11-20T12:00:19.815 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:19.851 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:19 +0000 (0:00:00.043) 0:00:28.006 ******* 2020-11-20T12:00:19.858 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:19.896 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:19 +0000 (0:00:00.044) 0:00:28.050 ******* 2020-11-20T12:00:20.008 INFO:teuthology.task.ansible.out: TASK [common : Check for podman] *********************************************** 2020-11-20T12:00:20.010 INFO:teuthology.task.ansible.out:fatal: [smithi047.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": "podman --version", "msg": "[Errno 2] No such file or directory", "rc": 2} ...ignoring 2020-11-20T12:00:20.045 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:20 +0000 (0:00:00.149) 0:00:28.200 ******* 2020-11-20T12:00:20.072 INFO:teuthology.task.ansible.out: TASK [common : set_fact] ******************************************************* 2020-11-20T12:00:20.074 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:20.107 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:20 +0000 (0:00:00.062) 0:00:28.262 ******* 2020-11-20T12:00:20.259 INFO:teuthology.task.ansible.out: TASK [common : Check for podman] *********************************************** 2020-11-20T12:00:20.261 INFO:teuthology.task.ansible.out:fatal: [smithi189.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": "podman --version", "msg": "[Errno 2] No such file or directory", "rc": 2} ...ignoring 2020-11-20T12:00:20.297 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:20 +0000 (0:00:00.189) 0:00:28.451 ******* 2020-11-20T12:00:20.323 INFO:teuthology.task.ansible.out: TASK [common : set_fact] ******************************************************* 2020-11-20T12:00:20.325 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:20.359 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:20 +0000 (0:00:00.062) 0:00:28.513 ******* 2020-11-20T12:00:24.429 INFO:teuthology.task.ansible.out: TASK [common : Check for docker] *********************************************** 2020-11-20T12:00:24.431 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "cmd": ["docker", "--version"], "delta": "0:00:03.920348", "end": "2020-11-20 12:00:24.398675", "rc": 0, "start": "2020-11-20 12:00:20.478327", "stderr": "", "stderr_lines": [], "stdout": "Docker version 19.03.6, build 369ce74a3c", "stdout_lines": ["Docker version 19.03.6, build 369ce74a3c"]} 2020-11-20T12:00:24.465 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:24 +0000 (0:00:04.106) 0:00:32.619 ******* 2020-11-20T12:00:24.503 INFO:teuthology.task.ansible.out: TASK [common : set_fact] ******************************************************* 2020-11-20T12:00:24.504 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"ansible_facts": {"container_mirror_cert_path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000"}, "changed": false} 2020-11-20T12:00:24.539 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:24 +0000 (0:00:00.073) 0:00:32.693 ******* 2020-11-20T12:00:24.599 INFO:teuthology.task.ansible.out: TASK [common : Include encrypted variables] ************************************ ok: [smithi189.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T12:00:24.601 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T12:00:24.638 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:24 +0000 (0:00:00.099) 0:00:32.793 ******* 2020-11-20T12:00:24.772 INFO:teuthology.task.ansible.out: TASK [common : Create /etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000] *** 2020-11-20T12:00:24.773 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2020-11-20T12:00:24.811 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:24 +0000 (0:00:00.172) 0:00:32.965 ******* 2020-11-20T12:00:24.950 INFO:teuthology.task.ansible.out: TASK [common : Check for docker] *********************************************** changed: [smithi047.front.sepia.ceph.com] => {"changed": true, "cmd": ["docker", "--version"], "delta": "0:00:04.703584", "end": "2020-11-20 12:00:24.783558", "rc": 0, "start": "2020-11-20 12:00:20.079974", "stderr": "", "stderr_lines": [], "stdout": "Docker version 19.03.6, build 369ce74a3c", "stdout_lines": ["Docker version 19.03.6, build 369ce74a3c"]} 2020-11-20T12:00:24.989 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:24 +0000 (0:00:00.176) 0:00:33.142 ******* 2020-11-20T12:00:25.026 INFO:teuthology.task.ansible.out: TASK [common : set_fact] ******************************************************* 2020-11-20T12:00:25.028 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"ansible_facts": {"container_mirror_cert_path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000"}, "changed": false} 2020-11-20T12:00:25.071 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:25 +0000 (0:00:00.083) 0:00:33.226 ******* 2020-11-20T12:00:25.119 INFO:teuthology.task.ansible.out: TASK [common : Include encrypted variables] ************************************ 2020-11-20T12:00:25.121 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ok: [smithi047.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T12:00:25.165 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:25 +0000 (0:00:00.094) 0:00:33.320 ******* 2020-11-20T12:00:25.175 INFO:teuthology.task.ansible.out: TASK [common : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2020-11-20T12:00:25.175 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "checksum": "d386a5d594d8bb6c408cf08541a4eb609577a03d", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "md5sum": "e2cf8b6a8ad65ba91ea9fc53d49f1bf4", "mode": "0644", "owner": "root", "size": 2236, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1605873624.841636-240615039058360/source", "state": "file", "uid": 0} 2020-11-20T12:00:25.227 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:25 +0000 (0:00:00.062) 0:00:33.382 ******* 2020-11-20T12:00:25.257 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-11-20T12:00:25.259 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-11-20T12:00:25.293 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:25 +0000 (0:00:00.065) 0:00:33.447 ******* 2020-11-20T12:00:25.320 INFO:teuthology.task.ansible.out: TASK [common : Create /etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000] *** 2020-11-20T12:00:25.322 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => {"changed": true, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2020-11-20T12:00:25.364 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:25 +0000 (0:00:00.071) 0:00:33.519 ******* 2020-11-20T12:00:25.372 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-11-20T12:00:25.373 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-11-20T12:00:25.430 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:25 +0000 (0:00:00.065) 0:00:33.585 ******* 2020-11-20T12:00:25.467 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-11-20T12:00:25.469 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-11-20T12:00:25.509 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:25 +0000 (0:00:00.079) 0:00:33.664 ******* 2020-11-20T12:00:25.687 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ ok: [smithi189.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-11-20T12:00:25.823 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:25 +0000 (0:00:00.313) 0:00:33.978 ******* 2020-11-20T12:00:25.831 INFO:teuthology.task.ansible.out: TASK [common : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2020-11-20T12:00:25.833 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => {"changed": true, "checksum": "d386a5d594d8bb6c408cf08541a4eb609577a03d", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "md5sum": "e2cf8b6a8ad65ba91ea9fc53d49f1bf4", "mode": "0644", "owner": "root", "size": 2236, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1605873625.3914034-132774869266623/source", "state": "file", "uid": 0} 2020-11-20T12:00:25.888 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:25 +0000 (0:00:00.065) 0:00:34.043 ******* 2020-11-20T12:00:25.918 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** ok: [smithi047.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-11-20T12:00:25.959 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:25 +0000 (0:00:00.070) 0:00:34.113 ******* 2020-11-20T12:00:25.965 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-11-20T12:00:25.967 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-11-20T12:00:26.003 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:25 +0000 (0:00:00.044) 0:00:34.158 ******* 2020-11-20T12:00:26.009 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-11-20T12:00:26.012 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-11-20T12:00:26.048 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:26 +0000 (0:00:00.045) 0:00:34.203 ******* 2020-11-20T12:00:26.087 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** ok: [smithi047.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-11-20T12:00:26.123 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:26 +0000 (0:00:00.075) 0:00:34.278 ******* 2020-11-20T12:00:26.129 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-11-20T12:00:26.131 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2020-11-20T12:00:26.168 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:26 +0000 (0:00:00.044) 0:00:34.322 ******* 2020-11-20T12:00:26.173 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-11-20T12:00:26.175 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-11-20T12:00:26.212 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:26 +0000 (0:00:00.043) 0:00:34.365 ******* 2020-11-20T12:00:26.217 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-11-20T12:00:26.218 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:26.279 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:26 +0000 (0:00:00.066) 0:00:34.432 ******* 2020-11-20T12:00:26.311 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-11-20T12:00:26.314 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:26.350 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:26 +0000 (0:00:00.072) 0:00:34.505 ******* 2020-11-20T12:00:26.358 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-11-20T12:00:26.360 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-11-20T12:00:26.402 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:26 +0000 (0:00:00.051) 0:00:34.556 ******* 2020-11-20T12:00:26.519 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-11-20T12:00:26.525 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-11-20T12:00:26.562 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:26 +0000 (0:00:00.160) 0:00:34.717 ******* 2020-11-20T12:00:26.593 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2020-11-20T12:00:26.644 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:26 +0000 (0:00:00.082) 0:00:34.799 ******* 2020-11-20T12:00:26.676 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:26.709 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:26 +0000 (0:00:00.065) 0:00:34.864 ******* 2020-11-20T12:00:26.748 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-11-20T12:00:26.752 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:26.791 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:26 +0000 (0:00:00.081) 0:00:34.946 ******* 2020-11-20T12:00:26.951 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-11-20T12:00:26.952 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-11-20T12:00:26.996 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:26 +0000 (0:00:00.204) 0:00:35.150 ******* 2020-11-20T12:00:27.012 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-11-20T12:00:27.014 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-11-20T12:00:27.054 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:27 +0000 (0:00:00.058) 0:00:35.209 ******* 2020-11-20T12:00:27.083 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-11-20T12:00:27.085 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi189"}, "changed": false} 2020-11-20T12:00:27.121 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:27 +0000 (0:00:00.066) 0:00:35.276 ******* 2020-11-20T12:00:27.383 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-11-20T12:00:27.386 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-11-20T12:00:27.419 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:27 +0000 (0:00:00.298) 0:00:35.574 ******* 2020-11-20T12:00:27.451 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi047.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi047"}, "changed": false} 2020-11-20T12:00:27.487 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:27 +0000 (0:00:00.067) 0:00:35.642 ******* 2020-11-20T12:00:27.963 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-11-20T12:00:27.966 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi047.front.sepia.ceph.com", "ansible_hostname": "smithi047", "ansible_nodename": "smithi047"}, "changed": true, "name": "smithi047"} 2020-11-20T12:00:27.999 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:27 +0000 (0:00:00.511) 0:00:36.154 ******* 2020-11-20T12:00:28.006 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi189.front.sepia.ceph.com", "ansible_hostname": "smithi189", "ansible_nodename": "smithi189"}, "changed": true, "name": "smithi189"} 2020-11-20T12:00:28.045 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:28 +0000 (0:00:00.046) 0:00:36.200 ******* 2020-11-20T12:00:28.341 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-11-20T12:00:28.343 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-11-20T12:00:28.377 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:28 +0000 (0:00:00.331) 0:00:36.532 ******* 2020-11-20T12:00:28.384 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-11-20T12:00:28.443 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:28 +0000 (0:00:00.066) 0:00:36.598 ******* 2020-11-20T12:00:28.696 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-11-20T12:00:28.698 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-11-20T12:00:28.745 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:28 +0000 (0:00:00.298) 0:00:36.897 ******* 2020-11-20T12:00:28.764 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-11-20T12:00:28.803 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:28 +0000 (0:00:00.060) 0:00:36.957 ******* 2020-11-20T12:00:29.296 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** ok: [smithi047.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-11-20T12:00:29.331 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:29 +0000 (0:00:00.528) 0:00:37.486 ******* 2020-11-20T12:00:29.337 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-11-20T12:00:29.373 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:29 +0000 (0:00:00.042) 0:00:37.528 ******* 2020-11-20T12:00:29.379 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-11-20T12:00:29.381 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:29.424 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:29 +0000 (0:00:00.044) 0:00:37.573 ******* 2020-11-20T12:00:29.425 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:29.463 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:29 +0000 (0:00:00.043) 0:00:37.617 ******* 2020-11-20T12:00:29.468 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-20T12:00:29.470 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:29.514 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:29 +0000 (0:00:00.051) 0:00:37.669 ******* 2020-11-20T12:00:29.520 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:29.562 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:29 +0000 (0:00:00.047) 0:00:37.716 ******* 2020-11-20T12:00:29.568 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-20T12:00:29.572 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:29.612 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:29 +0000 (0:00:00.050) 0:00:37.767 ******* 2020-11-20T12:00:29.621 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:29.669 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:29 +0000 (0:00:00.049) 0:00:37.817 ******* TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:29.927 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:29 +0000 (0:00:00.047) 0:00:37.864 ******* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:29 +0000 (0:00:00.048) 0:00:37.913 ******* TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:29 +0000 (0:00:00.088) 0:00:38.001 ******* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:29 +0000 (0:00:00.048) 0:00:38.050 ******* TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:29.944 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:29 +0000 (0:00:00.045) 0:00:38.096 ******* 2020-11-20T12:00:29.948 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:29.985 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:29 +0000 (0:00:00.044) 0:00:38.140 ******* 2020-11-20T12:00:29.992 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-20T12:00:29.994 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:30.029 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:30 +0000 (0:00:00.044) 0:00:38.184 ******* 2020-11-20T12:00:30.036 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:30.077 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:30 +0000 (0:00:00.047) 0:00:38.232 ******* 2020-11-20T12:00:30.083 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:30.123 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:30 +0000 (0:00:00.045) 0:00:38.277 ******* 2020-11-20T12:00:30.130 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:30.167 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:30 +0000 (0:00:00.044) 0:00:38.321 ******* 2020-11-20T12:00:30.173 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-11-20T12:00:30.212 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:30 +0000 (0:00:00.045) 0:00:38.367 ******* 2020-11-20T12:00:30.262 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:30 +0000 (0:00:00.049) 0:00:38.417 ******* 2020-11-20T12:00:30.269 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-11-20T12:00:30.318 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:30 +0000 (0:00:00.054) 0:00:38.471 ******* 2020-11-20T12:00:30.380 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:30 +0000 (0:00:00.061) 0:00:38.532 ******* 2020-11-20T12:00:30.384 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-11-20T12:00:30.429 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:30 +0000 (0:00:00.045) 0:00:38.578 ******* 2020-11-20T12:00:30.467 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:30 +0000 (0:00:00.043) 0:00:38.622 ******* 2020-11-20T12:00:30.473 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:30.551 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:30 +0000 (0:00:00.081) 0:00:38.703 ******* 2020-11-20T12:00:30.556 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:30.610 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:30 +0000 (0:00:00.062) 0:00:38.765 ******* 2020-11-20T12:00:30.619 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* skipping: [smithi047.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:30.660 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:30 +0000 (0:00:00.049) 0:00:38.815 ******* 2020-11-20T12:00:30.667 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:30.669 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-11-20T12:00:30.707 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:30 +0000 (0:00:00.046) 0:00:38.861 ******* 2020-11-20T12:00:30.717 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:30.751 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:30 +0000 (0:00:00.044) 0:00:38.906 ******* 2020-11-20T12:00:30.765 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:30.800 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:30 +0000 (0:00:00.044) 0:00:38.950 ******* 2020-11-20T12:00:30.802 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:30.838 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:30 +0000 (0:00:00.043) 0:00:38.993 ******* 2020-11-20T12:00:30.845 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:30.892 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:30 +0000 (0:00:00.053) 0:00:39.047 ******* 2020-11-20T12:00:30.898 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:30.936 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:30 +0000 (0:00:00.043) 0:00:39.091 ******* 2020-11-20T12:00:30.943 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:30.987 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:30 +0000 (0:00:00.050) 0:00:39.141 ******* 2020-11-20T12:00:30.992 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-11-20T12:00:30.994 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:31.032 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:31 +0000 (0:00:00.046) 0:00:39.187 ******* 2020-11-20T12:00:31.040 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:31.074 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:31 +0000 (0:00:00.042) 0:00:39.229 ******* 2020-11-20T12:00:31.084 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:31.118 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:31 +0000 (0:00:00.042) 0:00:39.272 ******* 2020-11-20T12:00:31.125 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:31.162 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:31 +0000 (0:00:00.044) 0:00:39.316 ******* 2020-11-20T12:00:31.167 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-11-20T12:00:31.172 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:31.206 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:31 +0000 (0:00:00.044) 0:00:39.361 ******* 2020-11-20T12:00:31.213 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:31.251 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:31 +0000 (0:00:00.044) 0:00:39.406 ******* 2020-11-20T12:00:31.257 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-20T12:00:31.259 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:31.295 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:31 +0000 (0:00:00.044) 0:00:39.450 ******* 2020-11-20T12:00:31.304 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:31.356 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:31 +0000 (0:00:00.059) 0:00:39.510 ******* 2020-11-20T12:00:31.368 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-20T12:00:31.372 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:31.447 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:31 +0000 (0:00:00.092) 0:00:39.602 ******* 2020-11-20T12:00:31.454 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:31.492 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:31 +0000 (0:00:00.043) 0:00:39.646 ******* 2020-11-20T12:00:31.497 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:31.535 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:31 +0000 (0:00:00.043) 0:00:39.690 ******* 2020-11-20T12:00:33.505 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:31 +0000 (0:00:00.045) 0:00:39.735 ******* TASK [testnode : Install epel packages] **************************************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:31 +0000 (0:00:00.045) 0:00:39.781 ******* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:31 +0000 (0:00:00.051) 0:00:39.832 ******* TASK [testnode : Remove packages] ********************************************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:31 +0000 (0:00:00.044) 0:00:39.876 ******* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:31 +0000 (0:00:00.042) 0:00:39.919 ******* TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:31 +0000 (0:00:00.054) 0:00:39.973 ******* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:31 +0000 (0:00:00.058) 0:00:40.031 ******* TASK [testnode : Stop and disable firewalld] *********************************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:31 +0000 (0:00:00.047) 0:00:40.079 ******* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was 2020-11-20T12:00:33.507 INFO:teuthology.task.ansible.out:False"} Friday 20 November 2020 12:00:31 +0000 (0:00:00.044) 0:00:40.123 ******* TASK [testnode : Stop and disable iptables] ************************************ skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:32 +0000 (0:00:00.042) 0:00:40.166 ******* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:32 +0000 (0:00:00.044) 0:00:40.211 ******* TASK [testnode : Enable SELinux] *********************************************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:32 +0000 (0:00:00.043) 0:00:40.254 ******* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:32 +0000 (0:00:00.045) 0:00:40.300 ******* TASK [testnode : Install abrt] ************************************************* skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:32 +0000 (0:00:00.043) 0:00:40.344 ******* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:32 +0000 (0:00:00.043) 0:00:40.387 ******* TASK [testnode : Enable abrt-auto-reporting] *********************************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:32 +0000 (0:00:00.047) 0:00:40.435 ******* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:32 +0000 (0:00:00.043) 0:00:40.478 ******* TASK [testn 2020-11-20T12:00:33.508 INFO:teuthology.task.ansible.out:ode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:32 +0000 (0:00:00.051) 0:00:40.530 ******* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:32 +0000 (0:00:00.043) 0:00:40.574 ******* TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:32 +0000 (0:00:00.042) 0:00:40.616 ******* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:32 +0000 (0:00:00.042) 0:00:40.659 ******* TASK [testnode : Restart abrtd] ************************************************ skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:32 +0000 (0:00:00.043) 0:00:40.702 ******* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:32 +0000 (0:00:00.045) 0:00:40.748 ******* TASK [testnode : Check for custom repos] *************************************** ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005231", "end": "2020-11-20 12:00:32.528992", "rc": 0, "start": "2020-11-20 12:00:32.523761", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Friday 20 November 2020 12:00:32 +0000 (0:00:00.141) 0:00:40.890 ******* ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.003492", "end": "2020-11- 2020-11-20T12:00:33.510 INFO:teuthology.task.ansible.out:20 12:00:32.710526", "rc": 0, "start": "2020-11-20 12:00:32.707034", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Friday 20 November 2020 12:00:32 +0000 (0:00:00.046) 0:00:40.936 ******* TASK [testnode : Remove custom repos] ****************************************** Friday 20 November 2020 12:00:32 +0000 (0:00:00.045) 0:00:40.982 ******* Friday 20 November 2020 12:00:32 +0000 (0:00:00.043) 0:00:41.026 ******* TASK [testnode : Set apt preferences] ****************************************** ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} Friday 20 November 2020 12:00:33 +0000 (0:00:00.344) 0:00:41.370 ******* ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} Friday 20 November 2020 12:00:33 +0000 (0:00:00.043) 0:00:41.414 ******* TASK [testnode : Add sources list] ********************************************* skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:33 +0000 (0:00:00.047) 0:00:41.461 ******* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:00:33 +0000 (0:00:00.043) 0:00:41.505 ******* 2020-11-20T12:00:34.690 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-11-20T12:00:34.692 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-11-20T12:00:34.733 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-11-20T12:00:35.760 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-11-20T12:00:35.797 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:35 +0000 (0:00:02.446) 0:00:43.952 ******* 2020-11-20T12:00:35.805 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-11-20T12:00:35.844 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:35 +0000 (0:00:00.047) 0:00:43.999 ******* 2020-11-20T12:00:36.787 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-11-20T12:00:36.788 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"cache_update_time": 1605873376, "cache_updated": false, "changed": false} ok: [smithi047.front.sepia.ceph.com] => {"cache_update_time": 1605873420, "cache_updated": false, "changed": false} 2020-11-20T12:00:36.824 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:36 +0000 (0:00:00.979) 0:00:44.979 ******* 2020-11-20T12:00:36.860 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:36 +0000 (0:00:00.035) 0:00:45.015 ******* 2020-11-20T12:00:36.867 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-11-20T12:00:36.904 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:36 +0000 (0:00:00.043) 0:00:45.058 ******* 2020-11-20T12:00:36.947 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:36 +0000 (0:00:00.043) 0:00:45.101 ******* 2020-11-20T12:00:37.069 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-11-20T12:00:37.071 INFO:teuthology.task.ansible.out:[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi047.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.035369", "end": "2020-11-20 12:00:36.909687", "rc": 0, "start": "2020-11-20 12:00:36.874318", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:00:37.109 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:37 +0000 (0:00:00.162) 0:00:45.264 ******* 2020-11-20T12:00:37.129 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.034603", "end": "2020-11-20 12:00:37.094871", "rc": 0, "start": "2020-11-20 12:00:37.060268", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:00:37.190 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:37 +0000 (0:00:00.081) 0:00:45.345 ******* 2020-11-20T12:00:39.952 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** ok: [smithi189.front.sepia.ceph.com] => {"cache_update_time": 1605873376, "cache_updated": false, "changed": false} 2020-11-20T12:00:39.987 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:39 +0000 (0:00:02.796) 0:00:48.141 ******* 2020-11-20T12:00:40.160 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"cache_update_time": 1605873420, "cache_updated": false, "changed": false} 2020-11-20T12:00:40.196 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:40 +0000 (0:00:00.208) 0:00:48.350 ******* 2020-11-20T12:00:49.013 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-11-20T12:00:49.015 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 20 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137606 files and directorie 2020-11-20T12:00:49.016 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 20 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2020-11-20T12:00:49.018 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137606 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-11-20T12:00:49.051 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:49 +0000 (0:00:08.855) 0:00:57.206 ******* 2020-11-20T12:00:49.418 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 20 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137606 files and directorie 2020-11-20T12:00:49.420 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 20 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2020-11-20T12:00:49.421 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137606 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-11-20T12:00:49.457 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:49 +0000 (0:00:00.406) 0:00:57.612 ******* 2020-11-20T12:00:49.950 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-20T12:00:49.952 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"cache_update_time": 1605873376, "cache_updated": false, "changed": false} 2020-11-20T12:00:49.999 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:49 +0000 (0:00:00.541) 0:00:58.154 ******* 2020-11-20T12:00:50.026 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-11-20T12:00:50.030 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:50.061 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:50 +0000 (0:00:00.062) 0:00:58.216 ******* 2020-11-20T12:00:50.384 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-20T12:00:50.386 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"cache_update_time": 1605873420, "cache_updated": false, "changed": false} 2020-11-20T12:00:50.421 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:50 +0000 (0:00:00.359) 0:00:58.576 ******* 2020-11-20T12:00:50.448 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-11-20T12:00:50.450 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:00:50.484 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:50 +0000 (0:00:00.062) 0:00:58.639 ******* 2020-11-20T12:00:58.991 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-20T12:00:58.992 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"cache_update_time": 1605873376, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 20 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.5 [2723 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.5 [923 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.33 [40.1 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.33 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.33 [5193 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.33 [873 kB]\nFetched 10.4 MB in 1s (9577 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2020-11-20T12:00:58.994 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137464 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.5_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.5) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.5_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.5) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.5) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.5) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2020-11-20T12:00:58.995 INFO:teuthology.task.ansible.out:u1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 20 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.5 [2723 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.5 [923 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.33 [40.1 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.33 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.33 [5193 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.33 [873 kB]", "Fetched 10.4 MB in 1s (9577 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(R 2020-11-20T12:00:58.997 INFO:teuthology.task.ansible.out:eading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137464 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.5_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.5) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.5_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.5) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.5) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.5) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-11-20T12:00:59.032 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:59 +0000 (0:00:08.548) 0:01:07.187 ******* 2020-11-20T12:00:59.324 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => {"cache_update_time": 1605873420, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 20 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.5 [2723 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.5 [923 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.33 [40.1 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.33 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.33 [5193 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.33 [873 kB]\nFetched 10.4 MB in 1s (9704 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2020-11-20T12:00:59.326 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137464 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.5_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.5) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.5_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.5) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.5) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.5) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2020-11-20T12:00:59.327 INFO:teuthology.task.ansible.out:u1.3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 20 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.5 [2723 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.5 [923 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.33 [40.1 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.33 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.33 [5193 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.33 [873 kB]", "Fetched 10.4 MB in 1s (9704 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(R 2020-11-20T12:00:59.329 INFO:teuthology.task.ansible.out:eading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137464 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.5_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.5) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.5_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.5) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.33_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.5) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.5) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.33) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.33) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.3) ..."]} 2020-11-20T12:00:59.360 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:59 +0000 (0:00:00.328) 0:01:07.515 ******* 2020-11-20T12:00:59.876 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-11-20T12:00:59.878 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"cache_update_time": 1605873376, "cache_updated": false, "changed": false} 2020-11-20T12:00:59.921 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:00:59 +0000 (0:00:00.559) 0:01:08.075 ******* 2020-11-20T12:01:00.048 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-11-20T12:01:00.103 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:00 +0000 (0:00:00.182) 0:01:08.258 ******* 2020-11-20T12:01:00.130 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-11-20T12:01:00.132 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:00.167 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:00 +0000 (0:00:00.062) 0:01:08.320 ******* 2020-11-20T12:01:00.267 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-11-20T12:01:00.269 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"cache_update_time": 1605873420, "cache_updated": false, "changed": false} 2020-11-20T12:01:00.314 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:00 +0000 (0:00:00.148) 0:01:08.469 ******* 2020-11-20T12:01:00.321 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-11-20T12:01:00.323 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-11-20T12:01:00.359 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:00 +0000 (0:00:00.044) 0:01:08.513 ******* 2020-11-20T12:01:00.439 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-11-20T12:01:00.479 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:00 +0000 (0:00:00.120) 0:01:08.634 ******* 2020-11-20T12:01:00.507 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-11-20T12:01:00.510 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:00.552 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:00 +0000 (0:00:00.072) 0:01:08.707 ******* 2020-11-20T12:01:00.696 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-11-20T12:01:00.699 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} TASK [testnode : Upload /etc/fuse.conf.] *************************************** ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-11-20T12:01:00.732 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:00 +0000 (0:00:00.180) 0:01:08.887 ******* 2020-11-20T12:01:00.770 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:00 +0000 (0:00:00.037) 0:01:08.925 ******* 2020-11-20T12:01:00.944 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ ok: [smithi189.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-11-20T12:01:00.979 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:00 +0000 (0:00:00.209) 0:01:09.134 ******* 2020-11-20T12:01:01.092 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-11-20T12:01:01.094 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-11-20T12:01:01.145 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:01 +0000 (0:00:00.162) 0:01:09.297 ******* 2020-11-20T12:01:01.152 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** ok: [smithi189.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-20T12:01:01.197 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:01 +0000 (0:00:00.054) 0:01:09.352 ******* 2020-11-20T12:01:01.305 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-11-20T12:01:01.306 INFO:teuthology.task.ansible.out: 2020-11-20T12:01:01.306 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-11-20T12:01:01.345 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:01 +0000 (0:00:00.147) 0:01:09.499 ******* 2020-11-20T12:01:01.477 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** ok: [smithi047.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-20T12:01:01.514 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:01 +0000 (0:00:00.169) 0:01:09.668 ******* 2020-11-20T12:01:01.533 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-11-20T12:01:01.541 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Fri 2020-11-20 11:59:37 UTC", "ActiveEnterTimestampMonotonic": "20598014", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "nss-lookup.target network.target systemd-journald.socket remote-fs.target system.slice sysinit.target -.mount basic.target systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2020-11-20 11:59:36 UTC", "AssertTimestampMonotonic": "20181991", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2020-11-20 11:59:36 UTC", "ConditionTimestampMonotonic": "20181991", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-11-20T12:01:01.545 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1238", "ExecMainStartTimestamp": "Fri 2020-11-20 11:59:37 UTC", "ExecMainStartTimestampMonotonic": "20597999", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2020-11-20 11:59:36 UTC", "InactiveExitTimestampMonotonic": "20183428", "InvocationID": "e84e1de31a9d40178ff715dc30e0184a", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-11-20T12:01:01.549 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1238", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-11-20T12:01:01.553 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2020-11-20 11:59:37 UTC", "StateChangeTimestampMonotonic": "20598014", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2020-11-20 11:59:37 UTC", "WatchdogTimestampMonotonic": "20598012", "WatchdogUSec": "0"}} 2020-11-20T12:01:01.573 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:01 +0000 (0:00:00.058) 0:01:09.726 ******* 2020-11-20T12:01:01.600 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:01.638 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:01 +0000 (0:00:00.065) 0:01:09.792 ******* 2020-11-20T12:01:01.664 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-20T12:01:01.666 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:01.702 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:01 +0000 (0:00:00.064) 0:01:09.857 ******* 2020-11-20T12:01:01.729 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:01.767 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:01 +0000 (0:00:00.065) 0:01:09.922 ******* 2020-11-20T12:01:01.794 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:01.795 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-11-20T12:01:01.797 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Fri 2020-11-20 11:59:23 UTC", "ActiveEnterTimestampMonotonic": "18623752", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "nss-lookup.target systemd-journald.socket -.mount systemd-tmpfiles-setup.service system.slice basic.target network.target remote-fs.target sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2020-11-20 11:59:22 UTC", "AssertTimestampMonotonic": "18160011", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2020-11-20 11:59:22 UTC", "ConditionTimestampMonotonic": "18160011", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-11-20T12:01:01.798 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1288", "ExecMainStartTimestamp": "Fri 2020-11-20 11:59:23 UTC", "ExecMainStartTimestampMonotonic": "18623741", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2020-11-20 11:59:22 UTC", "InactiveExitTimestampMonotonic": "18161561", "InvocationID": "5b92870898064ad6967d072e67e18458", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-11-20T12:01:01.800 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1288", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-11-20T12:01:01.801 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2020-11-20 11:59:23 UTC", "StateChangeTimestampMonotonic": "18623752", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2020-11-20 11:59:23 UTC", "WatchdogTimestampMonotonic": "18623751", "WatchdogUSec": "0"}} 2020-11-20T12:01:01.832 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:01 +0000 (0:00:00.065) 0:01:09.987 ******* 2020-11-20T12:01:01.870 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:01 +0000 (0:00:00.037) 0:01:10.025 ******* 2020-11-20T12:01:01.876 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-20T12:01:01.878 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:01.918 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:01 +0000 (0:00:00.047) 0:01:10.073 ******* 2020-11-20T12:01:01.925 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-20T12:01:01.927 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:01.963 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:01 +0000 (0:00:00.044) 0:01:10.117 ******* 2020-11-20T12:01:01.969 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-20T12:01:01.970 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:02.006 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:02 +0000 (0:00:00.043) 0:01:10.160 ******* 2020-11-20T12:01:02.012 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-20T12:01:02.016 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:02.053 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:02 +0000 (0:00:00.047) 0:01:10.208 ******* 2020-11-20T12:01:02.059 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-20T12:01:02.062 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:02.097 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:02 +0000 (0:00:00.044) 0:01:10.252 ******* 2020-11-20T12:01:02.104 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-20T12:01:02.106 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:02.144 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:02 +0000 (0:00:00.046) 0:01:10.299 ******* 2020-11-20T12:01:02.151 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-20T12:01:02.153 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:02.194 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:02 +0000 (0:00:00.050) 0:01:10.349 ******* 2020-11-20T12:01:02.201 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-20T12:01:02.203 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:02.240 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:02 +0000 (0:00:00.045) 0:01:10.395 ******* 2020-11-20T12:01:02.247 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-20T12:01:02.249 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:02.288 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:02 +0000 (0:00:00.048) 0:01:10.443 ******* 2020-11-20T12:01:02.297 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T12:01:02.299 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:02.333 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:02 +0000 (0:00:00.044) 0:01:10.488 ******* 2020-11-20T12:01:02.339 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-20T12:01:02.341 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:02.376 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:02 +0000 (0:00:00.043) 0:01:10.531 ******* 2020-11-20T12:01:02.383 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-20T12:01:02.384 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:02.419 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:02 +0000 (0:00:00.042) 0:01:10.574 ******* 2020-11-20T12:01:02.425 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-20T12:01:02.426 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:02.462 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:02 +0000 (0:00:00.042) 0:01:10.617 ******* 2020-11-20T12:01:02.468 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-20T12:01:02.470 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:02.507 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:02 +0000 (0:00:00.045) 0:01:10.662 ******* 2020-11-20T12:01:02.514 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:02.551 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:02 +0000 (0:00:00.043) 0:01:10.705 ******* 2020-11-20T12:01:02.557 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-20T12:01:02.559 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:02.593 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:02 +0000 (0:00:00.043) 0:01:10.748 ******* 2020-11-20T12:01:02.599 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T12:01:02.601 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:02.637 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:02 +0000 (0:00:00.043) 0:01:10.791 ******* 2020-11-20T12:01:02.643 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-20T12:01:02.644 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:02.682 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:02 +0000 (0:00:00.045) 0:01:10.837 ******* 2020-11-20T12:01:02.689 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-20T12:01:02.691 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:02.726 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:02 +0000 (0:00:00.044) 0:01:10.881 ******* 2020-11-20T12:01:02.733 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-20T12:01:02.735 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:02.769 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:02 +0000 (0:00:00.042) 0:01:10.924 ******* 2020-11-20T12:01:02.775 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-20T12:01:02.777 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:02.817 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:02 +0000 (0:00:00.047) 0:01:10.971 ******* 2020-11-20T12:01:02.824 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-20T12:01:02.825 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:02.860 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:02 +0000 (0:00:00.043) 0:01:11.015 ******* 2020-11-20T12:01:02.871 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:02.907 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:02 +0000 (0:00:00.047) 0:01:11.062 ******* 2020-11-20T12:01:02.914 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:02.950 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:02 +0000 (0:00:00.042) 0:01:11.105 ******* 2020-11-20T12:01:02.957 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-20T12:01:02.959 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:02.994 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:02 +0000 (0:00:00.043) 0:01:11.149 ******* 2020-11-20T12:01:03.001 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:03.037 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:03 +0000 (0:00:00.042) 0:01:11.192 ******* 2020-11-20T12:01:03.044 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:03.080 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:03 +0000 (0:00:00.042) 0:01:11.235 ******* 2020-11-20T12:01:03.086 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-20T12:01:03.087 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:03.125 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:03 +0000 (0:00:00.045) 0:01:11.280 ******* 2020-11-20T12:01:03.131 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-20T12:01:03.133 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:03.168 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:03 +0000 (0:00:00.042) 0:01:11.322 ******* 2020-11-20T12:01:03.175 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-20T12:01:03.177 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:03.210 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:03 +0000 (0:00:00.042) 0:01:11.365 ******* 2020-11-20T12:01:03.217 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-20T12:01:03.219 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:03.254 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:03 +0000 (0:00:00.043) 0:01:11.409 ******* 2020-11-20T12:01:03.260 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-20T12:01:03.262 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:03.298 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:03 +0000 (0:00:00.043) 0:01:11.452 ******* 2020-11-20T12:01:03.303 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-20T12:01:03.305 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:03.340 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:03 +0000 (0:00:00.042) 0:01:11.495 ******* 2020-11-20T12:01:03.347 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-20T12:01:03.349 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:03.386 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:03 +0000 (0:00:00.045) 0:01:11.541 ******* 2020-11-20T12:01:03.392 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-20T12:01:03.394 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:03.432 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:03 +0000 (0:00:00.046) 0:01:11.587 ******* 2020-11-20T12:01:03.439 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-20T12:01:03.441 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:03.477 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:03 +0000 (0:00:00.044) 0:01:11.632 ******* 2020-11-20T12:01:03.484 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:03.521 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:03 +0000 (0:00:00.044) 0:01:11.676 ******* 2020-11-20T12:01:03.528 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:03.566 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:03 +0000 (0:00:00.044) 0:01:11.720 ******* 2020-11-20T12:01:03.571 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-20T12:01:03.573 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:03.610 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:03 +0000 (0:00:00.044) 0:01:11.765 ******* 2020-11-20T12:01:03.616 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-20T12:01:03.618 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:03.653 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:03 +0000 (0:00:00.042) 0:01:11.807 ******* 2020-11-20T12:01:03.659 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-20T12:01:03.661 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:03.695 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:03 +0000 (0:00:00.042) 0:01:11.850 ******* 2020-11-20T12:01:03.703 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:03.741 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:03 +0000 (0:00:00.046) 0:01:11.896 ******* 2020-11-20T12:01:03.749 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:03.784 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:03 +0000 (0:00:00.043) 0:01:11.939 ******* 2020-11-20T12:01:03.792 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-11-20T12:01:03.793 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:03.829 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:03 +0000 (0:00:00.043) 0:01:11.983 ******* 2020-11-20T12:01:03.834 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-20T12:01:03.836 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:03.871 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:03 +0000 (0:00:00.043) 0:01:12.026 ******* 2020-11-20T12:01:03.899 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-20T12:01:03.901 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:03.934 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:03 +0000 (0:00:00.062) 0:01:12.089 ******* 2020-11-20T12:01:03.963 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-11-20T12:01:03.965 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:04.018 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:04 +0000 (0:00:00.083) 0:01:12.173 ******* 2020-11-20T12:01:04.305 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:04 +0000 (0:00:00.041) 0:01:12.214 ******* TASK [testnode : Set grub config.] ********************************************* skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 12:01:04 +0000 (0:00:00.046) 0:01:12.261 ******* TASK [testnode : Remove /etc/ceph] ********************************************* ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} Friday 20 November 2020 12:01:04 +0000 (0:00:00.178) 0:01:12.440 ******* 2020-11-20T12:01:04.431 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-20T12:01:04.433 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-11-20T12:01:04.475 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:04 +0000 (0:00:00.188) 0:01:12.629 ******* 2020-11-20T12:01:04.504 INFO:teuthology.task.ansible.out: 2020-11-20T12:01:04.506 INFO:teuthology.task.ansible.out:TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:04.506 INFO:teuthology.task.ansible.out: 2020-11-20T12:01:04.543 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:04 +0000 (0:00:00.069) 0:01:12.698 ******* 2020-11-20T12:01:04.625 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-20T12:01:04.626 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-11-20T12:01:04.668 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:04 +0000 (0:00:00.124) 0:01:12.823 ******* 2020-11-20T12:01:04.695 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-20T12:01:04.697 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:04.734 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:04 +0000 (0:00:00.065) 0:01:12.889 ******* 2020-11-20T12:01:04.857 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-11-20T12:01:04.892 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:04 +0000 (0:00:00.157) 0:01:13.047 ******* 2020-11-20T12:01:05.079 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-11-20T12:01:05.116 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:05 +0000 (0:00:00.224) 0:01:13.271 ******* 2020-11-20T12:01:05.216 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-11-20T12:01:05.218 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-11-20T12:01:05.253 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:05 +0000 (0:00:00.137) 0:01:13.408 ******* 2020-11-20T12:01:05.474 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-11-20T12:01:05.508 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:05 +0000 (0:00:00.255) 0:01:13.663 ******* 2020-11-20T12:01:05.658 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-11-20T12:01:05.660 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-11-20T12:01:05.696 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:05 +0000 (0:00:00.188) 0:01:13.851 ******* 2020-11-20T12:01:05.852 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-11-20T12:01:05.854 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-20T12:01:05.888 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:05 +0000 (0:00:00.191) 0:01:14.043 ******* 2020-11-20T12:01:05.896 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-11-20T12:01:05.898 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-11-20T12:01:05.943 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:05 +0000 (0:00:00.054) 0:01:14.098 ******* 2020-11-20T12:01:05.950 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-11-20T12:01:05.953 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:05.990 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:05 +0000 (0:00:00.046) 0:01:14.144 ******* 2020-11-20T12:01:06.018 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-11-20T12:01:06.021 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:06.059 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:06 +0000 (0:00:00.069) 0:01:14.214 ******* 2020-11-20T12:01:06.088 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-11-20T12:01:06.090 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:06.134 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:06 +0000 (0:00:00.075) 0:01:14.289 ******* 2020-11-20T12:01:06.180 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:06 +0000 (0:00:00.045) 0:01:14.335 ******* 2020-11-20T12:01:06.188 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-11-20T12:01:06.190 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:06.229 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:06 +0000 (0:00:00.049) 0:01:14.384 ******* 2020-11-20T12:01:06.240 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:06.290 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:06 +0000 (0:00:00.060) 0:01:14.445 ******* 2020-11-20T12:01:06.297 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:06.356 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:06 +0000 (0:00:00.066) 0:01:14.511 ******* 2020-11-20T12:01:06.363 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-11-20T12:01:06.365 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:06.401 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:06 +0000 (0:00:00.044) 0:01:14.555 ******* 2020-11-20T12:01:06.407 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-11-20T12:01:06.409 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:06.446 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:06 +0000 (0:00:00.045) 0:01:14.601 ******* 2020-11-20T12:01:06.453 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T12:01:06.458 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-11-20T12:01:06.510 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:06 +0000 (0:00:00.063) 0:01:14.664 ******* 2020-11-20T12:01:06.521 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-11-20T12:01:06.523 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:06.569 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:06 +0000 (0:00:00.059) 0:01:14.723 ******* 2020-11-20T12:01:06.579 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-11-20T12:01:06.581 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:06.582 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:06.593 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:06.601 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:06.629 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi189.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:06.632 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} skipping: [smithi189.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:06.669 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:06 +0000 (0:00:00.100) 0:01:14.824 ******* 2020-11-20T12:01:06.677 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:06.680 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:06.690 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:06.692 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:06.746 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:06 +0000 (0:00:00.076) 0:01:14.900 ******* 2020-11-20T12:01:06.759 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi047.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-11-20T12:01:06.829 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:06 +0000 (0:00:00.075) 0:01:14.976 ******* 2020-11-20T12:01:06.882 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi047.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:06.884 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:06.894 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:06.921 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:06.930 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:06.955 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-11-20T12:01:06.978 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:06.989 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:07.002 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:07.013 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:07.025 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:07.038 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} TASK [testnode : Check for teuthology-lock command] **************************** 2020-11-20T12:01:07.039 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.009484", "end": "2020-11-20 12:01:07.004859", "rc": 0, "start": "2020-11-20 12:01:06.995375", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-11-20T12:01:07.091 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:07 +0000 (0:00:00.270) 0:01:15.246 ******* 2020-11-20T12:01:07.098 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-11-20T12:01:07.100 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:07.167 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:07 +0000 (0:00:00.073) 0:01:15.320 ******* 2020-11-20T12:01:07.179 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:07.218 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:07 +0000 (0:00:00.052) 0:01:15.372 ******* 2020-11-20T12:01:07.269 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:07.321 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:07 +0000 (0:00:00.103) 0:01:15.476 ******* 2020-11-20T12:01:07.351 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:07.394 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:07 +0000 (0:00:00.071) 0:01:15.548 ******* 2020-11-20T12:01:07.426 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-11-20T12:01:07.427 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.004041", "end": "2020-11-20 12:01:07.396274", "rc": 0, "start": "2020-11-20 12:01:07.392233", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-11-20T12:01:07.469 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:07 +0000 (0:00:00.076) 0:01:15.624 ******* 2020-11-20T12:01:07.508 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-11-20T12:01:07.510 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:07.551 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:07 +0000 (0:00:00.081) 0:01:15.706 ******* 2020-11-20T12:01:07.589 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-11-20T12:01:07.591 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:07.638 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:07 +0000 (0:00:00.086) 0:01:15.793 ******* 2020-11-20T12:01:07.666 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-11-20T12:01:07.667 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:07.702 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:07 +0000 (0:00:00.064) 0:01:15.857 ******* 2020-11-20T12:01:08.300 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* ok: [smithi189.front.sepia.ceph.com] => {"cache_update_time": 1605873376, "cache_updated": false, "changed": false} 2020-11-20T12:01:08.338 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:08 +0000 (0:00:00.636) 0:01:16.493 ******* 2020-11-20T12:01:08.367 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-11-20T12:01:08.369 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:08.409 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:08 +0000 (0:00:00.070) 0:01:16.564 ******* 2020-11-20T12:01:08.475 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-20T12:01:08.477 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item={'block_used': 13753484, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341624, 'size_available': 927095291904, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149530, 'block_size': 4096, 'inode_available': 60905446}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341624, "block_size": 4096, "block_total": 240095108, "block_used": 13753484, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905446, "inode_total": 61054976, "inode_used": 149530, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927095291904, "size_total": 983429562368, "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}} 2020-11-20T12:01:08.512 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:08 +0000 (0:00:00.103) 0:01:16.667 ******* 2020-11-20T12:01:08.625 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-11-20T12:01:08.628 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"cache_update_time": 1605873420, "cache_updated": false, "changed": false} 2020-11-20T12:01:08.668 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:08 +0000 (0:00:00.155) 0:01:16.823 ******* 2020-11-20T12:01:08.676 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-11-20T12:01:08.678 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009696", "end": "2020-11-20 12:01:08.623860", "rc": 0, "start": "2020-11-20 12:01:08.614164", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:08.717 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:08 +0000 (0:00:00.048) 0:01:16.872 ******* 2020-11-20T12:01:08.726 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-11-20T12:01:08.727 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:08.773 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:08 +0000 (0:00:00.055) 0:01:16.928 ******* 2020-11-20T12:01:08.781 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-11-20T12:01:08.783 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'block_used': 13753484, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341624, 'size_available': 927095291904, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149530, 'block_size': 4096, 'inode_available': 60905446}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341624, "block_size": 4096, "block_total": 240095108, "block_used": 13753484, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905446, "inode_total": 61054976, "inode_used": 149530, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927095291904, "size_total": 983429562368, "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:08.824 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:08 +0000 (0:00:00.050) 0:01:16.978 ******* 2020-11-20T12:01:08.832 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-20T12:01:08.834 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => (item={'block_used': 13753495, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341613, 'size_available': 927095246848, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149529, 'block_size': 4096, 'inode_available': 60905447}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341613, "block_size": 4096, "block_total": 240095108, "block_used": 13753495, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905447, "inode_total": 61054976, "inode_used": 149529, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927095246848, "size_total": 983429562368, "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}} 2020-11-20T12:01:08.875 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:08 +0000 (0:00:00.051) 0:01:17.030 ******* 2020-11-20T12:01:08.887 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-11-20T12:01:08.926 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:08 +0000 (0:00:00.050) 0:01:17.081 ******* 2020-11-20T12:01:08.976 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-11-20T12:01:08.978 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:09.020 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-11-20T12:01:09.021 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.010014", "end": "2020-11-20 12:01:08.874280", "rc": 0, "start": "2020-11-20 12:01:08.864266", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:09.059 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:09 +0000 (0:00:00.132) 0:01:17.214 ******* 2020-11-20T12:01:09.098 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-11-20T12:01:09.100 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'block_used': 13753495, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341613, 'size_available': 927095246848, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149529, 'block_size': 4096, 'inode_available': 60905447}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341613, "block_size": 4096, "block_total": 240095108, "block_used": 13753495, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905447, "inode_total": 61054976, "inode_used": 149529, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927095246848, "size_total": 983429562368, "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:09.148 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:09 +0000 (0:00:00.088) 0:01:17.303 ******* 2020-11-20T12:01:09.175 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-11-20T12:01:09.215 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:09 +0000 (0:00:00.067) 0:01:17.370 ******* 2020-11-20T12:01:09.284 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-11-20T12:01:09.286 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:11.214 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-NdZkdhX3FLndHpk0ssipV4UHaQhelR4Vj4ivRz0a2qMHBFmyN4ho6uQajg54fDeS'], 'uuids': ['c0f678d0-034d-422b-9347-87bee5b68dfa']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:02.117974", "end": "2020-11-20 12:01:11.186746", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-NdZkdhX3FLndHpk0ssipV4UHaQhelR4Vj4ivRz0a2qMHBFmyN4ho6uQajg54fDeS"], "labels": [], "masters": [], "uuids": ["c0f678d0-034d-422b-9347-87bee5b68dfa"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:09.068772", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T12:01:11.229 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:11.241 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:11.443 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-eM6LIoTUBBO6I4ywr5mZfhUK6tyUMpxA465yOYuN4Ug0nLFFCt3YVtzkuVwxxOBr'], 'uuids': ['2f0c97aa-443a-4e45-9d68-425256d1a1c3']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:02.042088", "end": "2020-11-20 12:01:11.300406", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-eM6LIoTUBBO6I4ywr5mZfhUK6tyUMpxA465yOYuN4Ug0nLFFCt3YVtzkuVwxxOBr"], "labels": [], "masters": [], "uuids": ["2f0c97aa-443a-4e45-9d68-425256d1a1c3"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:09.258318", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T12:01:11.455 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:11.467 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:12.507 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-NdZkdhX3FLndHpk0ssipV4UHaQhelR4VCGlpHxCnEDrIhjU4orSecwkarGDK1mnJ'], 'uuids': ['a14f22a6-599e-40b4-8ef3-d01bdbf9989c']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.123351", "end": "2020-11-20 12:01:12.482133", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-NdZkdhX3FLndHpk0ssipV4UHaQhelR4VCGlpHxCnEDrIhjU4orSecwkarGDK1mnJ"], "labels": [], "masters": [], "uuids": ["a14f22a6-599e-40b4-8ef3-d01bdbf9989c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:11.358782", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T12:01:12.519 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XKLY', 'wwn-0x5000c50092937477'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XKLY-part1', 'wwn-0x5000c50092937477-part1'], 'uuids': ['5b37dc69-d2db-44bf-9122-6cf42a1bab47']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092937477', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XKLY', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XKLY", "wwn-0x5000c50092937477"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XKLY-part1", "wwn-0x5000c50092937477-part1"], "labels": [], "masters": [], "uuids": ["5b37dc69-d2db-44bf-9122-6cf42a1bab47"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XKLY", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092937477"}}, "skip_reason": 2020-11-20T12:01:12.521 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-20T12:01:12.711 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-eM6LIoTUBBO6I4ywr5mZfhUK6tyUMpxAKlER41WqxQKcvTemasRAa5rCwm2iQ2Id'], 'uuids': ['405b87c8-7895-4061-8505-7e0ee1536a76']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.114538", "end": "2020-11-20 12:01:12.563521", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-eM6LIoTUBBO6I4ywr5mZfhUK6tyUMpxAKlER41WqxQKcvTemasRAa5rCwm2iQ2Id"], "labels": [], "masters": [], "uuids": ["405b87c8-7895-4061-8505-7e0ee1536a76"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:11.448983", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T12:01:12.723 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HN80', 'wwn-0x5000c500871ae812'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HN80-part1', 'wwn-0x5000c500871ae812-part1'], 'uuids': ['5b37dc69-d2db-44bf-9122-6cf42a1bab47']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871ae812', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4HN80', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HN80", "wwn-0x5000c500871ae812"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HN80-part1", "wwn-0x5000c500871ae812-part1"], "labels": [], "masters": [], "uuids": ["5b37dc69-d2db-44bf-9122-6cf42a1bab47"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HN80", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871ae812"}}, "skip_reason": 2020-11-20T12:01:12.724 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-20T12:01:13.848 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-O1mnQ4-Cd1b-1DXx-EJAb-PrW7-m0Ti-Y0ypWS', 'nvme-INTEL_SSDPEDMD400G4_CVFT6232004L400BGN', 'nvme-nvme.8086-43564654363233323030344c34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.199763", "end": "2020-11-20 12:01:13.818928", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-O1mnQ4-Cd1b-1DXx-EJAb-PrW7-m0Ti-Y0ypWS", "nvme-INTEL_SSDPEDMD400G4_CVFT6232004L400BGN", "nvme-nvme.8086-43564654363233323030344c34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:12.619165", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-11-20T12:01:13.850 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T12:01:13.861 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:13.872 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:14.011 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-6QZXsd-SmO8-Be3c-k9gP-QJf4-MCQB-uGcgkf', 'nvme-INTEL_SSDPED1D480GA_PHMB75130022480DGN', 'nvme-nvme.8086-50484d42373531333030323234383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPED1D480GA', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.168179", "end": "2020-11-20 12:01:13.869063", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-6QZXsd-SmO8-Be3c-k9gP-QJf4-MCQB-uGcgkf", "nvme-INTEL_SSDPED1D480GA_PHMB75130022480DGN", "nvme-nvme.8086-50484d42373531333030323234383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:12.700884", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now pa 2020-11-20T12:01:14.013 INFO:teuthology.task.ansible.out:rtition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T12:01:14.022 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:14.034 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:15.156 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-NdZkdhX3FLndHpk0ssipV4UHaQhelR4VEi95gtwxrnGQNLr51tt5sYFamBSLinxL'], 'uuids': ['a094f7cf-7e40-479b-ab82-3c1f5a441789']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.143712", "end": "2020-11-20 12:01:15.125495", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-NdZkdhX3FLndHpk0ssipV4UHaQhelR4VEi95gtwxrnGQNLr51tt5sYFamBSLinxL"], "labels": [], "masters": [], "uuids": ["a094f7cf-7e40-479b-ab82-3c1f5a441789"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:13.981783", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T12:01:15.167 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:15.178 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:15.305 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-eM6LIoTUBBO6I4ywr5mZfhUK6tyUMpxAYLEAr4izHB6958xkLii6k5IdI2DsN4M4'], 'uuids': ['5afda03c-4168-4b7a-8bf7-3f8d36aa03b0']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.154621", "end": "2020-11-20 12:01:15.157040", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-eM6LIoTUBBO6I4ywr5mZfhUK6tyUMpxAYLEAr4izHB6958xkLii6k5IdI2DsN4M4"], "labels": [], "masters": [], "uuids": ["5afda03c-4168-4b7a-8bf7-3f8d36aa03b0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:14.002419", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T12:01:15.317 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:15.319 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:16.440 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-NdZkdhX3FLndHpk0ssipV4UHaQhelR4Vj6jkYHnsL95DKgLx8ZFDFW0Jr2l8NLS6'], 'uuids': ['bc6fc9dd-feae-492d-929c-3b772b12dfbe']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.126786", "end": "2020-11-20 12:01:16.409479", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-NdZkdhX3FLndHpk0ssipV4UHaQhelR4Vj6jkYHnsL95DKgLx8ZFDFW0Jr2l8NLS6"], "labels": [], "masters": [], "uuids": ["bc6fc9dd-feae-492d-929c-3b772b12dfbe"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:15.282693", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T12:01:16.452 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:16.558 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-eM6LIoTUBBO6I4ywr5mZfhUK6tyUMpxAbUEB6rz9C28fXZbLX6RRDSW0jJUHytxX'], 'uuids': ['10ea0d73-af30-4a9e-9f7c-37d9dbb85d45']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.112386", "end": "2020-11-20 12:01:16.408104", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-eM6LIoTUBBO6I4ywr5mZfhUK6tyUMpxAbUEB6rz9C28fXZbLX6RRDSW0jJUHytxX"], "labels": [], "masters": [], "uuids": ["10ea0d73-af30-4a9e-9f7c-37d9dbb85d45"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:15.295718", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T12:01:16.569 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:17.699 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-NdZkdhX3FLndHpk0ssipV4UHaQhelR4VyUOSspN9tqPn24ppGcfkSQ4VKdMKULDe'], 'uuids': ['a11698aa-d11d-4326-bdbc-553a70239813']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.136111", "end": "2020-11-20 12:01:17.675420", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-NdZkdhX3FLndHpk0ssipV4UHaQhelR4VyUOSspN9tqPn24ppGcfkSQ4VKdMKULDe"], "labels": [], "masters": [], "uuids": ["a11698aa-d11d-4326-bdbc-553a70239813"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:16.539309", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T12:01:17.743 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:17 +0000 (0:00:08.528) 0:01:25.898 ******* 2020-11-20T12:01:17.802 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-20T12:01:17.804 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:17.825 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-11-20T12:01:17.827 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-eM6LIoTUBBO6I4ywr5mZfhUK6tyUMpxArnaRUGBYNbPfaiRbplHGtkBqV0zZn4k1'], 'uuids': ['92d0dccd-825f-4a76-8013-fa7b1883a106']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.117608", "end": "2020-11-20 12:01:17.676768", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-eM6LIoTUBBO6I4ywr5mZfhUK6tyUMpxArnaRUGBYNbPfaiRbplHGtkBqV0zZn4k1"], "labels": [], "masters": [], "uuids": ["92d0dccd-825f-4a76-8013-fa7b1883a106"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:16.559160", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T12:01:17.889 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:17 +0000 (0:00:00.145) 0:01:26.043 ******* 2020-11-20T12:01:17.939 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-20T12:01:17.941 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:18.098 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-NdZkdhX3FLndHpk0ssipV4UHaQhelR4Vj4ivRz0a2qMHBFmyN4ho6uQajg54fDeS'], 'uuids': ['c0f678d0-034d-422b-9347-87bee5b68dfa']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.156286", "end": "2020-11-20 12:01:18.068320", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-NdZkdhX3FLndHpk0ssipV4UHaQhelR4Vj4ivRz0a2qMHBFmyN4ho6uQajg54fDeS"], "labels": [], "masters": [], "uuids": ["c0f678d0-034d-422b-9347-87bee5b68dfa"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:17.912034", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.151532 s, 761 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.151532 s, 761 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:18.110 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:18.122 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:18.186 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-eM6LIoTUBBO6I4ywr5mZfhUK6tyUMpxA465yOYuN4Ug0nLFFCt3YVtzkuVwxxOBr'], 'uuids': ['2f0c97aa-443a-4e45-9d68-425256d1a1c3']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.111785", "end": "2020-11-20 12:01:18.041310", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-eM6LIoTUBBO6I4ywr5mZfhUK6tyUMpxA465yOYuN4Ug0nLFFCt3YVtzkuVwxxOBr"], "labels": [], "masters": [], "uuids": ["2f0c97aa-443a-4e45-9d68-425256d1a1c3"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:17.929525", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.107047 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.107047 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:18.198 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:18.202 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:18.407 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-NdZkdhX3FLndHpk0ssipV4UHaQhelR4VCGlpHxCnEDrIhjU4orSecwkarGDK1mnJ'], 'uuids': ['a14f22a6-599e-40b4-8ef3-d01bdbf9989c']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.159746", "end": "2020-11-20 12:01:18.381337", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-NdZkdhX3FLndHpk0ssipV4UHaQhelR4VCGlpHxCnEDrIhjU4orSecwkarGDK1mnJ"], "labels": [], "masters": [], "uuids": ["a14f22a6-599e-40b4-8ef3-d01bdbf9989c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:18.221591", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.15492 s, 745 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.15492 s, 745 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:18.421 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XKLY', 'wwn-0x5000c50092937477'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XKLY-part1', 'wwn-0x5000c50092937477-part1'], 'uuids': ['5b37dc69-d2db-44bf-9122-6cf42a1bab47']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092937477', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XKLY', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XKLY", "wwn-0x5000c50092937477"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XKLY-part1", "wwn-0x5000c50092937477-part1"], "labels": [], "masters": [], "uuids": ["5b37dc69-d2db-44bf-9122-6cf42a1bab47"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XKLY", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092937477"}}, "skip_reason": 2020-11-20T12:01:18.423 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-20T12:01:18.445 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-eM6LIoTUBBO6I4ywr5mZfhUK6tyUMpxAKlER41WqxQKcvTemasRAa5rCwm2iQ2Id'], 'uuids': ['405b87c8-7895-4061-8505-7e0ee1536a76']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.111345", "end": "2020-11-20 12:01:18.298756", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-eM6LIoTUBBO6I4ywr5mZfhUK6tyUMpxAKlER41WqxQKcvTemasRAa5rCwm2iQ2Id"], "labels": [], "masters": [], "uuids": ["405b87c8-7895-4061-8505-7e0ee1536a76"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:18.187411", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.106571 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.106571 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:18.457 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HN80', 'wwn-0x5000c500871ae812'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HN80-part1', 'wwn-0x5000c500871ae812-part1'], 'uuids': ['5b37dc69-d2db-44bf-9122-6cf42a1bab47']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871ae812', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4HN80', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HN80", "wwn-0x5000c500871ae812"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HN80-part1", "wwn-0x5000c500871ae812-part1"], "labels": [], "masters": [], "uuids": ["5b37dc69-d2db-44bf-9122-6cf42a1bab47"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HN80", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871ae812"}}, "skip_reason": 2020-11-20T12:01:18.459 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-11-20T12:01:18.605 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-O1mnQ4-Cd1b-1DXx-EJAb-PrW7-m0Ti-Y0ypWS', 'nvme-INTEL_SSDPEDMD400G4_CVFT6232004L400BGN', 'nvme-nvme.8086-43564654363233323030344c34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.054312", "end": "2020-11-20 12:01:18.570421", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-O1mnQ4-Cd1b-1DXx-EJAb-PrW7-m0Ti-Y0ypWS", "nvme-INTEL_SSDPEDMD400G4_CVFT6232004L400BGN", "nvme-nvme.8086-43564654363233323030344c34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:18.516109", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0495707 s, 2.3 GB/s", "std 2020-11-20T12:01:18.608 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0495707 s, 2.3 GB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:18.618 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:18.640 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-6QZXsd-SmO8-Be3c-k9gP-QJf4-MCQB-uGcgkf', 'nvme-INTEL_SSDPED1D480GA_PHMB75130022480DGN', 'nvme-nvme.8086-50484d42373531333030323234383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPED1D480GA', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.050172", "end": "2020-11-20 12:01:18.492978", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-6QZXsd-SmO8-Be3c-k9gP-QJf4-MCQB-uGcgkf", "nvme-INTEL_SSDPED1D480GA_PHMB75130022480DGN", "nvme-nvme.8086-50484d42373531333030323234383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:18.442806", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.046021 s, 2.5 GB/s", "stderr_lines": ["1 2020-11-20T12:01:18.643 INFO:teuthology.task.ansible.out:10+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.046021 s, 2.5 GB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:18.663 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:18.677 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:18.887 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-NdZkdhX3FLndHpk0ssipV4UHaQhelR4VEi95gtwxrnGQNLr51tt5sYFamBSLinxL'], 'uuids': ['a094f7cf-7e40-479b-ab82-3c1f5a441789']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.151032", "end": "2020-11-20 12:01:18.861144", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-NdZkdhX3FLndHpk0ssipV4UHaQhelR4VEi95gtwxrnGQNLr51tt5sYFamBSLinxL"], "labels": [], "masters": [], "uuids": ["a094f7cf-7e40-479b-ab82-3c1f5a441789"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:18.710112", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148653 s, 776 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148653 s, 776 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:18.889 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-eM6LIoTUBBO6I4ywr5mZfhUK6tyUMpxAYLEAr4izHB6958xkLii6k5IdI2DsN4M4'], 'uuids': ['5afda03c-4168-4b7a-8bf7-3f8d36aa03b0']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.107615", "end": "2020-11-20 12:01:18.747599", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-eM6LIoTUBBO6I4ywr5mZfhUK6tyUMpxAYLEAr4izHB6958xkLii6k5IdI2DsN4M4"], "labels": [], "masters": [], "uuids": ["5afda03c-4168-4b7a-8bf7-3f8d36aa03b0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:18.639984", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.102637 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.102637 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:18.900 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:18.901 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:18.903 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:18.914 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:19.157 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-eM6LIoTUBBO6I4ywr5mZfhUK6tyUMpxAbUEB6rz9C28fXZbLX6RRDSW0jJUHytxX'], 'uuids': ['10ea0d73-af30-4a9e-9f7c-37d9dbb85d45']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.120007", "end": "2020-11-20 12:01:19.016651", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-eM6LIoTUBBO6I4ywr5mZfhUK6tyUMpxAbUEB6rz9C28fXZbLX6RRDSW0jJUHytxX"], "labels": [], "masters": [], "uuids": ["10ea0d73-af30-4a9e-9f7c-37d9dbb85d45"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:18.896644", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.115052 s, 1.0 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.115052 s, 1.0 GB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:19.169 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:19.204 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-NdZkdhX3FLndHpk0ssipV4UHaQhelR4Vj6jkYHnsL95DKgLx8ZFDFW0Jr2l8NLS6'], 'uuids': ['bc6fc9dd-feae-492d-929c-3b772b12dfbe']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.162825", "end": "2020-11-20 12:01:19.169330", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-NdZkdhX3FLndHpk0ssipV4UHaQhelR4Vj6jkYHnsL95DKgLx8ZFDFW0Jr2l8NLS6"], "labels": [], "masters": [], "uuids": ["bc6fc9dd-feae-492d-929c-3b772b12dfbe"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:19.006505", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.158019 s, 730 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.158019 s, 730 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:19.216 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:19.398 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-eM6LIoTUBBO6I4ywr5mZfhUK6tyUMpxArnaRUGBYNbPfaiRbplHGtkBqV0zZn4k1'], 'uuids': ['92d0dccd-825f-4a76-8013-fa7b1883a106']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.112436", "end": "2020-11-20 12:01:19.255161", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-eM6LIoTUBBO6I4ywr5mZfhUK6tyUMpxArnaRUGBYNbPfaiRbplHGtkBqV0zZn4k1"], "labels": [], "masters": [], "uuids": ["92d0dccd-825f-4a76-8013-fa7b1883a106"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:19.142725", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.107713 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.107713 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:19.450 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:19 +0000 (0:00:01.561) 0:01:27.605 ******* 2020-11-20T12:01:19.489 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-NdZkdhX3FLndHpk0ssipV4UHaQhelR4VyUOSspN9tqPn24ppGcfkSQ4VKdMKULDe'], 'uuids': ['a11698aa-d11d-4326-bdbc-553a70239813']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.160235", "end": "2020-11-20 12:01:19.463760", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-NdZkdhX3FLndHpk0ssipV4UHaQhelR4VyUOSspN9tqPn24ppGcfkSQ4VKdMKULDe"], "labels": [], "masters": [], "uuids": ["a11698aa-d11d-4326-bdbc-553a70239813"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 12:01:19.303525", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.155552 s, 742 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.155552 s, 742 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:19.537 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:19 +0000 (0:00:00.086) 0:01:27.692 ******* 2020-11-20T12:01:19.723 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-11-20T12:01:19.725 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.142055", "end": "2020-11-20 12:01:19.580946", "rc": 0, "start": "2020-11-20 12:01:19.438891", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:19.764 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:19 +0000 (0:00:00.226) 0:01:27.919 ******* 2020-11-20T12:01:19.803 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.128746", "end": "2020-11-20 12:01:19.774933", "rc": 0, "start": "2020-11-20 12:01:19.646187", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:19.839 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:19 +0000 (0:00:00.075) 0:01:27.994 ******* 2020-11-20T12:01:19.961 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-11-20T12:01:19.963 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.061628", "end": "2020-11-20 12:01:19.818773", "rc": 0, "start": "2020-11-20 12:01:19.757145", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:20.001 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:19 +0000 (0:00:00.162) 0:01:28.156 ******* 2020-11-20T12:01:20.029 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-11-20T12:01:20.069 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.067) 0:01:28.224 ******* 2020-11-20T12:01:20.095 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-11-20T12:01:20.097 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check for physical volumes] *********************************** changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.130764", "end": "2020-11-20 12:01:20.077225", "rc": 0, "start": "2020-11-20 12:01:19.946461", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:20.134 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.065) 0:01:28.289 ******* 2020-11-20T12:01:20.172 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.037) 0:01:28.327 ******* 2020-11-20T12:01:20.178 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-11-20T12:01:20.180 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:20.215 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.042) 0:01:28.370 ******* 2020-11-20T12:01:20.221 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-11-20T12:01:20.261 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.046) 0:01:28.416 ******* 2020-11-20T12:01:20.267 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-20T12:01:20.268 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:20.312 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.047) 0:01:28.463 ******* 2020-11-20T12:01:20.315 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-11-20T12:01:20.316 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:20.353 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.044) 0:01:28.508 ******* 2020-11-20T12:01:20.359 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-11-20T12:01:20.361 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:20.395 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.042) 0:01:28.550 ******* 2020-11-20T12:01:20.402 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-11-20T12:01:20.403 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:20.439 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.043) 0:01:28.594 ******* 2020-11-20T12:01:20.466 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-20T12:01:20.468 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:20.504 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.064) 0:01:28.658 ******* 2020-11-20T12:01:20.520 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-11-20T12:01:20.522 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-20T12:01:20.565 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.061) 0:01:28.720 ******* 2020-11-20T12:01:20.571 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-11-20T12:01:20.573 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:20.613 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.047) 0:01:28.767 ******* 2020-11-20T12:01:20.619 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-20T12:01:20.624 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'block_used': 13753495, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341613, 'size_available': 927095246848, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149529, 'block_size': 4096, 'inode_available': 60905447}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341613, "block_size": 4096, "block_total": 240095108, "block_used": 13753495, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905447, "inode_total": 61054976, "inode_used": 149529, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927095246848, "size_total": 983429562368, "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:20.664 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.051) 0:01:28.819 ******* 2020-11-20T12:01:20.692 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-11-20T12:01:20.694 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:20.727 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.063) 0:01:28.882 ******* 2020-11-20T12:01:20.735 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-11-20T12:01:20.737 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-20T12:01:20.772 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.044) 0:01:28.927 ******* 2020-11-20T12:01:20.778 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T12:01:20.780 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:20.816 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.043) 0:01:28.971 ******* 2020-11-20T12:01:20.822 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-20T12:01:20.823 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'block_used': 13753484, 'uuid': '5b37dc69-d2db-44bf-9122-6cf42a1bab47', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341624, 'size_available': 927095291904, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149530, 'block_size': 4096, 'inode_available': 60905446}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341624, "block_size": 4096, "block_total": 240095108, "block_used": 13753484, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905446, "inode_total": 61054976, "inode_used": 149530, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927095291904, "size_total": 983429562368, "uuid": "5b37dc69-d2db-44bf-9122-6cf42a1bab47"}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:20.860 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.043) 0:01:29.014 ******* 2020-11-20T12:01:20.866 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:20.902 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.042) 0:01:29.057 ******* 2020-11-20T12:01:20.909 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-11-20T12:01:20.910 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:20.945 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.043) 0:01:29.100 ******* 2020-11-20T12:01:20.952 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-11-20T12:01:20.953 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:20.988 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:20 +0000 (0:00:00.042) 0:01:29.143 ******* 2020-11-20T12:01:20.995 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T12:01:20.997 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:21.032 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:21 +0000 (0:00:00.043) 0:01:29.186 ******* 2020-11-20T12:01:21.038 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-11-20T12:01:21.040 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:21.075 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:21 +0000 (0:00:00.043) 0:01:29.230 ******* 2020-11-20T12:01:21.083 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:21.119 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:21 +0000 (0:00:00.043) 0:01:29.274 ******* 2020-11-20T12:01:21.125 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-11-20T12:01:21.127 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:21.163 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:21 +0000 (0:00:00.043) 0:01:29.317 ******* 2020-11-20T12:01:21.169 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-11-20T12:01:21.170 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:21.210 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:21 +0000 (0:00:00.047) 0:01:29.364 ******* 2020-11-20T12:01:21.238 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:21.274 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:21 +0000 (0:00:00.063) 0:01:29.428 ******* 2020-11-20T12:01:21.300 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-11-20T12:01:21.303 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:21.337 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:21 +0000 (0:00:00.062) 0:01:29.491 ******* 2020-11-20T12:01:21.685 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-11-20T12:01:21.687 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-11-20T12:01:21.723 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:21 +0000 (0:00:00.386) 0:01:29.878 ******* 2020-11-20T12:01:21.763 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-11-20T12:01:21.799 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:21 +0000 (0:00:00.075) 0:01:29.954 ******* 2020-11-20T12:01:22.262 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-11-20T12:01:22.263 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T12:01:22.294 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T12:01:22.494 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T12:01:22.548 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T12:01:22.716 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T12:01:22.780 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T12:01:22.949 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T12:01:23.018 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T12:01:23.224 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T12:01:25.257 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:23 +0000 (0:00:01.462) 0:01:31.417 ******* changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} Friday 20 November 2020 12:01:23 +0000 (0:00:00.070) 0:01:31.488 ******* TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** changed: [smithi047.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} Friday 20 November 2020 12:01:23 +0000 (0:00:00.131) 0:01:31.619 ******* changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} Friday 20 November 2020 12:01:23 +0000 (0:00:00.043) 0:01:31.663 ******* TASK [testnode : Write /scratch_devs] ****************************************** changed: [smithi047.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} changed: [smithi047.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'sc 2020-11-20T12:01:25.259 INFO:teuthology.task.ansible.out:ratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} changed: [smithi047.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} changed: [smithi189.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} changed: [smithi047.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} skipping: [smithi189.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:25.261 INFO:teuthology.task.ansible.out: Friday 20 November 2020 12:01:24 +0000 (0:00:00.545) 0:01:32.209 ******* Friday 20 November 2020 12:01:24 +0000 (0:00:00.045) 0:01:32.254 ******* TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} Friday 20 November 2020 12:01:24 +0000 (0:00:00.121) 0:01:32.376 ******* ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} Friday 20 November 2020 12:01:24 +0000 (0:00:00.054) 0:01:32.431 ******* TASK [testnode : Set xfs_opts on newer OSes] *********************************** ok: [smithi189.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} Friday 20 November 2020 12:01:24 +0000 (0:00:00.045) 0:01:32.477 ******* ok: [smithi047.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} Friday 20 November 2020 12:01:24 +0000 (0:00:00.043) 0:01:32.521 ******* TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi047.front.sepia.ceph.com] => {"changed": true} Friday 20 November 2020 12:01:24 +0000 (0:00:00.402) 0:01:32.923 ******* changed: [smithi189.front.sepia.ceph.com] => {"changed": true} Friday 20 November 2020 12:01:24 +0000 (0:00:00.066) 0:01:32.990 ******* TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* changed: [smithi047.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} Friday 20 November 2020 12:01:25 +0000 (0:00:00.415) 2020-11-20T12:01:25.262 INFO:teuthology.task.ansible.out:0:01:33.406 ******* 2020-11-20T12:01:25.280 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-11-20T12:01:25.315 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:25 +0000 (0:00:00.063) 0:01:33.469 ******* 2020-11-20T12:01:25.383 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-11-20T12:01:25.385 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-11-20T12:01:25.420 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:25 +0000 (0:00:00.105) 0:01:33.575 ******* 2020-11-20T12:01:25.458 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-11-20T12:01:25.495 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:25 +0000 (0:00:00.074) 0:01:33.649 ******* 2020-11-20T12:01:25.553 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-11-20T12:01:25.554 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-11-20T12:01:25.589 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:25 +0000 (0:00:00.094) 0:01:33.744 ******* 2020-11-20T12:01:25.627 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-11-20T12:01:25.665 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:25 +0000 (0:00:00.075) 0:01:33.820 ******* 2020-11-20T12:01:25.912 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-11-20T12:01:25.914 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1236, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-11-20T12:01:25.967 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:25 +0000 (0:00:00.302) 0:01:34.122 ******* 2020-11-20T12:01:25.985 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1236, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-11-20T12:01:26.022 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:26 +0000 (0:00:00.054) 0:01:34.177 ******* 2020-11-20T12:01:26.292 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-11-20T12:01:26.294 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-11-20T12:01:26.351 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:26 +0000 (0:00:00.329) 0:01:34.506 ******* 2020-11-20T12:01:26.359 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-11-20T12:01:26.397 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:26 +0000 (0:00:00.045) 0:01:34.552 ******* 2020-11-20T12:01:27.254 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-11-20T12:01:27.256 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"cache_update_time": 1605873376, "cache_updated": false, "changed": false} 2020-11-20T12:01:27.292 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:27 +0000 (0:00:00.894) 0:01:35.446 ******* 2020-11-20T12:01:27.297 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"cache_update_time": 1605873420, "cache_updated": false, "changed": false} 2020-11-20T12:01:27.334 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:27 +0000 (0:00:00.042) 0:01:35.489 ******* 2020-11-20T12:01:27.455 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-11-20T12:01:27.457 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.043174", "end": "2020-11-20 12:01:27.434827", "rc": 0, "start": "2020-11-20 12:01:27.391653", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-11-20T12:01:27.493 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:27 +0000 (0:00:00.159) 0:01:35.648 ******* 2020-11-20T12:01:27.521 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.048847", "end": "2020-11-20 12:01:27.383989", "rc": 0, "start": "2020-11-20 12:01:27.335142", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-11-20T12:01:27.558 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:27 +0000 (0:00:00.063) 0:01:35.712 ******* 2020-11-20T12:01:27.564 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:27.605 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:27 +0000 (0:00:00.042) 0:01:35.755 ******* 2020-11-20T12:01:27.609 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:27.645 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:27 +0000 (0:00:00.043) 0:01:35.798 ******* 2020-11-20T12:01:27.651 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:27.687 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:27 +0000 (0:00:00.043) 0:01:35.842 ******* 2020-11-20T12:01:27.693 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:27.733 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:27 +0000 (0:00:00.046) 0:01:35.888 ******* 2020-11-20T12:01:28.498 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-11-20T12:01:28.499 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"cache_update_time": 1605873376, "cache_updated": false, "changed": false} 2020-11-20T12:01:28.536 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:28 +0000 (0:00:00.803) 0:01:36.691 ******* 2020-11-20T12:01:28.563 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-11-20T12:01:28.566 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:28.603 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:28 +0000 (0:00:00.066) 0:01:36.758 ******* 2020-11-20T12:01:28.620 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-11-20T12:01:28.623 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"cache_update_time": 1605873420, "cache_updated": false, "changed": false} 2020-11-20T12:01:28.675 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:28 +0000 (0:00:00.071) 0:01:36.830 ******* 2020-11-20T12:01:28.705 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:28.743 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:28 +0000 (0:00:00.068) 0:01:36.898 ******* 2020-11-20T12:01:28.909 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-11-20T12:01:28.945 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:28 +0000 (0:00:00.202) 0:01:37.100 ******* 2020-11-20T12:01:28.972 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:29.008 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:29 +0000 (0:00:00.062) 0:01:37.163 ******* 2020-11-20T12:01:29.057 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-11-20T12:01:29.059 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-11-20T12:01:29.097 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:29 +0000 (0:00:00.088) 0:01:37.251 ******* 2020-11-20T12:01:29.125 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:29.162 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:29 +0000 (0:00:00.064) 0:01:37.316 ******* 2020-11-20T12:01:29.200 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-11-20T12:01:29.201 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2020-11-20 11:59:43 UTC", "ActiveEnterTimestampMonotonic": "26506360", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target sysinit.target network.target -.mount systemd-tmpfiles-setup.service systemd-journald.socket system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2020-11-20 11:59:36 UTC", "AssertTimestampMonotonic": "19932217", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2020-11-20 11:59:36 UTC", "ConditionTimestampMonotonic": "19932217", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-11-20T12:01:29.203 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1667", "ExecMainStartTimestamp": "Fri 2020-11-20 11:59:43 UTC", "ExecMainStartTimestampMonotonic": "26506295", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2020-11-20 11:59:36 UTC", "InactiveExitTimestampMonotonic": "19933082", "InvocationID": "10b5f430cc884e3ba45aa7c14cdfd720", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-11-20T12:01:29.207 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1667", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2020-11-20 11:59:43 UTC", "S 2020-11-20T12:01:29.209 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "26506360", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2020-11-20 11:59:43 UTC", "WatchdogTimestampMonotonic": "26506356", "WatchdogUSec": "0"}} 2020-11-20T12:01:29.256 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:29 +0000 (0:00:00.094) 0:01:37.410 ******* 2020-11-20T12:01:29.285 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T12:01:29.287 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-11-20T12:01:29.332 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:29 +0000 (0:00:00.075) 0:01:37.486 ******* 2020-11-20T12:01:29.359 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T12:01:29.360 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:29.399 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:29 +0000 (0:00:00.066) 0:01:37.553 ******* 2020-11-20T12:01:29.416 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-11-20T12:01:29.417 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2020-11-20 11:59:29 UTC", "ActiveEnterTimestampMonotonic": "24290000", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket system.slice network.target systemd-tmpfiles-setup.service sysinit.target basic.target -.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2020-11-20 11:59:22 UTC", "AssertTimestampMonotonic": "17748127", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2020-11-20 11:59:22 UTC", "ConditionTimestampMonotonic": "17748127", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-11-20T12:01:29.419 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1862", "ExecMainStartTimestamp": "Fri 2020-11-20 11:59:29 UTC", "ExecMainStartTimestampMonotonic": "24289987", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2020-11-20 11:59:22 UTC", "InactiveExitTimestampMonotonic": "17748887", "InvocationID": "7732c554e1074cebba7b896598132ba7", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-11-20T12:01:29.422 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1862", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2020-11-20 11:59:29 UTC", "S 2020-11-20T12:01:29.424 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "24290000", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2020-11-20 11:59:29 UTC", "WatchdogTimestampMonotonic": "24289999", "WatchdogUSec": "0"}} 2020-11-20T12:01:29.455 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:29 +0000 (0:00:00.056) 0:01:37.610 ******* 2020-11-20T12:01:29.462 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-11-20T12:01:29.463 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:29.498 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:29 +0000 (0:00:00.043) 0:01:37.653 ******* 2020-11-20T12:01:29.504 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T12:01:29.506 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-11-20T12:01:29.544 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:29 +0000 (0:00:00.045) 0:01:37.699 ******* 2020-11-20T12:01:29.571 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T12:01:29.573 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:29.607 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:29 +0000 (0:00:00.063) 0:01:37.762 ******* 2020-11-20T12:01:29.634 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-11-20T12:01:29.636 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:29.669 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:29 +0000 (0:00:00.062) 0:01:37.824 ******* 2020-11-20T12:01:30.347 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-11-20T12:01:30.352 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"cache_update_time": 1605873376, "cache_updated": false, "changed": false} 2020-11-20T12:01:30.383 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:30 +0000 (0:00:00.714) 0:01:38.538 ******* 2020-11-20T12:01:30.410 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-11-20T12:01:30.412 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:30.447 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:30 +0000 (0:00:00.062) 0:01:38.601 ******* 2020-11-20T12:01:30.560 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* ok: [smithi047.front.sepia.ceph.com] => {"cache_update_time": 1605873420, "cache_updated": false, "changed": false} 2020-11-20T12:01:30.604 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:30 +0000 (0:00:00.150) 0:01:38.752 ******* 2020-11-20T12:01:30.606 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-11-20T12:01:30.651 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:30 +0000 (0:00:00.053) 0:01:38.806 ******* 2020-11-20T12:01:30.657 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-11-20T12:01:30.658 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:30.697 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:30 +0000 (0:00:00.046) 0:01:38.852 ******* 2020-11-20T12:01:30.830 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-11-20T12:01:30.832 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-11-20T12:01:30.868 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:30 +0000 (0:00:00.168) 0:01:39.020 ******* 2020-11-20T12:01:31.194 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-11-20T12:01:31.195 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-11-20T12:01:31.247 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:31 +0000 (0:00:00.380) 0:01:39.401 ******* 2020-11-20T12:01:31.256 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-11-20T12:01:31.292 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:31 +0000 (0:00:00.045) 0:01:39.447 ******* 2020-11-20T12:01:32.257 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-11-20T12:01:32.258 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-11-20T12:01:32.294 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:32 +0000 (0:00:01.000) 0:01:40.448 ******* 2020-11-20T12:01:32.299 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-11-20T12:01:32.347 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:32 +0000 (0:00:00.053) 0:01:40.502 ******* 2020-11-20T12:01:32.818 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-11-20T12:01:32.820 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"changed": false} 2020-11-20T12:01:32.855 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:32 +0000 (0:00:00.508) 0:01:41.010 ******* 2020-11-20T12:01:32.881 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-11-20T12:01:32.883 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} TASK [testnode : Purge resolvconf] ********************************************* ok: [smithi047.front.sepia.ceph.com] => {"changed": false} 2020-11-20T12:01:32.917 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:32 +0000 (0:00:00.062) 0:01:41.072 ******* 2020-11-20T12:01:32.950 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:32 +0000 (0:00:00.032) 0:01:41.105 ******* 2020-11-20T12:01:32.956 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-11-20T12:01:32.958 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2020-11-20T12:01:32.993 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:32 +0000 (0:00:00.043) 0:01:41.148 ******* 2020-11-20T12:01:33.087 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1605873357.0600798, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1605873357.0560799, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 2817, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1605873357.0560799, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "2282523401", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-11-20T12:01:33.124 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:33 +0000 (0:00:00.130) 0:01:41.279 ******* 2020-11-20T12:01:33.131 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1605873312.06876, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1605873312.0647578, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 2817, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1605873312.0647578, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 67, "uid": 0, "version": "2282523401", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-11-20T12:01:33.167 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:33 +0000 (0:00:00.043) 0:01:41.322 ******* 2020-11-20T12:01:33.379 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-11-20T12:01:33.380 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"changed": false, "msg": ""} ok: [smithi189.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-11-20T12:01:33.451 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:33 +0000 (0:00:00.283) 0:01:41.605 ******* 2020-11-20T12:01:33.485 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:33 +0000 (0:00:00.034) 0:01:41.640 ******* 2020-11-20T12:01:33.492 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-11-20T12:01:33.494 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-11-20T12:01:33.529 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:33 +0000 (0:00:00.044) 0:01:41.684 ******* 2020-11-20T12:01:33.536 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-11-20T12:01:33.573 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:33 +0000 (0:00:00.044) 0:01:41.728 ******* 2020-11-20T12:01:33.579 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-11-20T12:01:33.581 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:33.616 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:33 +0000 (0:00:00.042) 0:01:41.771 ******* 2020-11-20T12:01:33.622 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:33.659 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:33 +0000 (0:00:00.043) 0:01:41.814 ******* 2020-11-20T12:01:33.752 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-11-20T12:01:33.754 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-11-20T12:01:33.790 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:33 +0000 (0:00:00.130) 0:01:41.945 ******* 2020-11-20T12:01:33.798 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-11-20T12:01:33.854 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:33 +0000 (0:00:00.063) 0:01:42.009 ******* 2020-11-20T12:01:33.938 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-11-20T12:01:33.940 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-11-20T12:01:33.975 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:33 +0000 (0:00:00.121) 0:01:42.130 ******* 2020-11-20T12:01:34.002 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:34.039 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:34 +0000 (0:00:00.063) 0:01:42.193 ******* 2020-11-20T12:01:34.045 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-11-20T12:01:34.047 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-11-20T12:01:34.084 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:34 +0000 (0:00:00.045) 0:01:42.239 ******* 2020-11-20T12:01:34.092 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-11-20T12:01:34.093 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:34.095 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:34.096 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:34.109 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-11-20T12:01:34.111 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:34.163 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:34 +0000 (0:00:00.078) 0:01:42.318 ******* 2020-11-20T12:01:34.208 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:34 +0000 (0:00:00.044) 0:01:42.363 ******* 2020-11-20T12:01:34.215 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-11-20T12:01:34.217 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:34.227 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:34.229 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi047.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:34.239 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:34.241 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi189.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:34.242 INFO:teuthology.task.ansible.out: 2020-11-20T12:01:34.287 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:34 +0000 (0:00:00.079) 0:01:42.442 ******* 2020-11-20T12:01:34.293 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:34.298 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2020-11-20T12:01:34.334 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:34 +0000 (0:00:00.047) 0:01:42.489 ******* 2020-11-20T12:01:34.341 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T12:01:34.343 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:34.395 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:34 +0000 (0:00:00.061) 0:01:42.550 ******* 2020-11-20T12:01:34.403 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-11-20T12:01:34.404 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:34.406 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi189.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:34.454 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:34 +0000 (0:00:00.058) 0:01:42.609 ******* 2020-11-20T12:01:34.462 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-11-20T12:01:34.463 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:34.510 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:34 +0000 (0:00:00.056) 0:01:42.665 ******* 2020-11-20T12:01:34.518 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T12:01:34.520 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:34.565 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:34 +0000 (0:00:00.054) 0:01:42.720 ******* 2020-11-20T12:01:34.572 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-11-20T12:01:34.573 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:34.619 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:34 +0000 (0:00:00.053) 0:01:42.773 ******* 2020-11-20T12:01:34.626 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:34.670 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:34 +0000 (0:00:00.051) 0:01:42.825 ******* 2020-11-20T12:01:34.677 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-11-20T12:01:34.679 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:34.716 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:34 +0000 (0:00:00.046) 0:01:42.871 ******* 2020-11-20T12:01:34.724 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:34.772 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:34 +0000 (0:00:00.055) 0:01:42.927 ******* 2020-11-20T12:01:34.780 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:34.829 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:34 +0000 (0:00:00.057) 0:01:42.984 ******* 2020-11-20T12:01:34.836 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-11-20T12:01:34.838 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:34.886 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:34 +0000 (0:00:00.056) 0:01:43.041 ******* 2020-11-20T12:01:34.893 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-11-20T12:01:34.896 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:34.931 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:34 +0000 (0:00:00.045) 0:01:43.086 ******* 2020-11-20T12:01:34.938 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-11-20T12:01:34.939 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:34.976 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:34 +0000 (0:00:00.045) 0:01:43.131 ******* 2020-11-20T12:01:34.983 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-11-20T12:01:34.985 INFO:teuthology.task.ansible.out:skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:35.021 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:35 +0000 (0:00:00.044) 0:01:43.176 ******* 2020-11-20T12:01:35.027 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:35.066 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:35 +0000 (0:00:00.044) 0:01:43.220 ******* 2020-11-20T12:01:35.072 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi047.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:35.108 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:35 +0000 (0:00:00.042) 0:01:43.263 ******* 2020-11-20T12:01:35.115 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:35.154 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:35 +0000 (0:00:00.045) 0:01:43.308 ******* 2020-11-20T12:01:35.180 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-11-20T12:01:35.182 INFO:teuthology.task.ansible.out:skipping: [smithi189.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T12:01:35.235 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:35 +0000 (0:00:00.081) 0:01:43.390 ******* 2020-11-20T12:01:35.265 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-11-20T12:01:35.267 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011294", "end": "2020-11-20 12:01:35.129588", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-11-20 12:01:35.118294", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:35.386 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.005972", "end": "2020-11-20 12:01:35.252217", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-11-20 12:01:35.246245", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:35.397 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011384", "end": "2020-11-20 12:01:35.363312", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-11-20 12:01:35.351928", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:35.508 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005562", "end": "2020-11-20 12:01:35.373125", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-11-20 12:01:35.367563", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:35.528 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.005110", "end": "2020-11-20 12:01:35.491404", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-11-20 12:01:35.486294", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:35.643 INFO:teuthology.task.ansible.out:ok: [smithi189.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005607", "end": "2020-11-20 12:01:35.615478", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-11-20 12:01:35.609871", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:35.654 INFO:teuthology.task.ansible.out:ok: [smithi047.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.033685", "end": "2020-11-20 12:01:35.519408", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-11-20 12:01:35.485723", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:35.693 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:35 +0000 (0:00:00.458) 0:01:43.848 ******* 2020-11-20T12:01:35.721 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-11-20T12:01:35.758 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:35 +0000 (0:00:00.064) 0:01:43.913 ******* 2020-11-20T12:01:35.788 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi189.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.026180", "end": "2020-11-20 12:01:35.754839", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-11-20 12:01:35.728659", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T12:01:35.828 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:35 +0000 (0:00:00.070) 0:01:43.983 ******* 2020-11-20T12:01:35.855 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-11-20T12:01:35.893 INFO:teuthology.task.ansible.out:Friday 20 November 2020 12:01:35 +0000 (0:00:00.065) 0:01:44.048 ******* 2020-11-20T12:01:35.900 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-11-20T12:01:35.902 INFO:teuthology.task.ansible.out:changed: [smithi047.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-11-20T12:01:36.015 INFO:teuthology.task.ansible.out:changed: [smithi189.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-11-20T12:01:36.114 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-11-20T12:01:36.118 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2020-11-20T12:01:36.121 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2020-11-20T12:01:36.123 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi047.front.sepia.ceph.com : ok=114 changed=30 unreachable=0 failed=0 skipped=182 rescued=0 ignored=1 smithi189.front.sepia.ceph.com : ok=114 changed=30 unreachable=0 failed=0 skipped=182 rescued=0 ignored=1 Friday 20 November 2020 12:01:36 +0000 (0:00:00.228) 0:01:44.277 ******* =============================================================================== testnode : Ensure packages are not present. ----------------------------- 8.86s testnode : Install packages --------------------------------------------- 8.55s testnode : Zap all non-root disks --------------------------------------- 8.53s common : Check for docker ----------------------------------------------- 4.11s common : Update apt cache ----------------------------------------------- 3.56s testnode : Update apt cache. -------------------------------------------- 2.80s testnode : Install apt keys --------------------------------------------- 2.45s Gathering Facts --------------------------------------------------------- 1.77s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.56s testnode : Create logical volume(s) ------------------------------------- 1.46s testnode : Install packages via pip ------------------------------------- 1.00s testnode : Install python-apt ------------------------------------------- 0.98s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 0.94s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 0.93s common : Install nrpe package and dependencies (Ubuntu/Fedora) ---------- 0.91s testnode : Ensure perl-doc is installed on apt systems. ----------------- 0.89s common : Install Kerberos Packages (Debian) ----------------------------- 0.88s common : Make sure smartmontools is installed --------------------------- 0.87s Gathering Facts 2020-11-20T12:01:36.125 INFO:teuthology.task.ansible.out: --------------------------------------------------------- 0.86s testnode : Install ntp package on apt based systems. -------------------- 0.80s 2020-11-20T12:01:36.474 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi189.front.sepia.ceph.com'), Remote(name='ubuntu@smithi047.front.sepia.ceph.com')] 2020-11-20T12:01:36.475 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-20T12:01:36.476 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T12:01:36.555 INFO:teuthology.orchestra.run.smithi189:> true 2020-11-20T12:01:36.754 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-20T12:01:36.755 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T12:01:36.836 INFO:teuthology.orchestra.run.smithi047:> true 2020-11-20T12:01:37.034 INFO:teuthology.run_tasks:Running task clock... 2020-11-20T12:01:37.044 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-11-20T12:01:37.044 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-11-20T12:01:37.044 INFO:teuthology.orchestra.run.smithi189:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-11-20T12:01:37.103 INFO:teuthology.orchestra.run.smithi189.stdout:20 Nov 12:01:37 ntpd[5858]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-11-20T12:01:37.104 INFO:teuthology.orchestra.run.smithi189.stdout:20 Nov 12:01:37 ntpd[5858]: Command line: ntpd -gq 2020-11-20T12:01:37.105 INFO:teuthology.orchestra.run.smithi189.stdout:20 Nov 12:01:37 ntpd[5858]: proto: precision = 0.120 usec (-23) 2020-11-20T12:01:37.105 INFO:teuthology.orchestra.run.smithi189.stdout:20 Nov 12:01:37 ntpd[5858]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-20T12:01:37.105 INFO:teuthology.orchestra.run.smithi189.stdout:20 Nov 12:01:37 ntpd[5858]: restrict ::: KOD does nothing without LIMITED. 2020-11-20T12:01:37.106 INFO:teuthology.orchestra.run.smithi189.stdout:20 Nov 12:01:37 ntpd[5858]: Listen and drop on 0 v6wildcard [::]:123 2020-11-20T12:01:37.106 INFO:teuthology.orchestra.run.smithi189.stdout:20 Nov 12:01:37 ntpd[5858]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-11-20T12:01:37.106 INFO:teuthology.orchestra.run.smithi189.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-20T12:01:37.106 INFO:teuthology.orchestra.run.smithi189.stderr:restrict ::: KOD does nothing without LIMITED. 2020-11-20T12:01:37.107 INFO:teuthology.orchestra.run.smithi189.stdout:20 Nov 12:01:37 ntpd[5858]: Listen normally on 2 lo 127.0.0.1:123 2020-11-20T12:01:37.107 INFO:teuthology.orchestra.run.smithi189.stdout:20 Nov 12:01:37 ntpd[5858]: Listen normally on 3 enp3s0f1 172.21.15.189:123 2020-11-20T12:01:37.108 INFO:teuthology.orchestra.run.smithi189.stdout:20 Nov 12:01:37 ntpd[5858]: Listen normally on 4 lo [::1]:123 2020-11-20T12:01:37.108 INFO:teuthology.orchestra.run.smithi189.stdout:20 Nov 12:01:37 ntpd[5858]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6df1%5]:123 2020-11-20T12:01:37.108 INFO:teuthology.orchestra.run.smithi189.stdout:20 Nov 12:01:37 ntpd[5858]: Listening on routing socket on fd #22 for interface updates 2020-11-20T12:01:38.105 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:38 ntpd[5858]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-20T12:01:38.105 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:38 ntpd[5858]: can't open /var/log/ntpstats/rawstats.20201120: Permission denied 2020-11-20T12:01:38.105 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:38 ntpd[5858]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-20T12:01:38.106 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:38 ntpd[5858]: can't open /var/log/ntpstats/peerstats.20201120: Permission denied 2020-11-20T12:01:38.106 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:38 ntpd[5858]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-20T12:01:38.106 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:38 ntpd[5858]: can't open /var/log/ntpstats/rawstats.20201120: Permission denied 2020-11-20T12:01:38.106 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:38 ntpd[5858]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-20T12:01:38.107 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:38 ntpd[5858]: can't open /var/log/ntpstats/peerstats.20201120: Permission denied 2020-11-20T12:01:40.104 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:40 ntpd[5858]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-20T12:01:40.105 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:40 ntpd[5858]: can't open /var/log/ntpstats/rawstats.20201120: Permission denied 2020-11-20T12:01:40.105 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:40 ntpd[5858]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-20T12:01:40.105 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:40 ntpd[5858]: can't open /var/log/ntpstats/peerstats.20201120: Permission denied 2020-11-20T12:01:40.105 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:40 ntpd[5858]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-20T12:01:40.106 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:40 ntpd[5858]: can't open /var/log/ntpstats/rawstats.20201120: Permission denied 2020-11-20T12:01:40.106 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:40 ntpd[5858]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-20T12:01:40.106 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:40 ntpd[5858]: can't open /var/log/ntpstats/peerstats.20201120: Permission denied 2020-11-20T12:01:41.104 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:41 ntpd[5858]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-20T12:01:41.105 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:41 ntpd[5858]: can't open /var/log/ntpstats/rawstats.20201120: Permission denied 2020-11-20T12:01:41.105 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:41 ntpd[5858]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-20T12:01:41.105 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:41 ntpd[5858]: can't open /var/log/ntpstats/peerstats.20201120: Permission denied 2020-11-20T12:01:42.105 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:42 ntpd[5858]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-20T12:01:42.105 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:42 ntpd[5858]: can't open /var/log/ntpstats/rawstats.20201120: Permission denied 2020-11-20T12:01:42.105 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:42 ntpd[5858]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-20T12:01:42.106 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:42 ntpd[5858]: can't open /var/log/ntpstats/peerstats.20201120: Permission denied 2020-11-20T12:01:42.106 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:42 ntpd[5858]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-20T12:01:42.106 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:42 ntpd[5858]: can't open /var/log/ntpstats/rawstats.20201120: Permission denied 2020-11-20T12:01:42.106 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:42 ntpd[5858]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-20T12:01:42.106 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:42 ntpd[5858]: can't open /var/log/ntpstats/peerstats.20201120: Permission denied 2020-11-20T12:01:42.107 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:42 ntpd[5858]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-20T12:01:42.107 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:42 ntpd[5858]: can't open /var/log/ntpstats/rawstats.20201120: Permission denied 2020-11-20T12:01:42.107 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:42 ntpd[5858]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-20T12:01:42.107 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:42 ntpd[5858]: can't open /var/log/ntpstats/peerstats.20201120: Permission denied 2020-11-20T12:01:43.104 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:43 ntpd[5858]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-20T12:01:43.104 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:43 ntpd[5858]: can't open /var/log/ntpstats/rawstats.20201120: Permission denied 2020-11-20T12:01:43.105 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:43 ntpd[5858]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-20T12:01:43.105 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:43 ntpd[5858]: can't open /var/log/ntpstats/peerstats.20201120: Permission denied 2020-11-20T12:01:44.104 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:44 ntpd[5858]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-20T12:01:44.105 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:44 ntpd[5858]: can't open /var/log/ntpstats/rawstats.20201120: Permission denied 2020-11-20T12:01:44.105 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:44 ntpd[5858]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-20T12:01:44.105 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:44 ntpd[5858]: can't open /var/log/ntpstats/peerstats.20201120: Permission denied 2020-11-20T12:01:44.105 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:44 ntpd[5858]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-11-20T12:01:44.105 INFO:teuthology.orchestra.run.smithi189.stderr:20 Nov 12:01:44 ntpd[5858]: can't open /var/log/ntpstats/loopstats.20201120: Permission denied 2020-11-20T12:01:44.106 INFO:teuthology.orchestra.run.smithi189.stdout:20 Nov 12:01:44 ntpd[5858]: ntpd: time slew -0.001490 s 2020-11-20T12:01:44.106 INFO:teuthology.orchestra.run.smithi189.stdout:ntpd: time slew -0.001490s 2020-11-20T12:01:44.154 INFO:teuthology.orchestra.run.smithi189.stdout: remote refid st t when poll reach delay offset jitter 2020-11-20T12:01:44.154 INFO:teuthology.orchestra.run.smithi189.stdout:============================================================================== 2020-11-20T12:01:44.154 INFO:teuthology.orchestra.run.smithi189.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-20T12:01:44.154 INFO:teuthology.orchestra.run.smithi189.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-20T12:01:44.155 INFO:teuthology.orchestra.run.smithi189.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-20T12:01:44.155 INFO:teuthology.orchestra.run.smithi189.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-20T12:01:44.155 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-11-20T12:01:44.156 INFO:teuthology.orchestra.run.smithi047:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-11-20T12:01:44.226 INFO:teuthology.orchestra.run.smithi047.stdout:20 Nov 12:01:44 ntpd[6012]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-11-20T12:01:44.226 INFO:teuthology.orchestra.run.smithi047.stdout:20 Nov 12:01:44 ntpd[6012]: Command line: ntpd -gq 2020-11-20T12:01:44.227 INFO:teuthology.orchestra.run.smithi047.stdout:20 Nov 12:01:44 ntpd[6012]: proto: precision = 0.127 usec (-23) 2020-11-20T12:01:44.228 INFO:teuthology.orchestra.run.smithi047.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-20T12:01:44.228 INFO:teuthology.orchestra.run.smithi047.stderr:restrict ::: KOD does nothing without LIMITED. 2020-11-20T12:01:44.229 INFO:teuthology.orchestra.run.smithi047.stdout:20 Nov 12:01:44 ntpd[6012]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-11-20T12:01:44.229 INFO:teuthology.orchestra.run.smithi047.stdout:20 Nov 12:01:44 ntpd[6012]: restrict ::: KOD does nothing without LIMITED. 2020-11-20T12:01:44.229 INFO:teuthology.orchestra.run.smithi047.stdout:20 Nov 12:01:44 ntpd[6012]: Listen and drop on 0 v6wildcard [::]:123 2020-11-20T12:01:44.229 INFO:teuthology.orchestra.run.smithi047.stdout:20 Nov 12:01:44 ntpd[6012]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-11-20T12:01:44.229 INFO:teuthology.orchestra.run.smithi047.stdout:20 Nov 12:01:44 ntpd[6012]: Listen normally on 2 lo 127.0.0.1:123 2020-11-20T12:01:44.230 INFO:teuthology.orchestra.run.smithi047.stdout:20 Nov 12:01:44 ntpd[6012]: Listen normally on 3 ens1f0 172.21.15.47:123 2020-11-20T12:01:44.230 INFO:teuthology.orchestra.run.smithi047.stdout:20 Nov 12:01:44 ntpd[6012]: Listen normally on 4 lo [::1]:123 2020-11-20T12:01:44.230 INFO:teuthology.orchestra.run.smithi047.stdout:20 Nov 12:01:44 ntpd[6012]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:148e%3]:123 2020-11-20T12:01:44.230 INFO:teuthology.orchestra.run.smithi047.stdout:20 Nov 12:01:44 ntpd[6012]: Listening on routing socket on fd #22 for interface updates 2020-11-20T12:01:45.226 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:45 ntpd[6012]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-20T12:01:45.275 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:45 ntpd[6012]: can't open /var/log/ntpstats/rawstats.20201120: Permission denied 2020-11-20T12:01:45.275 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:45 ntpd[6012]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-20T12:01:45.276 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:45 ntpd[6012]: can't open /var/log/ntpstats/peerstats.20201120: Permission denied 2020-11-20T12:01:47.226 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:47 ntpd[6012]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-20T12:01:47.227 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:47 ntpd[6012]: can't open /var/log/ntpstats/rawstats.20201120: Permission denied 2020-11-20T12:01:47.227 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:47 ntpd[6012]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-20T12:01:47.227 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:47 ntpd[6012]: can't open /var/log/ntpstats/peerstats.20201120: Permission denied 2020-11-20T12:01:47.227 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:47 ntpd[6012]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-20T12:01:47.227 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:47 ntpd[6012]: can't open /var/log/ntpstats/rawstats.20201120: Permission denied 2020-11-20T12:01:47.228 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:47 ntpd[6012]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-20T12:01:47.228 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:47 ntpd[6012]: can't open /var/log/ntpstats/peerstats.20201120: Permission denied 2020-11-20T12:01:47.228 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:47 ntpd[6012]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-20T12:01:47.228 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:47 ntpd[6012]: can't open /var/log/ntpstats/rawstats.20201120: Permission denied 2020-11-20T12:01:47.229 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:47 ntpd[6012]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-20T12:01:47.229 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:47 ntpd[6012]: can't open /var/log/ntpstats/peerstats.20201120: Permission denied 2020-11-20T12:01:47.229 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:47 ntpd[6012]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-20T12:01:47.229 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:47 ntpd[6012]: can't open /var/log/ntpstats/rawstats.20201120: Permission denied 2020-11-20T12:01:47.229 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:47 ntpd[6012]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-20T12:01:47.230 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:47 ntpd[6012]: can't open /var/log/ntpstats/peerstats.20201120: Permission denied 2020-11-20T12:01:49.225 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:49 ntpd[6012]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-20T12:01:49.225 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:49 ntpd[6012]: can't open /var/log/ntpstats/rawstats.20201120: Permission denied 2020-11-20T12:01:49.226 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:49 ntpd[6012]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-20T12:01:49.226 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:49 ntpd[6012]: can't open /var/log/ntpstats/peerstats.20201120: Permission denied 2020-11-20T12:01:49.226 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:49 ntpd[6012]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-20T12:01:49.226 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:49 ntpd[6012]: can't open /var/log/ntpstats/rawstats.20201120: Permission denied 2020-11-20T12:01:49.226 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:49 ntpd[6012]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-20T12:01:49.227 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:49 ntpd[6012]: can't open /var/log/ntpstats/peerstats.20201120: Permission denied 2020-11-20T12:01:49.227 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:49 ntpd[6012]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-20T12:01:49.227 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:49 ntpd[6012]: can't open /var/log/ntpstats/rawstats.20201120: Permission denied 2020-11-20T12:01:49.227 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:49 ntpd[6012]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-20T12:01:49.227 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:49 ntpd[6012]: can't open /var/log/ntpstats/peerstats.20201120: Permission denied 2020-11-20T12:01:49.228 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:49 ntpd[6012]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-20T12:01:49.228 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:49 ntpd[6012]: can't open /var/log/ntpstats/rawstats.20201120: Permission denied 2020-11-20T12:01:49.228 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:49 ntpd[6012]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-20T12:01:49.228 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:49 ntpd[6012]: can't open /var/log/ntpstats/peerstats.20201120: Permission denied 2020-11-20T12:01:51.224 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:51 ntpd[6012]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-20T12:01:51.224 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:51 ntpd[6012]: can't open /var/log/ntpstats/rawstats.20201120: Permission denied 2020-11-20T12:01:51.224 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:51 ntpd[6012]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-20T12:01:51.225 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:51 ntpd[6012]: can't open /var/log/ntpstats/peerstats.20201120: Permission denied 2020-11-20T12:01:51.225 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:51 ntpd[6012]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-11-20T12:01:51.225 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:51 ntpd[6012]: can't open /var/log/ntpstats/rawstats.20201120: Permission denied 2020-11-20T12:01:51.225 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:51 ntpd[6012]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-11-20T12:01:51.225 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:51 ntpd[6012]: can't open /var/log/ntpstats/peerstats.20201120: Permission denied 2020-11-20T12:01:51.226 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:51 ntpd[6012]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-11-20T12:01:51.226 INFO:teuthology.orchestra.run.smithi047.stdout:20 Nov 12:01:51 ntpd[6012]: ntpd: time set +0.096903 s 2020-11-20T12:01:51.226 INFO:teuthology.orchestra.run.smithi047.stdout:ntpd: time set +0.096903s 2020-11-20T12:01:51.227 INFO:teuthology.orchestra.run.smithi047.stderr:20 Nov 12:01:51 ntpd[6012]: can't open /var/log/ntpstats/loopstats.20201120: Permission denied 2020-11-20T12:01:51.279 INFO:teuthology.orchestra.run.smithi047.stdout: remote refid st t when poll reach delay offset jitter 2020-11-20T12:01:51.279 INFO:teuthology.orchestra.run.smithi047.stdout:============================================================================== 2020-11-20T12:01:51.280 INFO:teuthology.orchestra.run.smithi047.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-20T12:01:51.280 INFO:teuthology.orchestra.run.smithi047.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-20T12:01:51.280 INFO:teuthology.orchestra.run.smithi047.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-20T12:01:51.280 INFO:teuthology.orchestra.run.smithi047.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-11-20T12:01:51.281 INFO:teuthology.run_tasks:Running task install... 2020-11-20T12:01:51.290 DEBUG:teuthology.task.install:project ceph 2020-11-20T12:01:51.291 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb'}} 2020-11-20T12:01:51.291 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb'} 2020-11-20T12:01:51.291 INFO:teuthology.task.install:Using flavor: basic 2020-11-20T12:01:51.298 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-11-20T12:01:51.299 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']} 2020-11-20T12:01:51.299 INFO:teuthology.orchestra.run.smithi189:> sudo lsb_release -is 2020-11-20T12:01:51.357 INFO:teuthology.orchestra.run.smithi189.stdout:Ubuntu 2020-11-20T12:01:51.362 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-20T12:01:51.363 INFO:teuthology.orchestra.run.smithi047:> sudo lsb_release -is 2020-11-20T12:01:51.363 INFO:teuthology.orchestra.run.smithi189:> sudo apt-key list | grep Ceph 2020-11-20T12:01:51.417 INFO:teuthology.orchestra.run.smithi047.stdout:Ubuntu 2020-11-20T12:01:51.423 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-20T12:01:51.424 INFO:teuthology.orchestra.run.smithi047:> sudo apt-key list | grep Ceph 2020-11-20T12:01:51.463 INFO:teuthology.orchestra.run.smithi189.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-11-20T12:01:51.515 INFO:teuthology.orchestra.run.smithi047.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-11-20T12:01:51.521 INFO:teuthology.orchestra.run.smithi189.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-11-20T12:01:51.522 INFO:teuthology.orchestra.run.smithi189.stdout:uid [ unknown] Ceph.com (release key) 2020-11-20T12:01:51.523 INFO:teuthology.orchestra.run.smithi189:> sudo lsb_release -is 2020-11-20T12:01:51.572 INFO:teuthology.orchestra.run.smithi189.stdout:Ubuntu 2020-11-20T12:01:51.577 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-20T12:01:51.578 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2020-11-20T12:01:51.579 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2020-11-20T12:01:51.579 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=b86e691bf191566922a8f08e2a5abf64fde8cfcb 2020-11-20T12:01:51.585 INFO:teuthology.orchestra.run.smithi047.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-11-20T12:01:51.586 INFO:teuthology.orchestra.run.smithi047.stdout:uid [ unknown] Ceph.com (release key) 2020-11-20T12:01:51.586 INFO:teuthology.orchestra.run.smithi047:> sudo lsb_release -is 2020-11-20T12:01:51.636 INFO:teuthology.orchestra.run.smithi047.stdout:Ubuntu 2020-11-20T12:01:51.642 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-20T12:01:51.643 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2020-11-20T12:01:51.643 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2020-11-20T12:01:51.644 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=b86e691bf191566922a8f08e2a5abf64fde8cfcb 2020-11-20T12:01:51.765 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default/ 2020-11-20T12:01:51.765 INFO:teuthology.task.install.deb:Package version is 16.0.0-7397-gb86e691b-1bionic 2020-11-20T12:01:51.809 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default/ 2020-11-20T12:01:51.809 INFO:teuthology.task.install.deb:Package version is 16.0.0-7397-gb86e691b-1bionic 2020-11-20T12:01:51.891 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-20T12:01:51.892 INFO:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-20T12:01:51.898 INFO:teuthology.orchestra.run.smithi189.stderr:+ sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-20T12:01:51.908 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2020-11-20T12:01:51.908 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2020-11-20T12:01:51.908 INFO:teuthology.orchestra.run.smithi189.stderr:172 bytes copied, 6.8177e-05 s, 2.5 MB/s 2020-11-20T12:01:51.909 INFO:teuthology.orchestra.run.smithi189:> sudo apt-get update 2020-11-20T12:01:51.937 INFO:teuthology.orchestra.run.smithi047:> set -ex 2020-11-20T12:01:51.938 INFO:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-20T12:01:51.944 INFO:teuthology.orchestra.run.smithi047.stderr:+ sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-11-20T12:01:51.953 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records in 2020-11-20T12:01:51.954 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records out 2020-11-20T12:01:51.954 INFO:teuthology.orchestra.run.smithi047.stderr:172 bytes copied, 7.2803e-05 s, 2.4 MB/s 2020-11-20T12:01:51.955 INFO:teuthology.orchestra.run.smithi047:> sudo apt-get update 2020-11-20T12:01:52.090 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-11-20T12:01:52.145 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-11-20T12:01:52.174 INFO:teuthology.orchestra.run.smithi189.stdout:Ign:2 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic InRelease 2020-11-20T12:01:52.198 INFO:teuthology.orchestra.run.smithi189.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-11-20T12:01:52.203 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-20T12:01:52.223 INFO:teuthology.orchestra.run.smithi189.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic Release.gpg 2020-11-20T12:01:52.230 INFO:teuthology.orchestra.run.smithi047.stdout:Ign:2 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic InRelease 2020-11-20T12:01:52.246 INFO:teuthology.orchestra.run.smithi189.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,964 B] 2020-11-20T12:01:52.253 INFO:teuthology.orchestra.run.smithi047.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-11-20T12:01:52.279 INFO:teuthology.orchestra.run.smithi047.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic Release.gpg 2020-11-20T12:01:52.283 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-11-20T12:01:52.284 INFO:teuthology.orchestra.run.smithi189.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.1 kB] 2020-11-20T12:01:52.298 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:5 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-11-20T12:01:52.302 INFO:teuthology.orchestra.run.smithi047.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.1 kB] 2020-11-20T12:01:52.348 INFO:teuthology.orchestra.run.smithi047.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,964 B] 2020-11-20T12:01:52.363 INFO:teuthology.orchestra.run.smithi189.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-11-20T12:01:52.377 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-11-20T12:01:52.457 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-11-20T12:01:52.684 INFO:teuthology.orchestra.run.smithi189.stdout:Fetched 26.8 kB in 1s (46.4 kB/s) 2020-11-20T12:01:52.781 INFO:teuthology.orchestra.run.smithi047.stdout:Fetched 26.8 kB in 1s (43.5 kB/s) 2020-11-20T12:01:53.914 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2020-11-20T12:01:53.963 INFO:teuthology.orchestra.run.smithi189:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-7397-gb86e691b-1bionic cephadm=16.0.0-7397-gb86e691b-1bionic ceph-mds=16.0.0-7397-gb86e691b-1bionic ceph-mgr=16.0.0-7397-gb86e691b-1bionic ceph-common=16.0.0-7397-gb86e691b-1bionic ceph-fuse=16.0.0-7397-gb86e691b-1bionic ceph-test=16.0.0-7397-gb86e691b-1bionic radosgw=16.0.0-7397-gb86e691b-1bionic python3-rados=16.0.0-7397-gb86e691b-1bionic python3-rgw=16.0.0-7397-gb86e691b-1bionic python3-cephfs=16.0.0-7397-gb86e691b-1bionic python3-rbd=16.0.0-7397-gb86e691b-1bionic libcephfs2=16.0.0-7397-gb86e691b-1bionic libcephfs-dev=16.0.0-7397-gb86e691b-1bionic librados2=16.0.0-7397-gb86e691b-1bionic librbd1=16.0.0-7397-gb86e691b-1bionic rbd-fuse=16.0.0-7397-gb86e691b-1bionic python3-cephfs=16.0.0-7397-gb86e691b-1bionic cephfs-shell=16.0.0-7397-gb86e691b-1bionic 2020-11-20T12:01:54.046 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2020-11-20T12:01:54.060 INFO:teuthology.orchestra.run.smithi047.stdout:Reading package lists... 2020-11-20T12:01:54.108 INFO:teuthology.orchestra.run.smithi047:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-7397-gb86e691b-1bionic cephadm=16.0.0-7397-gb86e691b-1bionic ceph-mds=16.0.0-7397-gb86e691b-1bionic ceph-mgr=16.0.0-7397-gb86e691b-1bionic ceph-common=16.0.0-7397-gb86e691b-1bionic ceph-fuse=16.0.0-7397-gb86e691b-1bionic ceph-test=16.0.0-7397-gb86e691b-1bionic radosgw=16.0.0-7397-gb86e691b-1bionic python3-rados=16.0.0-7397-gb86e691b-1bionic python3-rgw=16.0.0-7397-gb86e691b-1bionic python3-cephfs=16.0.0-7397-gb86e691b-1bionic python3-rbd=16.0.0-7397-gb86e691b-1bionic libcephfs2=16.0.0-7397-gb86e691b-1bionic libcephfs-dev=16.0.0-7397-gb86e691b-1bionic librados2=16.0.0-7397-gb86e691b-1bionic librbd1=16.0.0-7397-gb86e691b-1bionic rbd-fuse=16.0.0-7397-gb86e691b-1bionic python3-cephfs=16.0.0-7397-gb86e691b-1bionic cephfs-shell=16.0.0-7397-gb86e691b-1bionic 2020-11-20T12:01:54.189 INFO:teuthology.orchestra.run.smithi047.stdout:Reading package lists... 2020-11-20T12:01:54.302 INFO:teuthology.orchestra.run.smithi189.stdout:Building dependency tree... 2020-11-20T12:01:54.302 INFO:teuthology.orchestra.run.smithi189.stdout:Reading state information... 2020-11-20T12:01:54.428 INFO:teuthology.orchestra.run.smithi047.stdout:Building dependency tree... 2020-11-20T12:01:54.428 INFO:teuthology.orchestra.run.smithi047.stdout:Reading state information... 2020-11-20T12:01:54.468 INFO:teuthology.orchestra.run.smithi189.stdout:The following additional packages will be installed: 2020-11-20T12:01:54.468 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2020-11-20T12:01:54.469 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2020-11-20T12:01:54.469 INFO:teuthology.orchestra.run.smithi189.stdout: libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-11-20T12:01:54.470 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2020-11-20T12:01:54.470 INFO:teuthology.orchestra.run.smithi189.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2020-11-20T12:01:54.470 INFO:teuthology.orchestra.run.smithi189.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-11-20T12:01:54.470 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2020-11-20T12:01:54.470 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-cryptography 2020-11-20T12:01:54.471 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cycler python3-dateutil python3-decorator python3-fasteners 2020-11-20T12:01:54.471 INFO:teuthology.orchestra.run.smithi189.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib 2020-11-20T12:01:54.471 INFO:teuthology.orchestra.run.smithi189.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes 2020-11-20T12:01:54.471 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-11-20T12:01:54.471 INFO:teuthology.orchestra.run.smithi189.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client 2020-11-20T12:01:54.472 INFO:teuthology.orchestra.run.smithi189.stdout: python3-olefile python3-openssl python3-paste python3-pastedeploy 2020-11-20T12:01:54.472 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pastescript python3-pecan python3-pil python3-pluggy 2020-11-20T12:01:54.472 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable python3-py python3-pyasn1 python3-pyasn1-modules 2020-11-20T12:01:54.472 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyinotify python3-pyparsing python3-pyperclip python3-pytest 2020-11-20T12:01:54.472 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-11-20T12:01:54.473 INFO:teuthology.orchestra.run.smithi189.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-20T12:01:54.473 INFO:teuthology.orchestra.run.smithi189.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-20T12:01:54.473 INFO:teuthology.orchestra.run.smithi189.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-20T12:01:54.473 INFO:teuthology.orchestra.run.smithi189.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-20T12:01:54.474 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-11-20T12:01:54.474 INFO:teuthology.orchestra.run.smithi189.stdout:Suggested packages: 2020-11-20T12:01:54.474 INFO:teuthology.orchestra.run.smithi189.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-11-20T12:01:54.474 INFO:teuthology.orchestra.run.smithi189.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-11-20T12:01:54.475 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-11-20T12:01:54.475 INFO:teuthology.orchestra.run.smithi189.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-11-20T12:01:54.475 INFO:teuthology.orchestra.run.smithi189.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-11-20T12:01:54.475 INFO:teuthology.orchestra.run.smithi189.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-11-20T12:01:54.476 INFO:teuthology.orchestra.run.smithi189.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-11-20T12:01:54.476 INFO:teuthology.orchestra.run.smithi189.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-11-20T12:01:54.476 INFO:teuthology.orchestra.run.smithi189.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-11-20T12:01:54.476 INFO:teuthology.orchestra.run.smithi189.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-11-20T12:01:54.476 INFO:teuthology.orchestra.run.smithi189.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-11-20T12:01:54.477 INFO:teuthology.orchestra.run.smithi189.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-11-20T12:01:54.477 INFO:teuthology.orchestra.run.smithi189.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-11-20T12:01:54.477 INFO:teuthology.orchestra.run.smithi189.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-11-20T12:01:54.477 INFO:teuthology.orchestra.run.smithi189.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-11-20T12:01:54.576 INFO:teuthology.orchestra.run.smithi189.stdout:The following NEW packages will be installed: 2020-11-20T12:01:54.576 INFO:teuthology.orchestra.run.smithi189.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-11-20T12:01:54.576 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2020-11-20T12:01:54.576 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm 2020-11-20T12:01:54.578 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-shell jq libcephfs-dev libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 2020-11-20T12:01:54.578 INFO:teuthology.orchestra.run.smithi189.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2020-11-20T12:01:54.578 INFO:teuthology.orchestra.run.smithi189.stdout: libsnappy1v5 libxslt1.1 nvme-cli python-attr python-funcsigs 2020-11-20T12:01:54.578 INFO:teuthology.orchestra.run.smithi189.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2020-11-20T12:01:54.579 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-11-20T12:01:54.579 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2020-11-20T12:01:54.579 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi-backend python3-cherrypy3 python3-cmd2 python3-colorama 2020-11-20T12:01:54.579 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography python3-cycler python3-dateutil python3-decorator 2020-11-20T12:01:54.579 INFO:teuthology.orchestra.run.smithi189.stdout: python3-fasteners python3-html5lib python3-httplib2 python3-jinja2 2020-11-20T12:01:54.580 INFO:teuthology.orchestra.run.smithi189.stdout: python3-joblib python3-json-pointer python3-jsonpatch python3-jwt 2020-11-20T12:01:54.580 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2020-11-20T12:01:54.580 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-11-20T12:01:54.580 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-11-20T12:01:54.580 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-11-20T12:01:54.581 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-11-20T12:01:54.581 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pyperclip 2020-11-20T12:01:54.581 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2020-11-20T12:01:54.581 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2020-11-20T12:01:54.582 INFO:teuthology.orchestra.run.smithi189.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2020-11-20T12:01:54.582 INFO:teuthology.orchestra.run.smithi189.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2020-11-20T12:01:54.582 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2020-11-20T12:01:54.582 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2020-11-20T12:01:54.582 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-11-20T12:01:54.583 INFO:teuthology.orchestra.run.smithi189.stdout:The following packages will be upgraded: 2020-11-20T12:01:54.584 INFO:teuthology.orchestra.run.smithi189.stdout: librados2 librbd1 2020-11-20T12:01:54.606 INFO:teuthology.orchestra.run.smithi047.stdout:The following additional packages will be installed: 2020-11-20T12:01:54.607 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2020-11-20T12:01:54.608 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2020-11-20T12:01:54.608 INFO:teuthology.orchestra.run.smithi047.stdout: libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-11-20T12:01:54.608 INFO:teuthology.orchestra.run.smithi047.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2020-11-20T12:01:54.609 INFO:teuthology.orchestra.run.smithi047.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2020-11-20T12:01:54.609 INFO:teuthology.orchestra.run.smithi047.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-11-20T12:01:54.609 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2020-11-20T12:01:54.609 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-cryptography 2020-11-20T12:01:54.610 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cycler python3-dateutil python3-decorator python3-fasteners 2020-11-20T12:01:54.610 INFO:teuthology.orchestra.run.smithi047.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib 2020-11-20T12:01:54.610 INFO:teuthology.orchestra.run.smithi047.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes 2020-11-20T12:01:54.610 INFO:teuthology.orchestra.run.smithi047.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-11-20T12:01:54.610 INFO:teuthology.orchestra.run.smithi047.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client 2020-11-20T12:01:54.611 INFO:teuthology.orchestra.run.smithi047.stdout: python3-olefile python3-openssl python3-paste python3-pastedeploy 2020-11-20T12:01:54.611 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pastescript python3-pecan python3-pil python3-pluggy 2020-11-20T12:01:54.611 INFO:teuthology.orchestra.run.smithi047.stdout: python3-prettytable python3-py python3-pyasn1 python3-pyasn1-modules 2020-11-20T12:01:54.611 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyinotify python3-pyparsing python3-pyperclip python3-pytest 2020-11-20T12:01:54.611 INFO:teuthology.orchestra.run.smithi047.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-11-20T12:01:54.612 INFO:teuthology.orchestra.run.smithi047.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-11-20T12:01:54.612 INFO:teuthology.orchestra.run.smithi047.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-11-20T12:01:54.612 INFO:teuthology.orchestra.run.smithi047.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-11-20T12:01:54.612 INFO:teuthology.orchestra.run.smithi047.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-11-20T12:01:54.612 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-11-20T12:01:54.613 INFO:teuthology.orchestra.run.smithi047.stdout:Suggested packages: 2020-11-20T12:01:54.613 INFO:teuthology.orchestra.run.smithi047.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-11-20T12:01:54.613 INFO:teuthology.orchestra.run.smithi047.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-11-20T12:01:54.614 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-11-20T12:01:54.614 INFO:teuthology.orchestra.run.smithi047.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-11-20T12:01:54.614 INFO:teuthology.orchestra.run.smithi047.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-11-20T12:01:54.614 INFO:teuthology.orchestra.run.smithi047.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-11-20T12:01:54.614 INFO:teuthology.orchestra.run.smithi047.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-11-20T12:01:54.615 INFO:teuthology.orchestra.run.smithi047.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-11-20T12:01:54.615 INFO:teuthology.orchestra.run.smithi047.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-11-20T12:01:54.615 INFO:teuthology.orchestra.run.smithi047.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-11-20T12:01:54.615 INFO:teuthology.orchestra.run.smithi047.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-11-20T12:01:54.615 INFO:teuthology.orchestra.run.smithi047.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-11-20T12:01:54.616 INFO:teuthology.orchestra.run.smithi047.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-11-20T12:01:54.616 INFO:teuthology.orchestra.run.smithi047.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-11-20T12:01:54.616 INFO:teuthology.orchestra.run.smithi047.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-11-20T12:01:54.719 INFO:teuthology.orchestra.run.smithi047.stdout:The following NEW packages will be installed: 2020-11-20T12:01:54.720 INFO:teuthology.orchestra.run.smithi047.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-11-20T12:01:54.720 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2020-11-20T12:01:54.720 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm 2020-11-20T12:01:54.721 INFO:teuthology.orchestra.run.smithi047.stdout: cephfs-shell jq libcephfs-dev libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 2020-11-20T12:01:54.722 INFO:teuthology.orchestra.run.smithi047.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2020-11-20T12:01:54.722 INFO:teuthology.orchestra.run.smithi047.stdout: libsnappy1v5 libxslt1.1 nvme-cli python-attr python-funcsigs 2020-11-20T12:01:54.722 INFO:teuthology.orchestra.run.smithi047.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2020-11-20T12:01:54.722 INFO:teuthology.orchestra.run.smithi047.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-11-20T12:01:54.722 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2020-11-20T12:01:54.723 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cffi-backend python3-cherrypy3 python3-cmd2 python3-colorama 2020-11-20T12:01:54.723 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cryptography python3-cycler python3-dateutil python3-decorator 2020-11-20T12:01:54.723 INFO:teuthology.orchestra.run.smithi047.stdout: python3-fasteners python3-html5lib python3-httplib2 python3-jinja2 2020-11-20T12:01:54.723 INFO:teuthology.orchestra.run.smithi047.stdout: python3-joblib python3-json-pointer python3-jsonpatch python3-jwt 2020-11-20T12:01:54.723 INFO:teuthology.orchestra.run.smithi047.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2020-11-20T12:01:54.724 INFO:teuthology.orchestra.run.smithi047.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-11-20T12:01:54.724 INFO:teuthology.orchestra.run.smithi047.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-11-20T12:01:54.724 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-11-20T12:01:54.724 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-11-20T12:01:54.725 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pyperclip 2020-11-20T12:01:54.725 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2020-11-20T12:01:54.725 INFO:teuthology.orchestra.run.smithi047.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2020-11-20T12:01:54.725 INFO:teuthology.orchestra.run.smithi047.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2020-11-20T12:01:54.725 INFO:teuthology.orchestra.run.smithi047.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2020-11-20T12:01:54.726 INFO:teuthology.orchestra.run.smithi047.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2020-11-20T12:01:54.726 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2020-11-20T12:01:54.726 INFO:teuthology.orchestra.run.smithi047.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-11-20T12:01:54.727 INFO:teuthology.orchestra.run.smithi047.stdout:The following packages will be upgraded: 2020-11-20T12:01:54.727 INFO:teuthology.orchestra.run.smithi047.stdout: librados2 librbd1 2020-11-20T12:01:54.769 INFO:teuthology.orchestra.run.smithi189.stdout:2 upgraded, 114 newly installed, 0 to remove and 20 not upgraded. 2020-11-20T12:01:54.769 INFO:teuthology.orchestra.run.smithi189.stdout:Need to get 168 MB of archives. 2020-11-20T12:01:54.769 INFO:teuthology.orchestra.run.smithi189.stdout:After this operation, 782 MB of additional disk space will be used. 2020-11-20T12:01:54.770 INFO:teuthology.orchestra.run.smithi189.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-11-20T12:01:54.913 INFO:teuthology.orchestra.run.smithi047.stdout:2 upgraded, 114 newly installed, 0 to remove and 20 not upgraded. 2020-11-20T12:01:54.913 INFO:teuthology.orchestra.run.smithi047.stdout:Need to get 168 MB of archives. 2020-11-20T12:01:54.914 INFO:teuthology.orchestra.run.smithi047.stdout:After this operation, 782 MB of additional disk space will be used. 2020-11-20T12:01:54.914 INFO:teuthology.orchestra.run.smithi047.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-11-20T12:01:54.969 INFO:teuthology.orchestra.run.smithi189.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-7397-gb86e691b-1bionic [1,921 kB] 2020-11-20T12:01:55.018 INFO:teuthology.orchestra.run.smithi189.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyperclip all 1.6.0-1 [9,376 B] 2020-11-20T12:01:55.026 INFO:teuthology.orchestra.run.smithi189.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cmd2 all 0.7.9-0ubuntu1 [34.2 kB] 2020-11-20T12:01:55.053 INFO:teuthology.orchestra.run.smithi189.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-11-20T12:01:55.054 INFO:teuthology.orchestra.run.smithi189.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-11-20T12:01:55.117 INFO:teuthology.orchestra.run.smithi047.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-7397-gb86e691b-1bionic [1,921 kB] 2020-11-20T12:01:55.160 INFO:teuthology.orchestra.run.smithi047.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyperclip all 1.6.0-1 [9,376 B] 2020-11-20T12:01:55.160 INFO:teuthology.orchestra.run.smithi047.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cmd2 all 0.7.9-0ubuntu1 [34.2 kB] 2020-11-20T12:01:55.162 INFO:teuthology.orchestra.run.smithi047.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-11-20T12:01:55.163 INFO:teuthology.orchestra.run.smithi047.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-11-20T12:01:55.183 INFO:teuthology.orchestra.run.smithi189.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-11-20T12:01:55.191 INFO:teuthology.orchestra.run.smithi189.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-11-20T12:01:55.210 INFO:teuthology.orchestra.run.smithi189.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-11-20T12:01:55.287 INFO:teuthology.orchestra.run.smithi189.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-11-20T12:01:55.292 INFO:teuthology.orchestra.run.smithi189.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-11-20T12:01:55.317 INFO:teuthology.orchestra.run.smithi189.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-11-20T12:01:55.325 INFO:teuthology.orchestra.run.smithi189.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-11-20T12:01:55.335 INFO:teuthology.orchestra.run.smithi189.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-11-20T12:01:55.337 INFO:teuthology.orchestra.run.smithi047.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-11-20T12:01:55.339 INFO:teuthology.orchestra.run.smithi047.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-11-20T12:01:55.345 INFO:teuthology.orchestra.run.smithi189.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-11-20T12:01:55.345 INFO:teuthology.orchestra.run.smithi047.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-11-20T12:01:55.352 INFO:teuthology.orchestra.run.smithi189.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2020-11-20T12:01:55.371 INFO:teuthology.orchestra.run.smithi189.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-11-20T12:01:55.374 INFO:teuthology.orchestra.run.smithi189.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-11-20T12:01:55.424 INFO:teuthology.orchestra.run.smithi189.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-11-20T12:01:55.433 INFO:teuthology.orchestra.run.smithi189.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-11-20T12:01:55.433 INFO:teuthology.orchestra.run.smithi189.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-11-20T12:01:55.435 INFO:teuthology.orchestra.run.smithi189.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-11-20T12:01:55.436 INFO:teuthology.orchestra.run.smithi189.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-11-20T12:01:55.436 INFO:teuthology.orchestra.run.smithi047.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-11-20T12:01:55.438 INFO:teuthology.orchestra.run.smithi047.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-11-20T12:01:55.455 INFO:teuthology.orchestra.run.smithi047.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-11-20T12:01:55.476 INFO:teuthology.orchestra.run.smithi047.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-11-20T12:01:55.492 INFO:teuthology.orchestra.run.smithi047.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-11-20T12:01:55.500 INFO:teuthology.orchestra.run.smithi047.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-11-20T12:01:55.503 INFO:teuthology.orchestra.run.smithi189.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-11-20T12:01:55.506 INFO:teuthology.orchestra.run.smithi189.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-11-20T12:01:55.507 INFO:teuthology.orchestra.run.smithi189.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-11-20T12:01:55.507 INFO:teuthology.orchestra.run.smithi047.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2020-11-20T12:01:55.508 INFO:teuthology.orchestra.run.smithi189.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-11-20T12:01:55.509 INFO:teuthology.orchestra.run.smithi189.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-11-20T12:01:55.531 INFO:teuthology.orchestra.run.smithi047.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-11-20T12:01:55.534 INFO:teuthology.orchestra.run.smithi047.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-11-20T12:01:55.580 INFO:teuthology.orchestra.run.smithi047.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-11-20T12:01:55.586 INFO:teuthology.orchestra.run.smithi189.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-11-20T12:01:55.587 INFO:teuthology.orchestra.run.smithi189.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-11-20T12:01:55.589 INFO:teuthology.orchestra.run.smithi047.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-11-20T12:01:55.590 INFO:teuthology.orchestra.run.smithi047.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-11-20T12:01:55.591 INFO:teuthology.orchestra.run.smithi047.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-11-20T12:01:55.592 INFO:teuthology.orchestra.run.smithi047.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-11-20T12:01:55.594 INFO:teuthology.orchestra.run.smithi047.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-11-20T12:01:55.603 INFO:teuthology.orchestra.run.smithi189.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-11-20T12:01:55.603 INFO:teuthology.orchestra.run.smithi189.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-11-20T12:01:55.604 INFO:teuthology.orchestra.run.smithi189.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-11-20T12:01:55.659 INFO:teuthology.orchestra.run.smithi047.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-11-20T12:01:55.659 INFO:teuthology.orchestra.run.smithi047.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-11-20T12:01:55.662 INFO:teuthology.orchestra.run.smithi047.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-11-20T12:01:55.666 INFO:teuthology.orchestra.run.smithi047.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-11-20T12:01:55.669 INFO:teuthology.orchestra.run.smithi189.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-11-20T12:01:55.670 INFO:teuthology.orchestra.run.smithi189.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-11-20T12:01:55.672 INFO:teuthology.orchestra.run.smithi189.stdout:Get:36 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-7397-gb86e691b-1bionic [3,160 kB] 2020-11-20T12:01:55.680 INFO:teuthology.orchestra.run.smithi189.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-11-20T12:01:55.682 INFO:teuthology.orchestra.run.smithi189.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-11-20T12:01:55.683 INFO:teuthology.orchestra.run.smithi189.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-11-20T12:01:55.738 INFO:teuthology.orchestra.run.smithi047.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-11-20T12:01:55.738 INFO:teuthology.orchestra.run.smithi047.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-11-20T12:01:55.746 INFO:teuthology.orchestra.run.smithi189.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-11-20T12:01:55.763 INFO:teuthology.orchestra.run.smithi047.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-11-20T12:01:55.764 INFO:teuthology.orchestra.run.smithi047.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-11-20T12:01:55.764 INFO:teuthology.orchestra.run.smithi047.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-11-20T12:01:55.766 INFO:teuthology.orchestra.run.smithi047.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-11-20T12:01:55.817 INFO:teuthology.orchestra.run.smithi047.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-11-20T12:01:55.826 INFO:teuthology.orchestra.run.smithi047.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-11-20T12:01:55.831 INFO:teuthology.orchestra.run.smithi189.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-11-20T12:01:55.831 INFO:teuthology.orchestra.run.smithi047.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-11-20T12:01:55.832 INFO:teuthology.orchestra.run.smithi189.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-11-20T12:01:55.832 INFO:teuthology.orchestra.run.smithi047.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-11-20T12:01:55.843 INFO:teuthology.orchestra.run.smithi047.stdout:Get:39 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-7397-gb86e691b-1bionic [3,160 kB] 2020-11-20T12:01:55.896 INFO:teuthology.orchestra.run.smithi047.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-11-20T12:01:55.980 INFO:teuthology.orchestra.run.smithi047.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-11-20T12:01:55.981 INFO:teuthology.orchestra.run.smithi047.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-11-20T12:01:56.109 INFO:teuthology.orchestra.run.smithi189.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-11-20T12:01:56.152 INFO:teuthology.orchestra.run.smithi189.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-11-20T12:01:56.153 INFO:teuthology.orchestra.run.smithi189.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-11-20T12:01:56.185 INFO:teuthology.orchestra.run.smithi189.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-11-20T12:01:56.185 INFO:teuthology.orchestra.run.smithi189.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-11-20T12:01:56.185 INFO:teuthology.orchestra.run.smithi189.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-11-20T12:01:56.189 INFO:teuthology.orchestra.run.smithi189.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-11-20T12:01:56.189 INFO:teuthology.orchestra.run.smithi189.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-11-20T12:01:56.190 INFO:teuthology.orchestra.run.smithi189.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-11-20T12:01:56.190 INFO:teuthology.orchestra.run.smithi189.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-11-20T12:01:56.256 INFO:teuthology.orchestra.run.smithi047.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-11-20T12:01:56.267 INFO:teuthology.orchestra.run.smithi189.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-11-20T12:01:56.268 INFO:teuthology.orchestra.run.smithi189.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-11-20T12:01:56.271 INFO:teuthology.orchestra.run.smithi189.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-11-20T12:01:56.271 INFO:teuthology.orchestra.run.smithi189.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-11-20T12:01:56.273 INFO:teuthology.orchestra.run.smithi189.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-11-20T12:01:56.273 INFO:teuthology.orchestra.run.smithi189.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-11-20T12:01:56.274 INFO:teuthology.orchestra.run.smithi189.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-11-20T12:01:56.275 INFO:teuthology.orchestra.run.smithi189.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-11-20T12:01:56.278 INFO:teuthology.orchestra.run.smithi189.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-11-20T12:01:56.300 INFO:teuthology.orchestra.run.smithi047.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-11-20T12:01:56.301 INFO:teuthology.orchestra.run.smithi047.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-11-20T12:01:56.320 INFO:teuthology.orchestra.run.smithi189.stdout:Get:62 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-7397-gb86e691b-1bionic [481 kB] 2020-11-20T12:01:56.344 INFO:teuthology.orchestra.run.smithi047.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-11-20T12:01:56.345 INFO:teuthology.orchestra.run.smithi047.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-11-20T12:01:56.345 INFO:teuthology.orchestra.run.smithi047.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-11-20T12:01:56.345 INFO:teuthology.orchestra.run.smithi047.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-11-20T12:01:56.346 INFO:teuthology.orchestra.run.smithi047.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-11-20T12:01:56.346 INFO:teuthology.orchestra.run.smithi047.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-11-20T12:01:56.347 INFO:teuthology.orchestra.run.smithi047.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-11-20T12:01:56.348 INFO:teuthology.orchestra.run.smithi189.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-11-20T12:01:56.356 INFO:teuthology.orchestra.run.smithi189.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-11-20T12:01:56.360 INFO:teuthology.orchestra.run.smithi189.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-11-20T12:01:56.364 INFO:teuthology.orchestra.run.smithi189.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-colorama all 0.3.7-1 [14.9 kB] 2020-11-20T12:01:56.364 INFO:teuthology.orchestra.run.smithi189.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1 [183 kB] 2020-11-20T12:01:56.366 INFO:teuthology.orchestra.run.smithi189.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-11-20T12:01:56.367 INFO:teuthology.orchestra.run.smithi189.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-11-20T12:01:56.367 INFO:teuthology.orchestra.run.smithi189.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-11-20T12:01:56.368 INFO:teuthology.orchestra.run.smithi189.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-11-20T12:01:56.416 INFO:teuthology.orchestra.run.smithi047.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-11-20T12:01:56.418 INFO:teuthology.orchestra.run.smithi047.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-11-20T12:01:56.428 INFO:teuthology.orchestra.run.smithi189.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-11-20T12:01:56.439 INFO:teuthology.orchestra.run.smithi189.stdout:Get:73 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-7397-gb86e691b-1bionic [332 kB] 2020-11-20T12:01:56.448 INFO:teuthology.orchestra.run.smithi189.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-11-20T12:01:56.449 INFO:teuthology.orchestra.run.smithi189.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-11-20T12:01:56.450 INFO:teuthology.orchestra.run.smithi189.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-11-20T12:01:56.450 INFO:teuthology.orchestra.run.smithi189.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-11-20T12:01:56.452 INFO:teuthology.orchestra.run.smithi189.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-11-20T12:01:56.473 INFO:teuthology.orchestra.run.smithi189.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-11-20T12:01:56.474 INFO:teuthology.orchestra.run.smithi189.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-11-20T12:01:56.495 INFO:teuthology.orchestra.run.smithi047.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-11-20T12:01:56.496 INFO:teuthology.orchestra.run.smithi047.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-11-20T12:01:56.502 INFO:teuthology.orchestra.run.smithi189.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-7397-gb86e691b-1bionic [20.6 kB] 2020-11-20T12:01:56.503 INFO:teuthology.orchestra.run.smithi047.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-11-20T12:01:56.503 INFO:teuthology.orchestra.run.smithi047.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-11-20T12:01:56.504 INFO:teuthology.orchestra.run.smithi047.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-11-20T12:01:56.504 INFO:teuthology.orchestra.run.smithi189.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-7397-gb86e691b-1bionic [168 kB] 2020-11-20T12:01:56.506 INFO:teuthology.orchestra.run.smithi047.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-11-20T12:01:56.508 INFO:teuthology.orchestra.run.smithi047.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-11-20T12:01:56.509 INFO:teuthology.orchestra.run.smithi047.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-11-20T12:01:56.516 INFO:teuthology.orchestra.run.smithi047.stdout:Get:63 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-7397-gb86e691b-1bionic [481 kB] 2020-11-20T12:01:56.543 INFO:teuthology.orchestra.run.smithi189.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-7397-gb86e691b-1bionic [27.7 kB] 2020-11-20T12:01:56.560 INFO:teuthology.orchestra.run.smithi189.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-7397-gb86e691b-1bionic [330 kB] 2020-11-20T12:01:56.575 INFO:teuthology.orchestra.run.smithi047.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-11-20T12:01:56.583 INFO:teuthology.orchestra.run.smithi047.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-11-20T12:01:56.590 INFO:teuthology.orchestra.run.smithi189.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-11-20T12:01:56.590 INFO:teuthology.orchestra.run.smithi189.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-11-20T12:01:56.591 INFO:teuthology.orchestra.run.smithi189.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.3 [330 kB] 2020-11-20T12:01:56.598 INFO:teuthology.orchestra.run.smithi189.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-11-20T12:01:56.598 INFO:teuthology.orchestra.run.smithi189.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-11-20T12:01:56.599 INFO:teuthology.orchestra.run.smithi189.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-11-20T12:01:56.599 INFO:teuthology.orchestra.run.smithi189.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-11-20T12:01:56.601 INFO:teuthology.orchestra.run.smithi189.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-11-20T12:01:56.601 INFO:teuthology.orchestra.run.smithi189.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-11-20T12:01:56.625 INFO:teuthology.orchestra.run.smithi189.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-7397-gb86e691b-1bionic [3,179 kB] 2020-11-20T12:01:56.639 INFO:teuthology.orchestra.run.smithi047.stdout:Get:66 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-7397-gb86e691b-1bionic [332 kB] 2020-11-20T12:01:56.654 INFO:teuthology.orchestra.run.smithi047.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-colorama all 0.3.7-1 [14.9 kB] 2020-11-20T12:01:56.655 INFO:teuthology.orchestra.run.smithi047.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1 [183 kB] 2020-11-20T12:01:56.657 INFO:teuthology.orchestra.run.smithi047.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-11-20T12:01:56.657 INFO:teuthology.orchestra.run.smithi047.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-11-20T12:01:56.657 INFO:teuthology.orchestra.run.smithi047.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-11-20T12:01:56.658 INFO:teuthology.orchestra.run.smithi047.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-11-20T12:01:56.658 INFO:teuthology.orchestra.run.smithi047.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-11-20T12:01:56.660 INFO:teuthology.orchestra.run.smithi047.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-11-20T12:01:56.704 INFO:teuthology.orchestra.run.smithi047.stdout:Get:75 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-7397-gb86e691b-1bionic [20.6 kB] 2020-11-20T12:01:56.705 INFO:teuthology.orchestra.run.smithi047.stdout:Get:76 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-7397-gb86e691b-1bionic [168 kB] 2020-11-20T12:01:56.711 INFO:teuthology.orchestra.run.smithi047.stdout:Get:77 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-7397-gb86e691b-1bionic [27.7 kB] 2020-11-20T12:01:56.712 INFO:teuthology.orchestra.run.smithi047.stdout:Get:78 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-7397-gb86e691b-1bionic [330 kB] 2020-11-20T12:01:56.735 INFO:teuthology.orchestra.run.smithi047.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-11-20T12:01:56.735 INFO:teuthology.orchestra.run.smithi047.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-11-20T12:01:56.813 INFO:teuthology.orchestra.run.smithi047.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-11-20T12:01:56.815 INFO:teuthology.orchestra.run.smithi047.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-11-20T12:01:56.831 INFO:teuthology.orchestra.run.smithi047.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-7397-gb86e691b-1bionic [3,179 kB] 2020-11-20T12:01:56.839 INFO:teuthology.orchestra.run.smithi047.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-11-20T12:01:56.839 INFO:teuthology.orchestra.run.smithi047.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-11-20T12:01:56.975 INFO:teuthology.orchestra.run.smithi047.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-11-20T12:01:56.976 INFO:teuthology.orchestra.run.smithi047.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-11-20T12:01:56.977 INFO:teuthology.orchestra.run.smithi047.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.3 [330 kB] 2020-11-20T12:01:56.982 INFO:teuthology.orchestra.run.smithi047.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-11-20T12:01:56.982 INFO:teuthology.orchestra.run.smithi047.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-11-20T12:01:56.983 INFO:teuthology.orchestra.run.smithi047.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-11-20T12:01:56.983 INFO:teuthology.orchestra.run.smithi047.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-11-20T12:01:56.984 INFO:teuthology.orchestra.run.smithi047.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-11-20T12:01:56.985 INFO:teuthology.orchestra.run.smithi047.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-11-20T12:01:57.275 INFO:teuthology.orchestra.run.smithi189.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-7397-gb86e691b-1bionic [99.1 kB] 2020-11-20T12:01:57.278 INFO:teuthology.orchestra.run.smithi189.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-7397-gb86e691b-1bionic [414 kB] 2020-11-20T12:01:57.349 INFO:teuthology.orchestra.run.smithi189.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-7397-gb86e691b-1bionic [20.7 MB] 2020-11-20T12:01:57.499 INFO:teuthology.orchestra.run.smithi047.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-7397-gb86e691b-1bionic [99.1 kB] 2020-11-20T12:01:57.502 INFO:teuthology.orchestra.run.smithi047.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-7397-gb86e691b-1bionic [414 kB] 2020-11-20T12:01:57.578 INFO:teuthology.orchestra.run.smithi047.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-7397-gb86e691b-1bionic [20.7 MB] 2020-11-20T12:02:01.359 INFO:teuthology.orchestra.run.smithi189.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-7397-gb86e691b-1bionic [5,571 kB] 2020-11-20T12:02:01.768 INFO:teuthology.orchestra.run.smithi047.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-7397-gb86e691b-1bionic [5,571 kB] 2020-11-20T12:02:02.134 INFO:teuthology.orchestra.run.smithi189.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-7397-gb86e691b-1bionic [165 kB] 2020-11-20T12:02:02.143 INFO:teuthology.orchestra.run.smithi189.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-7397-gb86e691b-1bionic [1,365 kB] 2020-11-20T12:02:02.309 INFO:teuthology.orchestra.run.smithi189.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-7397-gb86e691b-1bionic [6,660 kB] 2020-11-20T12:02:02.438 INFO:teuthology.orchestra.run.smithi047.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-7397-gb86e691b-1bionic [165 kB] 2020-11-20T12:02:02.476 INFO:teuthology.orchestra.run.smithi047.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-7397-gb86e691b-1bionic [1,365 kB] 2020-11-20T12:02:02.640 INFO:teuthology.orchestra.run.smithi047.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-7397-gb86e691b-1bionic [6,660 kB] 2020-11-20T12:02:03.073 INFO:teuthology.orchestra.run.smithi189.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-7397-gb86e691b-1bionic [24.6 MB] 2020-11-20T12:02:03.439 INFO:teuthology.orchestra.run.smithi047.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-7397-gb86e691b-1bionic [24.6 MB] 2020-11-20T12:02:06.074 INFO:teuthology.orchestra.run.smithi047.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-7397-gb86e691b-1bionic [3,808 B] 2020-11-20T12:02:06.075 INFO:teuthology.orchestra.run.smithi047.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-7397-gb86e691b-1bionic [566 kB] 2020-11-20T12:02:06.083 INFO:teuthology.orchestra.run.smithi189.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-7397-gb86e691b-1bionic [3,808 B] 2020-11-20T12:02:06.084 INFO:teuthology.orchestra.run.smithi189.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-7397-gb86e691b-1bionic [566 kB] 2020-11-20T12:02:06.101 INFO:teuthology.orchestra.run.smithi047.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-7397-gb86e691b-1bionic [1,948 kB] 2020-11-20T12:02:06.113 INFO:teuthology.orchestra.run.smithi189.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-7397-gb86e691b-1bionic [1,948 kB] 2020-11-20T12:02:06.227 INFO:teuthology.orchestra.run.smithi047.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-7397-gb86e691b-1bionic [51.7 kB] 2020-11-20T12:02:06.229 INFO:teuthology.orchestra.run.smithi047.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-7397-gb86e691b-1bionic [56.4 kB] 2020-11-20T12:02:06.231 INFO:teuthology.orchestra.run.smithi047.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-7397-gb86e691b-1bionic [2,878 kB] 2020-11-20T12:02:06.244 INFO:teuthology.orchestra.run.smithi189.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-7397-gb86e691b-1bionic [51.7 kB] 2020-11-20T12:02:06.246 INFO:teuthology.orchestra.run.smithi189.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-7397-gb86e691b-1bionic [56.4 kB] 2020-11-20T12:02:06.257 INFO:teuthology.orchestra.run.smithi189.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-7397-gb86e691b-1bionic [2,878 kB] 2020-11-20T12:02:06.414 INFO:teuthology.orchestra.run.smithi047.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-7397-gb86e691b-1bionic [7,883 kB] 2020-11-20T12:02:06.450 INFO:teuthology.orchestra.run.smithi189.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-7397-gb86e691b-1bionic [7,883 kB] 2020-11-20T12:02:06.910 INFO:teuthology.orchestra.run.smithi047.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-7397-gb86e691b-1bionic [3,928 B] 2020-11-20T12:02:08.806 INFO:teuthology.orchestra.run.smithi189.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-7397-gb86e691b-1bionic [3,928 B] 2020-11-20T12:02:08.807 INFO:teuthology.orchestra.run.smithi189.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-7397-gb86e691b-1bionic [22.3 kB] 2020-11-20T12:02:08.807 INFO:teuthology.orchestra.run.smithi189.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-7397-gb86e691b-1bionic [49.7 MB] 2020-11-20T12:02:08.807 INFO:teuthology.orchestra.run.smithi047.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-7397-gb86e691b-1bionic [22.3 kB] 2020-11-20T12:02:08.808 INFO:teuthology.orchestra.run.smithi047.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-7397-gb86e691b-1bionic [49.7 MB] 2020-11-20T12:02:09.840 INFO:teuthology.orchestra.run.smithi047.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 cephfs-shell all 16.0.0-7397-gb86e691b-1bionic [17.0 kB] 2020-11-20T12:02:09.840 INFO:teuthology.orchestra.run.smithi047.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-7397-gb86e691b-1bionic [19.5 kB] 2020-11-20T12:02:09.841 INFO:teuthology.orchestra.run.smithi047.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-7397-gb86e691b-1bionic [9,861 kB] 2020-11-20T12:02:10.467 INFO:teuthology.orchestra.run.smithi047.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-7397-gb86e691b-1bionic [77.3 kB] 2020-11-20T12:02:10.596 INFO:teuthology.orchestra.run.smithi189.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 cephfs-shell all 16.0.0-7397-gb86e691b-1bionic [17.0 kB] 2020-11-20T12:02:10.597 INFO:teuthology.orchestra.run.smithi189.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-7397-gb86e691b-1bionic [19.5 kB] 2020-11-20T12:02:10.597 INFO:teuthology.orchestra.run.smithi189.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-7397-gb86e691b-1bionic [9,861 kB] 2020-11-20T12:02:11.157 INFO:teuthology.orchestra.run.smithi189.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-7397-gb86e691b-1bionic [77.3 kB] 2020-11-20T12:02:11.184 INFO:teuthology.orchestra.run.smithi047.stdout:Fetched 168 MB in 16s (10.7 MB/s) 2020-11-20T12:02:11.299 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pyparsing. 2020-11-20T12:02:11.348 INFO:teuthology.orchestra.run.smithi047.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137607 files and directories currently installed.) 2020-11-20T12:02:11.352 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../000-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-11-20T12:02:11.390 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-20T12:02:11.701 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pyperclip. 2020-11-20T12:02:11.716 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../001-python3-pyperclip_1.6.0-1_all.deb ... 2020-11-20T12:02:11.759 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pyperclip (1.6.0-1) ... 2020-11-20T12:02:11.866 INFO:teuthology.orchestra.run.smithi189.stdout:Fetched 168 MB in 17s (10.2 MB/s) 2020-11-20T12:02:12.005 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pyparsing. 2020-11-20T12:02:12.055 INFO:teuthology.orchestra.run.smithi189.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137607 files and directories currently installed.) 2020-11-20T12:02:12.059 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../000-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-11-20T12:02:12.104 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-cmd2. 2020-11-20T12:02:12.104 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-20T12:02:12.119 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../002-python3-cmd2_0.7.9-0ubuntu1_all.deb ... 2020-11-20T12:02:12.162 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-cmd2 (0.7.9-0ubuntu1) ... 2020-11-20T12:02:12.415 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pyperclip. 2020-11-20T12:02:12.423 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-uritemplate. 2020-11-20T12:02:12.428 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../001-python3-pyperclip_1.6.0-1_all.deb ... 2020-11-20T12:02:12.438 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../003-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-11-20T12:02:12.465 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pyperclip (1.6.0-1) ... 2020-11-20T12:02:12.473 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-11-20T12:02:12.760 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-cmd2. 2020-11-20T12:02:12.774 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../002-python3-cmd2_0.7.9-0ubuntu1_all.deb ... 2020-11-20T12:02:12.776 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-11-20T12:02:12.790 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../004-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-11-20T12:02:12.810 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-cmd2 (0.7.9-0ubuntu1) ... 2020-11-20T12:02:12.868 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-20T12:02:13.104 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-uritemplate. 2020-11-20T12:02:13.119 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../003-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-11-20T12:02:13.162 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-11-20T12:02:13.212 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../005-librbd1_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:13.339 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking librbd1 (16.0.0-7397-gb86e691b-1bionic) over (12.2.13-0ubuntu0.18.04.5) ... 2020-11-20T12:02:13.599 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-11-20T12:02:13.612 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../004-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-11-20T12:02:13.716 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-20T12:02:13.857 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../006-librados2_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:13.959 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking librados2 (16.0.0-7397-gb86e691b-1bionic) over (12.2.13-0ubuntu0.18.04.5) ... 2020-11-20T12:02:14.050 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../005-librbd1_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:14.178 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking librbd1 (16.0.0-7397-gb86e691b-1bionic) over (12.2.13-0ubuntu0.18.04.5) ... 2020-11-20T12:02:14.596 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libcephfs2. 2020-11-20T12:02:14.613 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../007-libcephfs2_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:14.655 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libcephfs2 (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:14.721 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../006-librados2_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:14.824 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking librados2 (16.0.0-7397-gb86e691b-1bionic) over (12.2.13-0ubuntu0.18.04.5) ... 2020-11-20T12:02:14.957 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-rados. 2020-11-20T12:02:14.972 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../008-python3-rados_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:15.015 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-rados (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:15.360 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-11-20T12:02:15.375 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../009-python3-ceph-argparse_16.0.0-7397-gb86e691b-1bionic_all.deb ... 2020-11-20T12:02:15.410 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-ceph-argparse (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:15.453 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libcephfs2. 2020-11-20T12:02:15.467 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../007-libcephfs2_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:15.511 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libcephfs2 (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:15.713 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-cephfs. 2020-11-20T12:02:15.729 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../010-python3-cephfs_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:15.772 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-cephfs (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:15.830 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-rados. 2020-11-20T12:02:15.844 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../008-python3-rados_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:15.889 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-rados (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:16.082 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-ceph-common. 2020-11-20T12:02:16.096 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../011-python3-ceph-common_16.0.0-7397-gb86e691b-1bionic_all.deb ... 2020-11-20T12:02:16.140 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-ceph-common (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:16.233 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-11-20T12:02:16.249 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../009-python3-ceph-argparse_16.0.0-7397-gb86e691b-1bionic_all.deb ... 2020-11-20T12:02:16.283 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-ceph-argparse (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:16.402 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-prettytable. 2020-11-20T12:02:16.418 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../012-python3-prettytable_0.7.2-3_all.deb ... 2020-11-20T12:02:16.452 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-11-20T12:02:16.569 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-cephfs. 2020-11-20T12:02:16.585 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../010-python3-cephfs_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:16.627 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-cephfs (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:16.705 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-rbd. 2020-11-20T12:02:16.719 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../013-python3-rbd_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:16.763 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-rbd (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:16.955 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-ceph-common. 2020-11-20T12:02:16.969 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../011-python3-ceph-common_16.0.0-7397-gb86e691b-1bionic_all.deb ... 2020-11-20T12:02:17.006 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-ceph-common (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:17.115 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-11-20T12:02:17.131 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../014-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-11-20T12:02:17.174 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-20T12:02:17.300 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-prettytable. 2020-11-20T12:02:17.315 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../012-python3-prettytable_0.7.2-3_all.deb ... 2020-11-20T12:02:17.350 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-11-20T12:02:17.444 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-11-20T12:02:17.458 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../015-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-11-20T12:02:17.494 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-20T12:02:17.586 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-rbd. 2020-11-20T12:02:17.599 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../013-python3-rbd_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:17.628 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-rbd (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:17.804 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package librgw2. 2020-11-20T12:02:17.818 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../016-librgw2_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:17.854 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking librgw2 (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:18.006 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-11-20T12:02:18.022 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../014-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-11-20T12:02:18.064 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-20T12:02:18.400 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-11-20T12:02:18.415 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../015-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-11-20T12:02:18.459 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-20T12:02:18.549 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-rgw. 2020-11-20T12:02:18.562 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../017-python3-rgw_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:18.599 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-rgw (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:18.736 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package librgw2. 2020-11-20T12:02:18.751 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../016-librgw2_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:18.786 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking librgw2 (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:18.902 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-11-20T12:02:18.916 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../018-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-11-20T12:02:18.961 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-20T12:02:19.213 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-11-20T12:02:19.227 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../019-libleveldb1v5_1.20-2_amd64.deb ... 2020-11-20T12:02:19.271 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-11-20T12:02:19.358 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-rgw. 2020-11-20T12:02:19.374 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../017-python3-rgw_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:19.422 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-rgw (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:19.640 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package liboath0. 2020-11-20T12:02:19.655 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../020-liboath0_2.6.1-1_amd64.deb ... 2020-11-20T12:02:19.699 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-11-20T12:02:19.716 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-11-20T12:02:19.729 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../018-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-11-20T12:02:19.770 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-20T12:02:20.027 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-11-20T12:02:20.035 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libradosstriper1. 2020-11-20T12:02:20.042 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../019-libleveldb1v5_1.20-2_amd64.deb ... 2020-11-20T12:02:20.048 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../021-libradosstriper1_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:20.078 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-11-20T12:02:20.093 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libradosstriper1 (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:20.430 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package liboath0. 2020-11-20T12:02:20.444 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../020-liboath0_2.6.1-1_amd64.deb ... 2020-11-20T12:02:20.455 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-common. 2020-11-20T12:02:20.468 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../022-ceph-common_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:20.481 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-11-20T12:02:20.513 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-common (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:20.793 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libradosstriper1. 2020-11-20T12:02:20.807 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../021-libradosstriper1_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:20.850 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libradosstriper1 (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:21.220 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-common. 2020-11-20T12:02:21.236 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../022-ceph-common_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:21.295 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-common (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:22.345 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-base. 2020-11-20T12:02:22.359 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../023-ceph-base_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:22.404 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-base (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:23.167 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-base. 2020-11-20T12:02:23.184 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../023-ceph-base_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:23.242 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-base (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:23.308 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-dateutil. 2020-11-20T12:02:23.323 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../024-python3-dateutil_2.6.1-1_all.deb ... 2020-11-20T12:02:23.366 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-11-20T12:02:23.945 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-asn1crypto. 2020-11-20T12:02:23.961 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../025-python3-asn1crypto_0.24.0-1_all.deb ... 2020-11-20T12:02:24.003 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-11-20T12:02:24.289 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-cffi-backend. 2020-11-20T12:02:24.303 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../026-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-11-20T12:02:24.347 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-11-20T12:02:24.380 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-dateutil. 2020-11-20T12:02:24.396 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../024-python3-dateutil_2.6.1-1_all.deb ... 2020-11-20T12:02:24.430 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-11-20T12:02:24.608 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-cryptography. 2020-11-20T12:02:24.621 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../027-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2020-11-20T12:02:24.650 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-20T12:02:24.725 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-asn1crypto. 2020-11-20T12:02:24.739 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../025-python3-asn1crypto_0.24.0-1_all.deb ... 2020-11-20T12:02:24.783 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-11-20T12:02:25.037 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-openssl. 2020-11-20T12:02:25.044 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-cffi-backend. 2020-11-20T12:02:25.051 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../028-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-11-20T12:02:25.058 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../026-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-11-20T12:02:25.095 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-11-20T12:02:25.119 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-11-20T12:02:25.365 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-11-20T12:02:25.379 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../029-ceph-mgr-modules-core_16.0.0-7397-gb86e691b-1bionic_all.deb ... 2020-11-20T12:02:25.414 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mgr-modules-core (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:25.438 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-cryptography. 2020-11-20T12:02:25.452 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../027-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2020-11-20T12:02:25.488 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-20T12:02:25.760 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-bcrypt. 2020-11-20T12:02:25.775 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../030-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-11-20T12:02:25.818 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-11-20T12:02:25.867 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-openssl. 2020-11-20T12:02:25.879 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../028-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-11-20T12:02:25.917 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-11-20T12:02:26.130 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-cherrypy3. 2020-11-20T12:02:26.144 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../031-python3-cherrypy3_8.9.1-2_all.deb ... 2020-11-20T12:02:26.169 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-11-20T12:02:26.185 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../029-ceph-mgr-modules-core_16.0.0-7397-gb86e691b-1bionic_all.deb ... 2020-11-20T12:02:26.188 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-11-20T12:02:26.228 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mgr-modules-core (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:26.525 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-jwt. 2020-11-20T12:02:26.539 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../032-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-11-20T12:02:26.566 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-bcrypt. 2020-11-20T12:02:26.579 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../030-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-11-20T12:02:26.583 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-11-20T12:02:26.615 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-11-20T12:02:26.910 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-cherrypy3. 2020-11-20T12:02:26.926 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../031-python3-cherrypy3_8.9.1-2_all.deb ... 2020-11-20T12:02:26.993 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-11-20T12:02:27.011 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-logutils. 2020-11-20T12:02:27.026 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../033-python3-logutils_0.3.3-5_all.deb ... 2020-11-20T12:02:27.069 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-11-20T12:02:27.313 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-jwt. 2020-11-20T12:02:27.326 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../032-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-11-20T12:02:27.363 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-11-20T12:02:27.389 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-markupsafe. 2020-11-20T12:02:27.405 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../034-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-11-20T12:02:27.447 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-11-20T12:02:27.683 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-logutils. 2020-11-20T12:02:27.696 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../033-python3-logutils_0.3.3-5_all.deb ... 2020-11-20T12:02:27.733 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-11-20T12:02:27.750 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-mako. 2020-11-20T12:02:27.764 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../035-python3-mako_1.0.7+ds1-1_all.deb ... 2020-11-20T12:02:27.808 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-11-20T12:02:28.044 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-markupsafe. 2020-11-20T12:02:28.059 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../034-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-11-20T12:02:28.103 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-11-20T12:02:28.111 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-simplegeneric. 2020-11-20T12:02:28.124 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../036-python3-simplegeneric_0.8.1-1_all.deb ... 2020-11-20T12:02:28.162 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-11-20T12:02:28.405 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-mako. 2020-11-20T12:02:28.420 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../035-python3-mako_1.0.7+ds1-1_all.deb ... 2020-11-20T12:02:28.464 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-11-20T12:02:28.464 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-singledispatch. 2020-11-20T12:02:28.478 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../037-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-11-20T12:02:28.522 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-11-20T12:02:29.450 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-simplegeneric. 2020-11-20T12:02:29.583 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-webob. 2020-11-20T12:02:29.583 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../038-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-11-20T12:02:29.584 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-11-20T12:02:29.584 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../036-python3-simplegeneric_0.8.1-1_all.deb ... 2020-11-20T12:02:29.585 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-11-20T12:02:29.811 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-singledispatch. 2020-11-20T12:02:29.827 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../037-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-11-20T12:02:29.837 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-bs4. 2020-11-20T12:02:29.852 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../039-python3-bs4_4.6.0-1_all.deb ... 2020-11-20T12:02:29.870 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-11-20T12:02:29.896 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-11-20T12:02:30.181 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-webob. 2020-11-20T12:02:30.195 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../038-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-11-20T12:02:30.231 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-waitress. 2020-11-20T12:02:30.239 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-11-20T12:02:30.247 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../040-python3-waitress_1.0.1-1_all.deb ... 2020-11-20T12:02:30.289 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-11-20T12:02:30.542 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-bs4. 2020-11-20T12:02:30.558 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../039-python3-bs4_4.6.0-1_all.deb ... 2020-11-20T12:02:30.592 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-tempita. 2020-11-20T12:02:30.600 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-11-20T12:02:30.608 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../041-python3-tempita_0.5.2-2_all.deb ... 2020-11-20T12:02:30.650 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-11-20T12:02:30.953 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-waitress. 2020-11-20T12:02:30.969 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../040-python3-waitress_1.0.1-1_all.deb ... 2020-11-20T12:02:30.970 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-paste. 2020-11-20T12:02:30.983 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../042-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-11-20T12:02:31.011 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-11-20T12:02:31.019 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-20T12:02:31.339 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-11-20T12:02:31.348 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-tempita. 2020-11-20T12:02:31.354 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-11-20T12:02:31.364 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../041-python3-tempita_0.5.2-2_all.deb ... 2020-11-20T12:02:31.398 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-11-20T12:02:31.406 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-11-20T12:02:31.684 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pastedeploy. 2020-11-20T12:02:31.697 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../044-python3-pastedeploy_1.5.2-4_all.deb ... 2020-11-20T12:02:31.701 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-paste. 2020-11-20T12:02:31.715 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../042-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-11-20T12:02:31.734 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-11-20T12:02:31.759 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-20T12:02:32.053 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-webtest. 2020-11-20T12:02:32.069 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../045-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-11-20T12:02:32.070 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-11-20T12:02:32.084 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-11-20T12:02:32.120 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-11-20T12:02:32.129 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-11-20T12:02:32.456 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pecan. 2020-11-20T12:02:32.457 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pastedeploy. 2020-11-20T12:02:32.471 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../046-python3-pecan_1.2.1-2_all.deb ... 2020-11-20T12:02:32.471 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../044-python3-pastedeploy_1.5.2-4_all.deb ... 2020-11-20T12:02:32.514 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-11-20T12:02:32.523 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-11-20T12:02:32.851 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-webtest. 2020-11-20T12:02:32.852 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-werkzeug. 2020-11-20T12:02:32.867 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../045-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-11-20T12:02:32.867 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../047-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-11-20T12:02:32.909 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-11-20T12:02:32.910 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-20T12:02:33.246 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pecan. 2020-11-20T12:02:33.260 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../046-python3-pecan_1.2.1-2_all.deb ... 2020-11-20T12:02:33.280 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mgr. 2020-11-20T12:02:33.295 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../048-ceph-mgr_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:33.304 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-11-20T12:02:33.339 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mgr (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:33.644 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-werkzeug. 2020-11-20T12:02:33.657 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../047-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-11-20T12:02:33.694 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-20T12:02:33.817 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mon. 2020-11-20T12:02:33.831 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../049-ceph-mon_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:33.875 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mon (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:34.047 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mgr. 2020-11-20T12:02:34.062 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../048-ceph-mgr_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:34.105 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mgr (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:34.550 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mon. 2020-11-20T12:02:34.566 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../049-ceph-mon_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:34.608 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mon (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:34.845 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-osd. 2020-11-20T12:02:34.859 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../050-ceph-osd_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:34.912 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-osd (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:35.561 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-osd. 2020-11-20T12:02:35.577 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../050-ceph-osd_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:35.619 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-osd (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:37.084 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph. 2020-11-20T12:02:37.098 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../051-ceph_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:37.142 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:37.587 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-fuse. 2020-11-20T12:02:37.600 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../052-ceph-fuse_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:37.654 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-fuse (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:37.758 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph. 2020-11-20T12:02:37.771 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../051-ceph_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:37.808 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:38.057 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mds. 2020-11-20T12:02:38.071 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../053-ceph-mds_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:38.115 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mds (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:38.152 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-fuse. 2020-11-20T12:02:38.167 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../052-ceph-fuse_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:38.219 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-fuse (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:38.601 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package cephadm. 2020-11-20T12:02:38.617 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../054-cephadm_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:38.630 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mds. 2020-11-20T12:02:38.644 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../053-ceph-mds_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:38.659 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking cephadm (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:38.688 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mds (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:39.013 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-jinja2. 2020-11-20T12:02:39.021 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../055-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-11-20T12:02:39.054 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-20T12:02:39.178 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package cephadm. 2020-11-20T12:02:39.190 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../054-cephadm_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:39.233 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking cephadm (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:39.407 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-11-20T12:02:39.422 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../056-ceph-mgr-cephadm_16.0.0-7397-gb86e691b-1bionic_all.deb ... 2020-11-20T12:02:39.465 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mgr-cephadm (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:39.552 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-jinja2. 2020-11-20T12:02:39.567 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../055-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-11-20T12:02:39.619 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-20T12:02:39.827 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-repoze.lru. 2020-11-20T12:02:39.840 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../057-python3-repoze.lru_0.7-1_all.deb ... 2020-11-20T12:02:39.885 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-11-20T12:02:39.938 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-11-20T12:02:39.952 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../056-ceph-mgr-cephadm_16.0.0-7397-gb86e691b-1bionic_all.deb ... 2020-11-20T12:02:39.988 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mgr-cephadm (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:40.205 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-routes. 2020-11-20T12:02:40.219 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../058-python3-routes_2.4.1-1_all.deb ... 2020-11-20T12:02:40.263 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-11-20T12:02:40.342 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-repoze.lru. 2020-11-20T12:02:40.355 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../057-python3-repoze.lru_0.7-1_all.deb ... 2020-11-20T12:02:40.413 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-11-20T12:02:40.575 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-11-20T12:02:40.589 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../059-ceph-mgr-dashboard_16.0.0-7397-gb86e691b-1bionic_all.deb ... 2020-11-20T12:02:40.633 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mgr-dashboard (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:40.736 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-routes. 2020-11-20T12:02:40.749 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../058-python3-routes_2.4.1-1_all.deb ... 2020-11-20T12:02:40.786 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-11-20T12:02:41.080 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-11-20T12:02:41.096 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../059-ceph-mgr-dashboard_16.0.0-7397-gb86e691b-1bionic_all.deb ... 2020-11-20T12:02:41.138 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mgr-dashboard (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:41.516 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-numpy. 2020-11-20T12:02:41.531 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../060-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-11-20T12:02:41.582 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-20T12:02:41.979 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-numpy. 2020-11-20T12:02:41.995 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../060-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-11-20T12:02:42.037 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-20T12:02:42.155 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-decorator. 2020-11-20T12:02:42.170 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../061-python3-decorator_4.1.2-1_all.deb ... 2020-11-20T12:02:42.213 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-11-20T12:02:42.532 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-scipy. 2020-11-20T12:02:42.547 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../062-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-11-20T12:02:42.591 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-11-20T12:02:42.660 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-decorator. 2020-11-20T12:02:42.675 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../061-python3-decorator_4.1.2-1_all.deb ... 2020-11-20T12:02:42.718 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-11-20T12:02:43.029 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-scipy. 2020-11-20T12:02:43.043 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../062-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-11-20T12:02:43.087 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-11-20T12:02:43.908 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-11-20T12:02:43.923 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../063-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-11-20T12:02:44.000 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-11-20T12:02:44.438 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-11-20T12:02:44.452 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../063-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-11-20T12:02:44.495 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-joblib. 2020-11-20T12:02:44.495 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-11-20T12:02:44.508 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../064-python3-joblib_0.11-1_all.deb ... 2020-11-20T12:02:44.553 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-joblib (0.11-1) ... 2020-11-20T12:02:44.898 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-sklearn. 2020-11-20T12:02:44.912 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../065-python3-sklearn_0.19.1-3_all.deb ... 2020-11-20T12:02:44.981 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-11-20T12:02:44.982 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-joblib. 2020-11-20T12:02:44.998 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../064-python3-joblib_0.11-1_all.deb ... 2020-11-20T12:02:45.066 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-joblib (0.11-1) ... 2020-11-20T12:02:45.427 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-sklearn. 2020-11-20T12:02:45.443 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../065-python3-sklearn_0.19.1-3_all.deb ... 2020-11-20T12:02:45.486 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-11-20T12:02:45.538 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-11-20T12:02:45.552 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../066-ceph-mgr-diskprediction-local_16.0.0-7397-gb86e691b-1bionic_all.deb ... 2020-11-20T12:02:45.596 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:46.050 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-11-20T12:02:46.064 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../066-ceph-mgr-diskprediction-local_16.0.0-7397-gb86e691b-1bionic_all.deb ... 2020-11-20T12:02:46.142 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:46.691 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-monotonic. 2020-11-20T12:02:46.708 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../067-python3-monotonic_1.1-2_all.deb ... 2020-11-20T12:02:46.749 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-11-20T12:02:47.077 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-fasteners. 2020-11-20T12:02:47.091 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../068-python3-fasteners_0.12.0-3_all.deb ... 2020-11-20T12:02:47.135 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-11-20T12:02:47.211 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-monotonic. 2020-11-20T12:02:47.228 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../067-python3-monotonic_1.1-2_all.deb ... 2020-11-20T12:02:47.269 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-11-20T12:02:47.480 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-httplib2. 2020-11-20T12:02:47.495 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../069-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-11-20T12:02:47.547 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-20T12:02:47.572 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-fasteners. 2020-11-20T12:02:47.588 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../068-python3-fasteners_0.12.0-3_all.deb ... 2020-11-20T12:02:47.630 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-11-20T12:02:47.858 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pyasn1. 2020-11-20T12:02:47.874 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../070-python3-pyasn1_0.4.2-3_all.deb ... 2020-11-20T12:02:47.916 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-11-20T12:02:47.933 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-httplib2. 2020-11-20T12:02:47.950 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../069-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-11-20T12:02:47.992 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-20T12:02:48.227 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-11-20T12:02:48.244 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../071-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-11-20T12:02:48.286 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-20T12:02:48.328 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pyasn1. 2020-11-20T12:02:48.345 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../070-python3-pyasn1_0.4.2-3_all.deb ... 2020-11-20T12:02:48.386 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-11-20T12:02:48.597 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-rsa. 2020-11-20T12:02:48.614 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../072-python3-rsa_3.4.2-1_all.deb ... 2020-11-20T12:02:48.655 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-11-20T12:02:48.714 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-11-20T12:02:48.730 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../071-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-11-20T12:02:48.772 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-20T12:02:48.950 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-11-20T12:02:48.964 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../073-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-11-20T12:02:49.008 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-20T12:02:49.083 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-rsa. 2020-11-20T12:02:49.099 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../072-python3-rsa_3.4.2-1_all.deb ... 2020-11-20T12:02:49.141 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-11-20T12:02:49.436 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-11-20T12:02:49.453 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../073-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-11-20T12:02:49.464 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-oauth2client. 2020-11-20T12:02:49.477 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../074-python3-oauth2client_4.1.2-3_all.deb ... 2020-11-20T12:02:49.495 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-20T12:02:49.521 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-11-20T12:02:49.841 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-setuptools. 2020-11-20T12:02:49.854 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../075-python3-setuptools_39.0.1-2_all.deb ... 2020-11-20T12:02:49.899 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-11-20T12:02:49.974 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-oauth2client. 2020-11-20T12:02:49.988 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../074-python3-oauth2client_4.1.2-3_all.deb ... 2020-11-20T12:02:50.032 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-11-20T12:02:50.269 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-websocket. 2020-11-20T12:02:50.285 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../076-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-11-20T12:02:50.327 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-11-20T12:02:50.360 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-setuptools. 2020-11-20T12:02:50.375 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../075-python3-setuptools_39.0.1-2_all.deb ... 2020-11-20T12:02:50.418 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-11-20T12:02:50.647 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-kubernetes. 2020-11-20T12:02:50.662 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../077-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-11-20T12:02:50.705 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-20T12:02:50.796 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-websocket. 2020-11-20T12:02:50.813 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../076-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-11-20T12:02:50.854 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-11-20T12:02:51.101 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-11-20T12:02:51.116 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../078-ceph-mgr-k8sevents_16.0.0-7397-gb86e691b-1bionic_all.deb ... 2020-11-20T12:02:51.159 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:51.166 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-kubernetes. 2020-11-20T12:02:51.180 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../077-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-11-20T12:02:51.225 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-20T12:02:51.481 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-json-pointer. 2020-11-20T12:02:51.496 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../079-python3-json-pointer_1.10-1_all.deb ... 2020-11-20T12:02:51.537 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-11-20T12:02:51.645 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-11-20T12:02:51.659 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../078-ceph-mgr-k8sevents_16.0.0-7397-gb86e691b-1bionic_all.deb ... 2020-11-20T12:02:51.703 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:51.932 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-jsonpatch. 2020-11-20T12:02:51.947 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../080-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-11-20T12:02:52.047 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-json-pointer. 2020-11-20T12:02:52.063 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../079-python3-json-pointer_1.10-1_all.deb ... 2020-11-20T12:02:52.072 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-20T12:02:52.106 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-11-20T12:02:52.421 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-11-20T12:02:52.438 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../081-ceph-mgr-rook_16.0.0-7397-gb86e691b-1bionic_all.deb ... 2020-11-20T12:02:52.467 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-jsonpatch. 2020-11-20T12:02:52.479 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mgr-rook (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:52.484 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../080-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-11-20T12:02:52.631 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-20T12:02:52.832 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libonig4:amd64. 2020-11-20T12:02:52.847 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../082-libonig4_6.7.0-1_amd64.deb ... 2020-11-20T12:02:52.899 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-11-20T12:02:52.931 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-11-20T12:02:52.944 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../081-ceph-mgr-rook_16.0.0-7397-gb86e691b-1bionic_all.deb ... 2020-11-20T12:02:52.989 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-mgr-rook (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:53.260 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libjq1:amd64. 2020-11-20T12:02:53.275 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../083-libjq1_1.5+dfsg-2_amd64.deb ... 2020-11-20T12:02:53.317 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libonig4:amd64. 2020-11-20T12:02:53.318 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-11-20T12:02:53.330 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../082-libonig4_6.7.0-1_amd64.deb ... 2020-11-20T12:02:53.366 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-11-20T12:02:53.604 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package jq. 2020-11-20T12:02:53.621 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../084-jq_1.5+dfsg-2_amd64.deb ... 2020-11-20T12:02:53.662 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-11-20T12:02:53.795 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libjq1:amd64. 2020-11-20T12:02:53.809 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../083-libjq1_1.5+dfsg-2_amd64.deb ... 2020-11-20T12:02:53.853 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-11-20T12:02:53.923 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package socat. 2020-11-20T12:02:53.933 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../085-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-11-20T12:02:53.973 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-11-20T12:02:54.114 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package jq. 2020-11-20T12:02:54.128 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../084-jq_1.5+dfsg-2_amd64.deb ... 2020-11-20T12:02:54.172 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-11-20T12:02:54.652 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-11-20T12:02:54.669 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../086-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-11-20T12:02:54.702 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-20T12:02:54.799 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package socat. 2020-11-20T12:02:54.815 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../085-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-11-20T12:02:54.858 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-11-20T12:02:54.980 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package xmlstarlet. 2020-11-20T12:02:54.997 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../087-xmlstarlet_1.6.1-2_amd64.deb ... 2020-11-20T12:02:55.038 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-11-20T12:02:55.220 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-11-20T12:02:55.233 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../086-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-11-20T12:02:55.261 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-20T12:02:55.351 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-test. 2020-11-20T12:02:55.367 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../088-ceph-test_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:55.409 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-test (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:55.514 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package xmlstarlet. 2020-11-20T12:02:55.530 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../087-xmlstarlet_1.6.1-2_amd64.deb ... 2020-11-20T12:02:55.572 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-11-20T12:02:55.901 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ceph-test. 2020-11-20T12:02:55.917 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../088-ceph-test_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:55.951 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ceph-test (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:59.131 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-colorama. 2020-11-20T12:02:59.146 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../089-python3-colorama_0.3.7-1_all.deb ... 2020-11-20T12:02:59.189 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-colorama (0.3.7-1) ... 2020-11-20T12:02:59.417 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package cephfs-shell. 2020-11-20T12:02:59.431 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../090-cephfs-shell_16.0.0-7397-gb86e691b-1bionic_all.deb ... 2020-11-20T12:02:59.466 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking cephfs-shell (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:59.655 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-colorama. 2020-11-20T12:02:59.671 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../089-python3-colorama_0.3.7-1_all.deb ... 2020-11-20T12:02:59.711 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libcephfs-dev. 2020-11-20T12:02:59.713 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-colorama (0.3.7-1) ... 2020-11-20T12:02:59.724 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../091-libcephfs-dev_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:02:59.769 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libcephfs-dev (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:02:59.949 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package cephfs-shell. 2020-11-20T12:02:59.963 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../090-cephfs-shell_16.0.0-7397-gb86e691b-1bionic_all.deb ... 2020-11-20T12:03:00.007 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking cephfs-shell (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:00.046 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package nvme-cli. 2020-11-20T12:03:00.062 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../092-nvme-cli_1.5-1ubuntu1_amd64.deb ... 2020-11-20T12:03:00.105 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking nvme-cli (1.5-1ubuntu1) ... 2020-11-20T12:03:00.244 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libcephfs-dev. 2020-11-20T12:03:00.257 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../091-libcephfs-dev_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:03:00.293 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libcephfs-dev (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:00.451 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python-attr. 2020-11-20T12:03:00.465 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../093-python-attr_17.4.0-2_all.deb ... 2020-11-20T12:03:00.509 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python-attr (17.4.0-2) ... 2020-11-20T12:03:00.537 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package nvme-cli. 2020-11-20T12:03:00.552 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../092-nvme-cli_1.5-1ubuntu1_amd64.deb ... 2020-11-20T12:03:00.596 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking nvme-cli (1.5-1ubuntu1) ... 2020-11-20T12:03:00.795 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python-funcsigs. 2020-11-20T12:03:00.813 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../094-python-funcsigs_1.0.2-4_all.deb ... 2020-11-20T12:03:00.854 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-11-20T12:03:00.941 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python-attr. 2020-11-20T12:03:00.958 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../093-python-attr_17.4.0-2_all.deb ... 2020-11-20T12:03:00.999 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python-attr (17.4.0-2) ... 2020-11-20T12:03:01.108 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python-pluggy. 2020-11-20T12:03:01.122 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../095-python-pluggy_0.6.0-1_all.deb ... 2020-11-20T12:03:01.158 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-11-20T12:03:01.311 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python-funcsigs. 2020-11-20T12:03:01.327 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../094-python-funcsigs_1.0.2-4_all.deb ... 2020-11-20T12:03:01.360 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-11-20T12:03:01.419 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python-py. 2020-11-20T12:03:01.436 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../096-python-py_1.5.2-1_all.deb ... 2020-11-20T12:03:01.477 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python-py (1.5.2-1) ... 2020-11-20T12:03:01.638 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python-pluggy. 2020-11-20T12:03:01.655 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../095-python-pluggy_0.6.0-1_all.deb ... 2020-11-20T12:03:01.688 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-11-20T12:03:01.780 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python-pytest. 2020-11-20T12:03:01.797 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../097-python-pytest_3.3.2-2_all.deb ... 2020-11-20T12:03:01.838 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-11-20T12:03:01.999 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python-py. 2020-11-20T12:03:02.015 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../096-python-py_1.5.2-1_all.deb ... 2020-11-20T12:03:02.057 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python-py (1.5.2-1) ... 2020-11-20T12:03:02.167 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-attr. 2020-11-20T12:03:02.181 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../098-python3-attr_17.4.0-2_all.deb ... 2020-11-20T12:03:02.225 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-11-20T12:03:02.377 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python-pytest. 2020-11-20T12:03:02.392 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../097-python-pytest_3.3.2-2_all.deb ... 2020-11-20T12:03:02.435 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-11-20T12:03:02.511 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-cycler. 2020-11-20T12:03:02.528 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../099-python3-cycler_0.10.0-1_all.deb ... 2020-11-20T12:03:02.569 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-11-20T12:03:02.772 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-attr. 2020-11-20T12:03:02.787 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../098-python3-attr_17.4.0-2_all.deb ... 2020-11-20T12:03:02.830 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-11-20T12:03:02.855 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-webencodings. 2020-11-20T12:03:02.871 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../100-python3-webencodings_0.5-2_all.deb ... 2020-11-20T12:03:02.913 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-11-20T12:03:03.116 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-cycler. 2020-11-20T12:03:03.137 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../099-python3-cycler_0.10.0-1_all.deb ... 2020-11-20T12:03:03.174 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-11-20T12:03:03.183 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-html5lib. 2020-11-20T12:03:03.200 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../101-python3-html5lib_0.999999999-1_all.deb ... 2020-11-20T12:03:03.233 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-11-20T12:03:03.519 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-webencodings. 2020-11-20T12:03:03.533 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../100-python3-webencodings_0.5-2_all.deb ... 2020-11-20T12:03:03.569 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-11-20T12:03:03.577 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-11-20T12:03:03.584 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../102-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-11-20T12:03:03.627 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-20T12:03:03.863 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-html5lib. 2020-11-20T12:03:03.881 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../101-python3-html5lib_0.999999999-1_all.deb ... 2020-11-20T12:03:03.930 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-11-20T12:03:04.047 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-tz. 2020-11-20T12:03:04.063 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../103-python3-tz_2018.3-2_all.deb ... 2020-11-20T12:03:04.105 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-tz (2018.3-2) ... 2020-11-20T12:03:04.291 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-11-20T12:03:04.307 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../102-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-11-20T12:03:04.350 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-20T12:03:04.484 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-matplotlib. 2020-11-20T12:03:04.499 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../104-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-11-20T12:03:04.542 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-20T12:03:04.795 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-tz. 2020-11-20T12:03:04.812 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../103-python3-tz_2018.3-2_all.deb ... 2020-11-20T12:03:04.853 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-tz (2018.3-2) ... 2020-11-20T12:03:05.139 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-matplotlib. 2020-11-20T12:03:05.155 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../104-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-11-20T12:03:05.197 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-20T12:03:05.214 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-olefile. 2020-11-20T12:03:05.230 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../105-python3-olefile_0.45.1-1_all.deb ... 2020-11-20T12:03:05.272 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-11-20T12:03:05.601 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pastescript. 2020-11-20T12:03:05.615 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-2_all.deb ... 2020-11-20T12:03:05.659 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-11-20T12:03:05.861 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-olefile. 2020-11-20T12:03:05.876 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../105-python3-olefile_0.45.1-1_all.deb ... 2020-11-20T12:03:05.919 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-11-20T12:03:06.054 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pil:amd64. 2020-11-20T12:03:06.068 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../107-python3-pil_5.1.0-1ubuntu0.3_amd64.deb ... 2020-11-20T12:03:06.120 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-20T12:03:06.230 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pastescript. 2020-11-20T12:03:06.244 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-2_all.deb ... 2020-11-20T12:03:06.289 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-11-20T12:03:06.490 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pluggy. 2020-11-20T12:03:06.504 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../108-python3-pluggy_0.6.0-1_all.deb ... 2020-11-20T12:03:06.540 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-11-20T12:03:06.634 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pil:amd64. 2020-11-20T12:03:06.650 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../107-python3-pil_5.1.0-1ubuntu0.3_amd64.deb ... 2020-11-20T12:03:06.692 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-20T12:03:06.868 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-py. 2020-11-20T12:03:06.885 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../109-python3-py_1.5.2-1_all.deb ... 2020-11-20T12:03:06.935 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-py (1.5.2-1) ... 2020-11-20T12:03:07.053 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pluggy. 2020-11-20T12:03:07.068 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../108-python3-pluggy_0.6.0-1_all.deb ... 2020-11-20T12:03:07.131 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-11-20T12:03:07.288 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pyinotify. 2020-11-20T12:03:07.302 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../110-python3-pyinotify_0.9.6-1_all.deb ... 2020-11-20T12:03:07.346 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-11-20T12:03:07.422 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-py. 2020-11-20T12:03:07.439 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../109-python3-py_1.5.2-1_all.deb ... 2020-11-20T12:03:07.481 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-py (1.5.2-1) ... 2020-11-20T12:03:07.657 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pytest. 2020-11-20T12:03:07.672 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../111-python3-pytest_3.3.2-2_all.deb ... 2020-11-20T12:03:07.715 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-11-20T12:03:07.842 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pyinotify. 2020-11-20T12:03:07.857 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../110-python3-pyinotify_0.9.6-1_all.deb ... 2020-11-20T12:03:07.900 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-11-20T12:03:08.069 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-simplejson. 2020-11-20T12:03:08.085 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../112-python3-simplejson_3.13.2-1_amd64.deb ... 2020-11-20T12:03:08.127 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-11-20T12:03:08.203 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-pytest. 2020-11-20T12:03:08.220 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../111-python3-pytest_3.3.2-2_all.deb ... 2020-11-20T12:03:08.261 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-11-20T12:03:08.480 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-tk:amd64. 2020-11-20T12:03:08.497 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../113-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-11-20T12:03:08.546 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-20T12:03:08.640 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-simplejson. 2020-11-20T12:03:08.655 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../112-python3-simplejson_3.13.2-1_amd64.deb ... 2020-11-20T12:03:08.706 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-11-20T12:03:08.975 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package radosgw. 2020-11-20T12:03:08.993 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../114-radosgw_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:03:09.033 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking radosgw (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:09.076 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package python3-tk:amd64. 2020-11-20T12:03:09.091 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../113-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-11-20T12:03:09.134 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-20T12:03:09.588 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package radosgw. 2020-11-20T12:03:09.603 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../114-radosgw_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:03:09.646 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking radosgw (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:10.045 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package rbd-fuse. 2020-11-20T12:03:10.060 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../115-rbd-fuse_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:03:10.103 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking rbd-fuse (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:10.415 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-11-20T12:03:10.649 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package rbd-fuse. 2020-11-20T12:03:10.663 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../115-rbd-fuse_16.0.0-7397-gb86e691b-1bionic_amd64.deb ... 2020-11-20T12:03:10.707 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking rbd-fuse (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:10.750 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-20T12:03:11.040 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-11-20T12:03:11.118 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-20T12:03:11.244 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-20T12:03:11.362 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-11-20T12:03:11.730 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-11-20T12:03:11.856 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-11-20T12:03:11.871 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-20T12:03:12.273 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-11-20T12:03:12.459 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-11-20T12:03:12.557 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-webencodings (0.5-2) ... 2020-11-20T12:03:12.867 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python-attr (17.4.0-2) ... 2020-11-20T12:03:12.877 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-11-20T12:03:13.068 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-11-20T12:03:13.161 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-webencodings (0.5-2) ... 2020-11-20T12:03:13.194 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-py (1.5.2-1) ... 2020-11-20T12:03:13.437 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python-attr (17.4.0-2) ... 2020-11-20T12:03:13.546 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-11-20T12:03:13.640 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-11-20T12:03:13.764 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-py (1.5.2-1) ... 2020-11-20T12:03:13.906 INFO:teuthology.orchestra.run.smithi047.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-11-20T12:03:14.116 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-11-20T12:03:14.203 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-11-20T12:03:14.499 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-ceph-argparse (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:14.509 INFO:teuthology.orchestra.run.smithi189.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-11-20T12:03:14.809 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-11-20T12:03:14.827 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-11-20T12:03:15.110 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-11-20T12:03:15.128 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-ceph-argparse (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:15.236 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-20T12:03:15.429 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-11-20T12:03:15.747 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-11-20T12:03:15.873 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-11-20T12:03:16.197 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-11-20T12:03:16.540 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-11-20T12:03:16.786 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-11-20T12:03:16.825 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python-py (1.5.2-1) ... 2020-11-20T12:03:17.051 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-11-20T12:03:17.096 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-11-20T12:03:17.389 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python-py (1.5.2-1) ... 2020-11-20T12:03:17.402 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-11-20T12:03:17.492 INFO:teuthology.orchestra.run.smithi047.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-11-20T12:03:17.615 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-11-20T12:03:17.729 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-11-20T12:03:17.855 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-11-20T12:03:17.949 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-11-20T12:03:18.039 INFO:teuthology.orchestra.run.smithi189.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-11-20T12:03:18.056 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-colorama (0.3.7-1) ... 2020-11-20T12:03:18.284 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-11-20T12:03:18.365 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-11-20T12:03:18.410 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-11-20T12:03:18.628 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-colorama (0.3.7-1) ... 2020-11-20T12:03:18.675 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-20T12:03:18.929 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-11-20T12:03:19.001 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-11-20T12:03:19.230 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-11-20T12:03:19.302 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-11-20T12:03:19.548 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-11-20T12:03:19.687 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-11-20T12:03:19.857 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-11-20T12:03:20.004 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-11-20T12:03:20.242 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-11-20T12:03:20.297 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up cephadm (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:20.576 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-11-20T12:03:20.886 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up cephadm (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:20.956 INFO:teuthology.orchestra.run.smithi047.stdout:Adding system user cephadm....done 2020-11-20T12:03:21.095 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-11-20T12:03:21.380 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-20T12:03:21.495 INFO:teuthology.orchestra.run.smithi189.stdout:Adding system user cephadm....done 2020-11-20T12:03:21.505 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up nvme-cli (1.5-1ubuntu1) ... 2020-11-20T12:03:21.631 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-11-20T12:03:21.658 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-11-20T12:03:21.959 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-11-20T12:03:21.974 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-11-20T12:03:22.085 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up nvme-cli (1.5-1ubuntu1) ... 2020-11-20T12:03:22.211 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-11-20T12:03:22.311 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-11-20T12:03:22.436 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-11-20T12:03:22.546 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-11-20T12:03:22.761 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-ceph-common (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:22.822 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-11-20T12:03:22.948 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-11-20T12:03:23.038 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-monotonic (1.1-2) ... 2020-11-20T12:03:23.282 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-ceph-common (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:23.305 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-20T12:03:23.431 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-20T12:03:23.609 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-monotonic (1.1-2) ... 2020-11-20T12:03:23.790 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-11-20T12:03:23.893 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-11-20T12:03:24.019 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-11-20T12:03:24.134 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-20T12:03:24.223 INFO:teuthology.orchestra.run.smithi047.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-11-20T12:03:24.256 INFO:teuthology.orchestra.run.smithi047.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-11-20T12:03:24.370 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-11-20T12:03:24.494 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-11-20T12:03:24.705 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-11-20T12:03:24.778 INFO:teuthology.orchestra.run.smithi189.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-11-20T12:03:24.787 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-20T12:03:24.810 INFO:teuthology.orchestra.run.smithi189.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-11-20T12:03:25.296 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-20T12:03:25.449 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-11-20T12:03:25.614 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-attr (17.4.0-2) ... 2020-11-20T12:03:25.741 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-11-20T12:03:25.923 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-11-20T12:03:26.042 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-11-20T12:03:26.124 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-20T12:03:26.410 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-attr (17.4.0-2) ... 2020-11-20T12:03:26.542 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-11-20T12:03:26.632 INFO:teuthology.orchestra.run.smithi047.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-11-20T12:03:26.719 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-11-20T12:03:26.894 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up liboath0 (2.6.1-1) ... 2020-11-20T12:03:26.895 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-11-20T12:03:27.037 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-20T12:03:27.138 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-20T12:03:27.289 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-11-20T12:03:27.370 INFO:teuthology.orchestra.run.smithi189.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-11-20T12:03:27.372 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-11-20T12:03:27.498 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-11-20T12:03:27.624 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up liboath0 (2.6.1-1) ... 2020-11-20T12:03:27.808 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-11-20T12:03:27.900 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-20T12:03:27.934 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-11-20T12:03:28.026 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-11-20T12:03:28.218 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-11-20T12:03:28.335 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-tz (2018.3-2) ... 2020-11-20T12:03:28.353 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-11-20T12:03:28.628 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-11-20T12:03:28.729 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-11-20T12:03:28.847 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-11-20T12:03:28.904 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-11-20T12:03:29.114 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-tz (2018.3-2) ... 2020-11-20T12:03:29.188 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-joblib (0.11-1) ... 2020-11-20T12:03:29.416 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-11-20T12:03:29.556 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-routes (2.4.1-1) ... 2020-11-20T12:03:29.708 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-11-20T12:03:29.857 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-11-20T12:03:30.009 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-joblib (0.11-1) ... 2020-11-20T12:03:30.175 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-11-20T12:03:30.386 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-routes (2.4.1-1) ... 2020-11-20T12:03:30.510 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pyperclip (1.6.0-1) ... 2020-11-20T12:03:30.687 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-11-20T12:03:30.811 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-20T12:03:30.971 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-11-20T12:03:31.137 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-11-20T12:03:31.272 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pyperclip (1.6.0-1) ... 2020-11-20T12:03:31.422 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up jq (1.5+dfsg-2) ... 2020-11-20T12:03:31.548 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-11-20T12:03:31.557 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2020-11-20T12:03:31.917 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-11-20T12:03:32.226 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up jq (1.5+dfsg-2) ... 2020-11-20T12:03:32.352 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-11-20T12:03:32.753 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-20T12:03:33.055 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-11-20T12:03:33.364 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-11-20T12:03:33.490 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-11-20T12:03:33.524 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-11-20T12:03:33.842 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-11-20T12:03:33.875 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-cmd2 (0.7.9-0ubuntu1) ... 2020-11-20T12:03:34.176 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-11-20T12:03:34.192 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python-pytest (3.3.2-2) ... 2020-11-20T12:03:34.302 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-11-20T12:03:34.452 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-11-20T12:03:34.578 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-11-20T12:03:34.679 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up librados2 (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:34.687 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-cmd2 (0.7.9-0ubuntu1) ... 2020-11-20T12:03:34.780 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-11-20T12:03:34.955 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python-pytest (3.3.2-2) ... 2020-11-20T12:03:35.089 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libcephfs2 (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:35.189 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-11-20T12:03:35.223 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libcephfs-dev (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:35.307 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-11-20T12:03:35.350 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-rados (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:35.433 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up librados2 (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:35.484 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-11-20T12:03:35.561 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-11-20T12:03:35.878 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libcephfs2 (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:35.896 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-11-20T12:03:36.003 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libcephfs-dev (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:36.137 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-rados (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:36.172 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-11-20T12:03:36.246 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-11-20T12:03:36.473 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-20T12:03:36.632 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-11-20T12:03:36.933 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-11-20T12:03:37.251 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-11-20T12:03:37.334 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-11-20T12:03:38.104 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-11-20T12:03:38.179 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-11-20T12:03:38.447 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libradosstriper1 (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:38.573 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-cephfs (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:38.875 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up librgw2 (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:38.925 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-11-20T12:03:39.009 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-fuse (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:39.220 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libradosstriper1 (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:39.228 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-20T12:03:39.228 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-20T12:03:39.343 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-cephfs (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:39.552 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up librbd1 (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:39.653 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up librgw2 (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:39.678 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up cephfs-shell (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:39.770 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-fuse (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:39.787 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-11-20T12:03:39.860 INFO:teuthology.orchestra.run.smithi047.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-11-20T12:03:39.893 INFO:teuthology.orchestra.run.smithi047.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-11-20T12:03:39.966 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-20T12:03:39.967 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-11-20T12:03:40.207 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-20T12:03:40.280 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up librbd1 (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:40.381 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up cephfs-shell (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:40.507 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-11-20T12:03:40.597 INFO:teuthology.orchestra.run.smithi189.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-11-20T12:03:40.637 INFO:teuthology.orchestra.run.smithi189.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-11-20T12:03:40.918 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mgr-modules-core (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:40.935 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-11-20T12:03:41.043 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-11-20T12:03:41.378 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-rgw (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:41.487 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up rbd-fuse (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:41.588 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-rbd (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:41.596 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mgr-modules-core (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:41.689 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-common (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:41.697 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-11-20T12:03:41.998 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-rgw (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:42.116 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up rbd-fuse (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:42.198 INFO:teuthology.orchestra.run.smithi047.stdout:Adding group ceph....done 2020-11-20T12:03:42.241 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up python3-rbd (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:42.367 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-common (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:42.560 INFO:teuthology.orchestra.run.smithi047.stdout:Adding system user ceph....done 2020-11-20T12:03:42.785 INFO:teuthology.orchestra.run.smithi047.stdout:Setting system user ceph properties....done 2020-11-20T12:03:42.876 INFO:teuthology.orchestra.run.smithi189.stdout:Adding group ceph....done 2020-11-20T12:03:42.956 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-11-20T12:03:43.187 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-11-20T12:03:43.212 INFO:teuthology.orchestra.run.smithi189.stdout:Adding system user ceph....done 2020-11-20T12:03:43.455 INFO:teuthology.orchestra.run.smithi189.stdout:Setting system user ceph properties....done 2020-11-20T12:03:43.491 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up radosgw (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:43.616 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-11-20T12:03:43.772 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-20T12:03:43.772 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-20T12:03:43.843 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-11-20T12:03:44.143 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up radosgw (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:44.226 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-test (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:44.377 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-base (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:44.427 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-20T12:03:44.427 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-11-20T12:03:44.717 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-11-20T12:03:44.845 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-test (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:44.957 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-base (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:44.987 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mgr (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:45.206 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-20T12:03:45.207 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-20T12:03:45.253 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-11-20T12:03:45.539 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:45.657 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-osd (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:45.666 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mgr (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:45.884 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-20T12:03:45.884 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-11-20T12:03:46.136 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-20T12:03:46.137 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-20T12:03:46.200 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:46.342 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-osd (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:46.401 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mgr-cephadm (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:46.544 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mgr-rook (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:46.686 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mds (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:46.793 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-20T12:03:46.794 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-11-20T12:03:46.916 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-20T12:03:46.918 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-20T12:03:47.070 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mgr-cephadm (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:47.179 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mon (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:47.213 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mgr-rook (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:47.339 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mds (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:47.379 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-20T12:03:47.380 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-20T12:03:47.539 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-20T12:03:47.539 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-11-20T12:03:47.664 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mgr-k8sevents (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:47.765 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mgr-dashboard (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:47.831 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mon (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:47.891 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:48.019 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for systemd (237-3ubuntu10.42) ... 2020-11-20T12:03:48.040 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-20T12:03:48.040 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-11-20T12:03:48.210 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-11-20T12:03:48.324 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mgr-k8sevents (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:48.450 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph-mgr-dashboard (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:48.593 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ceph (16.0.0-7397-gb86e691b-1bionic) ... 2020-11-20T12:03:48.720 INFO:teuthology.orchestra.run.smithi189.stdout:Processing triggers for systemd (237-3ubuntu10.42) ... 2020-11-20T12:03:48.920 INFO:teuthology.orchestra.run.smithi189.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-11-20T12:03:50.312 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-11-20T12:03:50.354 INFO:teuthology.orchestra.run.smithi047.stdout:ureadahead will be reprofiled on next reboot 2020-11-20T12:03:50.413 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.3) ... 2020-11-20T12:03:50.948 INFO:teuthology.orchestra.run.smithi189.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-11-20T12:03:50.982 INFO:teuthology.orchestra.run.smithi189.stdout:ureadahead will be reprofiled on next reboot 2020-11-20T12:03:51.024 INFO:teuthology.orchestra.run.smithi189.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.3) ... 2020-11-20T12:03:51.117 INFO:teuthology.orchestra.run.smithi047.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-11-20T12:03:51.119 INFO:teuthology.orchestra.run.smithi047:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2020-11-20T12:03:51.195 INFO:teuthology.orchestra.run.smithi047.stdout:Reading package lists... 2020-11-20T12:03:51.458 INFO:teuthology.orchestra.run.smithi189.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-11-20T12:03:51.461 INFO:teuthology.orchestra.run.smithi189:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2020-11-20T12:03:51.462 INFO:teuthology.orchestra.run.smithi047.stdout:Building dependency tree... 2020-11-20T12:03:51.463 INFO:teuthology.orchestra.run.smithi047.stdout:Reading state information... 2020-11-20T12:03:51.543 INFO:teuthology.orchestra.run.smithi189.stdout:Reading package lists... 2020-11-20T12:03:51.648 INFO:teuthology.orchestra.run.smithi047.stdout:libtool-bin is already the newest version (2.4.6-2). 2020-11-20T12:03:51.648 INFO:teuthology.orchestra.run.smithi047.stdout:xfslibs-dev is already the newest version (4.9.0+nmu1ubuntu2). 2020-11-20T12:03:51.648 INFO:teuthology.orchestra.run.smithi047.stdout:iproute2 is already the newest version (4.15.0-2ubuntu1.2). 2020-11-20T12:03:51.648 INFO:teuthology.orchestra.run.smithi047.stdout:util-linux is already the newest version (2.31.1-0.4ubuntu3.7). 2020-11-20T12:03:51.649 INFO:teuthology.orchestra.run.smithi047.stdout:uuid-dev is already the newest version (2.31.1-0.4ubuntu3.7). 2020-11-20T12:03:51.649 INFO:teuthology.orchestra.run.smithi047.stdout:The following additional packages will be installed: 2020-11-20T12:03:51.649 INFO:teuthology.orchestra.run.smithi047.stdout: iputils-arping libbison-dev libfl-dev libfl2 libmbim-glib4 libmbim-proxy 2020-11-20T12:03:51.649 INFO:teuthology.orchestra.run.smithi047.stdout: libmm-glib0 libndp0 libnm0 libpolkit-agent-1-0 libpolkit-backend-1-0 2020-11-20T12:03:51.650 INFO:teuthology.orchestra.run.smithi047.stdout: libqmi-glib5 libqmi-proxy libteamdctl0 modemmanager network-manager-pptp 2020-11-20T12:03:51.650 INFO:teuthology.orchestra.run.smithi047.stdout: policykit-1 ppp pptp-linux usb-modeswitch usb-modeswitch-data wpasupplicant 2020-11-20T12:03:51.650 INFO:teuthology.orchestra.run.smithi047.stdout: zlib1g-dev 2020-11-20T12:03:51.651 INFO:teuthology.orchestra.run.smithi047.stdout:Suggested packages: 2020-11-20T12:03:51.651 INFO:teuthology.orchestra.run.smithi047.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils comgt 2020-11-20T12:03:51.651 INFO:teuthology.orchestra.run.smithi047.stdout: wvdial wpagui libengine-pkcs11-openssl 2020-11-20T12:03:51.688 INFO:teuthology.orchestra.run.smithi047.stdout:The following NEW packages will be installed: 2020-11-20T12:03:51.689 INFO:teuthology.orchestra.run.smithi047.stdout: bison dump flex indent iputils-arping libaio-dev libbison-dev libelf-dev 2020-11-20T12:03:51.689 INFO:teuthology.orchestra.run.smithi047.stdout: libfl-dev libfl2 libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 2020-11-20T12:03:51.690 INFO:teuthology.orchestra.run.smithi047.stdout: libpolkit-agent-1-0 libpolkit-backend-1-0 libqmi-glib5 libqmi-proxy 2020-11-20T12:03:51.690 INFO:teuthology.orchestra.run.smithi047.stdout: libssl-dev libteamdctl0 modemmanager network-manager network-manager-pptp 2020-11-20T12:03:51.690 INFO:teuthology.orchestra.run.smithi047.stdout: policykit-1 ppp pptp-linux usb-modeswitch usb-modeswitch-data wpasupplicant 2020-11-20T12:03:51.690 INFO:teuthology.orchestra.run.smithi047.stdout: zlib1g-dev 2020-11-20T12:03:51.800 INFO:teuthology.orchestra.run.smithi189.stdout:Building dependency tree... 2020-11-20T12:03:51.801 INFO:teuthology.orchestra.run.smithi189.stdout:Reading state information... 2020-11-20T12:03:51.873 INFO:teuthology.orchestra.run.smithi047.stdout:0 upgraded, 31 newly installed, 0 to remove and 20 not upgraded. 2020-11-20T12:03:51.873 INFO:teuthology.orchestra.run.smithi047.stdout:Need to get 7,861 kB of archives. 2020-11-20T12:03:51.873 INFO:teuthology.orchestra.run.smithi047.stdout:After this operation, 32.0 MB of additional disk space will be used. 2020-11-20T12:03:51.873 INFO:teuthology.orchestra.run.smithi047.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 flex amd64 2.6.4-6 [316 kB] 2020-11-20T12:03:51.980 INFO:teuthology.orchestra.run.smithi189.stdout:libtool-bin is already the newest version (2.4.6-2). 2020-11-20T12:03:51.980 INFO:teuthology.orchestra.run.smithi189.stdout:xfslibs-dev is already the newest version (4.9.0+nmu1ubuntu2). 2020-11-20T12:03:51.980 INFO:teuthology.orchestra.run.smithi189.stdout:iproute2 is already the newest version (4.15.0-2ubuntu1.2). 2020-11-20T12:03:51.981 INFO:teuthology.orchestra.run.smithi189.stdout:util-linux is already the newest version (2.31.1-0.4ubuntu3.7). 2020-11-20T12:03:51.981 INFO:teuthology.orchestra.run.smithi189.stdout:uuid-dev is already the newest version (2.31.1-0.4ubuntu3.7). 2020-11-20T12:03:51.981 INFO:teuthology.orchestra.run.smithi189.stdout:The following additional packages will be installed: 2020-11-20T12:03:51.981 INFO:teuthology.orchestra.run.smithi189.stdout: iputils-arping libbison-dev libfl-dev libfl2 libmbim-glib4 libmbim-proxy 2020-11-20T12:03:51.981 INFO:teuthology.orchestra.run.smithi189.stdout: libmm-glib0 libndp0 libnm0 libpolkit-agent-1-0 libpolkit-backend-1-0 2020-11-20T12:03:51.982 INFO:teuthology.orchestra.run.smithi189.stdout: libqmi-glib5 libqmi-proxy libteamdctl0 modemmanager network-manager-pptp 2020-11-20T12:03:51.982 INFO:teuthology.orchestra.run.smithi189.stdout: policykit-1 ppp pptp-linux usb-modeswitch usb-modeswitch-data wpasupplicant 2020-11-20T12:03:51.982 INFO:teuthology.orchestra.run.smithi189.stdout: zlib1g-dev 2020-11-20T12:03:51.983 INFO:teuthology.orchestra.run.smithi189.stdout:Suggested packages: 2020-11-20T12:03:51.983 INFO:teuthology.orchestra.run.smithi189.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils comgt 2020-11-20T12:03:51.983 INFO:teuthology.orchestra.run.smithi189.stdout: wvdial wpagui libengine-pkcs11-openssl 2020-11-20T12:03:52.018 INFO:teuthology.orchestra.run.smithi189.stdout:The following NEW packages will be installed: 2020-11-20T12:03:52.019 INFO:teuthology.orchestra.run.smithi189.stdout: bison dump flex indent iputils-arping libaio-dev libbison-dev libelf-dev 2020-11-20T12:03:52.019 INFO:teuthology.orchestra.run.smithi189.stdout: libfl-dev libfl2 libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 2020-11-20T12:03:52.019 INFO:teuthology.orchestra.run.smithi189.stdout: libpolkit-agent-1-0 libpolkit-backend-1-0 libqmi-glib5 libqmi-proxy 2020-11-20T12:03:52.020 INFO:teuthology.orchestra.run.smithi189.stdout: libssl-dev libteamdctl0 modemmanager network-manager network-manager-pptp 2020-11-20T12:03:52.020 INFO:teuthology.orchestra.run.smithi189.stdout: policykit-1 ppp pptp-linux usb-modeswitch usb-modeswitch-data wpasupplicant 2020-11-20T12:03:52.021 INFO:teuthology.orchestra.run.smithi189.stdout: zlib1g-dev 2020-11-20T12:03:52.199 INFO:teuthology.orchestra.run.smithi189.stdout:0 upgraded, 31 newly installed, 0 to remove and 20 not upgraded. 2020-11-20T12:03:52.199 INFO:teuthology.orchestra.run.smithi189.stdout:Need to get 7,861 kB of archives. 2020-11-20T12:03:52.199 INFO:teuthology.orchestra.run.smithi189.stdout:After this operation, 32.0 MB of additional disk space will be used. 2020-11-20T12:03:52.199 INFO:teuthology.orchestra.run.smithi189.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 flex amd64 2.6.4-6 [316 kB] 2020-11-20T12:03:52.317 INFO:teuthology.orchestra.run.smithi047.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 libbison-dev amd64 2:3.0.4.dfsg-1build1 [339 kB] 2020-11-20T12:03:52.396 INFO:teuthology.orchestra.run.smithi047.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/main amd64 bison amd64 2:3.0.4.dfsg-1build1 [266 kB] 2020-11-20T12:03:52.445 INFO:teuthology.orchestra.run.smithi047.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/universe amd64 dump amd64 0.4b46-3 [129 kB] 2020-11-20T12:03:52.467 INFO:teuthology.orchestra.run.smithi047.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 indent amd64 2.2.11-5 [80.7 kB] 2020-11-20T12:03:52.473 INFO:teuthology.orchestra.run.smithi047.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 iputils-arping amd64 3:20161105-1ubuntu3 [29.7 kB] 2020-11-20T12:03:52.477 INFO:teuthology.orchestra.run.smithi047.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libaio-dev amd64 0.3.110-5ubuntu0.1 [12.8 kB] 2020-11-20T12:03:52.478 INFO:teuthology.orchestra.run.smithi047.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 zlib1g-dev amd64 1:1.2.11.dfsg-0ubuntu2 [176 kB] 2020-11-20T12:03:52.493 INFO:teuthology.orchestra.run.smithi047.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libelf-dev amd64 0.170-0.4ubuntu0.1 [57.3 kB] 2020-11-20T12:03:52.497 INFO:teuthology.orchestra.run.smithi047.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 libfl2 amd64 2.6.4-6 [11.4 kB] 2020-11-20T12:03:52.499 INFO:teuthology.orchestra.run.smithi047.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 libfl-dev amd64 2.6.4-6 [6,320 B] 2020-11-20T12:03:52.499 INFO:teuthology.orchestra.run.smithi047.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libmbim-glib4 amd64 1.18.0-1~ubuntu18.04.1 [91.4 kB] 2020-11-20T12:03:52.557 INFO:teuthology.orchestra.run.smithi047.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libmbim-proxy amd64 1.18.0-1~ubuntu18.04.1 [5,572 B] 2020-11-20T12:03:52.558 INFO:teuthology.orchestra.run.smithi047.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libmm-glib0 amd64 1.10.0-1~ubuntu18.04.2 [179 kB] 2020-11-20T12:03:52.567 INFO:teuthology.orchestra.run.smithi047.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 libndp0 amd64 1.6-1 [10.7 kB] 2020-11-20T12:03:52.626 INFO:teuthology.orchestra.run.smithi189.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 libbison-dev amd64 2:3.0.4.dfsg-1build1 [339 kB] 2020-11-20T12:03:52.637 INFO:teuthology.orchestra.run.smithi047.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libnm0 amd64 1.10.6-2ubuntu1.4 [298 kB] 2020-11-20T12:03:52.649 INFO:teuthology.orchestra.run.smithi047.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpolkit-agent-1-0 amd64 0.105-20ubuntu0.18.04.5 [14.9 kB] 2020-11-20T12:03:52.649 INFO:teuthology.orchestra.run.smithi047.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpolkit-backend-1-0 amd64 0.105-20ubuntu0.18.04.5 [36.3 kB] 2020-11-20T12:03:52.650 INFO:teuthology.orchestra.run.smithi047.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libqmi-glib5 amd64 1.22.0-1.2~ubuntu18.04.1 [494 kB] 2020-11-20T12:03:52.670 INFO:teuthology.orchestra.run.smithi047.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libqmi-proxy amd64 1.22.0-1.2~ubuntu18.04.1 [5,632 B] 2020-11-20T12:03:52.671 INFO:teuthology.orchestra.run.smithi047.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libssl-dev amd64 1.1.1-1ubuntu2.1~18.04.6 [1,566 kB] 2020-11-20T12:03:52.705 INFO:teuthology.orchestra.run.smithi189.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/main amd64 bison amd64 2:3.0.4.dfsg-1build1 [266 kB] 2020-11-20T12:03:52.748 INFO:teuthology.orchestra.run.smithi047.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 libteamdctl0 amd64 1.26-1 [11.6 kB] 2020-11-20T12:03:52.748 INFO:teuthology.orchestra.run.smithi047.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 modemmanager amd64 1.10.0-1~ubuntu18.04.2 [728 kB] 2020-11-20T12:03:52.751 INFO:teuthology.orchestra.run.smithi189.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/universe amd64 dump amd64 0.4b46-3 [129 kB] 2020-11-20T12:03:52.769 INFO:teuthology.orchestra.run.smithi189.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 indent amd64 2.2.11-5 [80.7 kB] 2020-11-20T12:03:52.771 INFO:teuthology.orchestra.run.smithi047.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 wpasupplicant amd64 2:2.6-15ubuntu2.5 [954 kB] 2020-11-20T12:03:52.781 INFO:teuthology.orchestra.run.smithi189.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 iputils-arping amd64 3:20161105-1ubuntu3 [29.7 kB] 2020-11-20T12:03:52.784 INFO:teuthology.orchestra.run.smithi189.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libaio-dev amd64 0.3.110-5ubuntu0.1 [12.8 kB] 2020-11-20T12:03:52.785 INFO:teuthology.orchestra.run.smithi189.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 zlib1g-dev amd64 1:1.2.11.dfsg-0ubuntu2 [176 kB] 2020-11-20T12:03:52.798 INFO:teuthology.orchestra.run.smithi047.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 policykit-1 amd64 0.105-20ubuntu0.18.04.5 [53.5 kB] 2020-11-20T12:03:52.799 INFO:teuthology.orchestra.run.smithi047.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 network-manager amd64 1.10.6-2ubuntu1.4 [1,505 kB] 2020-11-20T12:03:52.799 INFO:teuthology.orchestra.run.smithi189.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libelf-dev amd64 0.170-0.4ubuntu0.1 [57.3 kB] 2020-11-20T12:03:52.804 INFO:teuthology.orchestra.run.smithi189.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 libfl2 amd64 2.6.4-6 [11.4 kB] 2020-11-20T12:03:52.805 INFO:teuthology.orchestra.run.smithi189.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 libfl-dev amd64 2.6.4-6 [6,320 B] 2020-11-20T12:03:52.806 INFO:teuthology.orchestra.run.smithi189.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libmbim-glib4 amd64 1.18.0-1~ubuntu18.04.1 [91.4 kB] 2020-11-20T12:03:52.832 INFO:teuthology.orchestra.run.smithi047.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 ppp amd64 2.4.7-2+2ubuntu1.3 [328 kB] 2020-11-20T12:03:52.857 INFO:teuthology.orchestra.run.smithi047.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 pptp-linux amd64 1.9.0+ds-2 [41.8 kB] 2020-11-20T12:03:52.858 INFO:teuthology.orchestra.run.smithi047.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 network-manager-pptp amd64 1.2.6-1 [30.0 kB] 2020-11-20T12:03:52.859 INFO:teuthology.orchestra.run.smithi047.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 usb-modeswitch-data all 20170806-2 [30.7 kB] 2020-11-20T12:03:52.864 INFO:teuthology.orchestra.run.smithi189.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libmbim-proxy amd64 1.18.0-1~ubuntu18.04.1 [5,572 B] 2020-11-20T12:03:52.864 INFO:teuthology.orchestra.run.smithi189.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libmm-glib0 amd64 1.10.0-1~ubuntu18.04.2 [179 kB] 2020-11-20T12:03:52.873 INFO:teuthology.orchestra.run.smithi189.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 libndp0 amd64 1.6-1 [10.7 kB] 2020-11-20T12:03:52.874 INFO:teuthology.orchestra.run.smithi189.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libnm0 amd64 1.10.6-2ubuntu1.4 [298 kB] 2020-11-20T12:03:52.876 INFO:teuthology.orchestra.run.smithi047.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 usb-modeswitch amd64 2.5.2+repack0-2ubuntu1 [53.6 kB] 2020-11-20T12:03:52.942 INFO:teuthology.orchestra.run.smithi189.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpolkit-agent-1-0 amd64 0.105-20ubuntu0.18.04.5 [14.9 kB] 2020-11-20T12:03:52.943 INFO:teuthology.orchestra.run.smithi189.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpolkit-backend-1-0 amd64 0.105-20ubuntu0.18.04.5 [36.3 kB] 2020-11-20T12:03:52.945 INFO:teuthology.orchestra.run.smithi189.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libqmi-glib5 amd64 1.22.0-1.2~ubuntu18.04.1 [494 kB] 2020-11-20T12:03:52.964 INFO:teuthology.orchestra.run.smithi189.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libqmi-proxy amd64 1.22.0-1.2~ubuntu18.04.1 [5,632 B] 2020-11-20T12:03:52.965 INFO:teuthology.orchestra.run.smithi189.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libssl-dev amd64 1.1.1-1ubuntu2.1~18.04.6 [1,566 kB] 2020-11-20T12:03:53.027 INFO:teuthology.orchestra.run.smithi189.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 libteamdctl0 amd64 1.26-1 [11.6 kB] 2020-11-20T12:03:53.028 INFO:teuthology.orchestra.run.smithi189.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 modemmanager amd64 1.10.0-1~ubuntu18.04.2 [728 kB] 2020-11-20T12:03:53.056 INFO:teuthology.orchestra.run.smithi189.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 wpasupplicant amd64 2:2.6-15ubuntu2.5 [954 kB] 2020-11-20T12:03:53.086 INFO:teuthology.orchestra.run.smithi189.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 policykit-1 amd64 0.105-20ubuntu0.18.04.5 [53.5 kB] 2020-11-20T12:03:53.088 INFO:teuthology.orchestra.run.smithi189.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 network-manager amd64 1.10.6-2ubuntu1.4 [1,505 kB] 2020-11-20T12:03:53.127 INFO:teuthology.orchestra.run.smithi189.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 ppp amd64 2.4.7-2+2ubuntu1.3 [328 kB] 2020-11-20T12:03:53.131 INFO:teuthology.orchestra.run.smithi189.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 pptp-linux amd64 1.9.0+ds-2 [41.8 kB] 2020-11-20T12:03:53.132 INFO:teuthology.orchestra.run.smithi189.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 network-manager-pptp amd64 1.2.6-1 [30.0 kB] 2020-11-20T12:03:53.132 INFO:teuthology.orchestra.run.smithi189.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 usb-modeswitch-data all 20170806-2 [30.7 kB] 2020-11-20T12:03:53.176 INFO:teuthology.orchestra.run.smithi189.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 usb-modeswitch amd64 2.5.2+repack0-2ubuntu1 [53.6 kB] 2020-11-20T12:03:53.409 INFO:teuthology.orchestra.run.smithi047.stdout:Fetched 7,861 kB in 1s (6,723 kB/s) 2020-11-20T12:03:53.553 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package flex. 2020-11-20T12:03:53.606 INFO:teuthology.orchestra.run.smithi047.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 145531 files and directories currently installed.) 2020-11-20T12:03:53.610 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../00-flex_2.6.4-6_amd64.deb ... 2020-11-20T12:03:53.652 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking flex (2.6.4-6) ... 2020-11-20T12:03:53.652 INFO:teuthology.orchestra.run.smithi189.stdout:Fetched 7,861 kB in 1s (6,897 kB/s) 2020-11-20T12:03:53.797 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package flex. 2020-11-20T12:03:53.852 INFO:teuthology.orchestra.run.smithi189.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 145531 files and directories currently installed.) 2020-11-20T12:03:53.856 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../00-flex_2.6.4-6_amd64.deb ... 2020-11-20T12:03:53.904 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking flex (2.6.4-6) ... 2020-11-20T12:03:54.039 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libbison-dev:amd64. 2020-11-20T12:03:54.056 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../01-libbison-dev_2%3a3.0.4.dfsg-1build1_amd64.deb ... 2020-11-20T12:03:54.097 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libbison-dev:amd64 (2:3.0.4.dfsg-1build1) ... 2020-11-20T12:03:54.249 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libbison-dev:amd64. 2020-11-20T12:03:54.265 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../01-libbison-dev_2%3a3.0.4.dfsg-1build1_amd64.deb ... 2020-11-20T12:03:54.299 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libbison-dev:amd64 (2:3.0.4.dfsg-1build1) ... 2020-11-20T12:03:54.400 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package bison. 2020-11-20T12:03:54.414 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../02-bison_2%3a3.0.4.dfsg-1build1_amd64.deb ... 2020-11-20T12:03:54.465 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking bison (2:3.0.4.dfsg-1build1) ... 2020-11-20T12:03:54.627 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package bison. 2020-11-20T12:03:54.644 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../02-bison_2%3a3.0.4.dfsg-1build1_amd64.deb ... 2020-11-20T12:03:54.714 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking bison (2:3.0.4.dfsg-1build1) ... 2020-11-20T12:03:54.847 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package dump. 2020-11-20T12:03:54.862 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../03-dump_0.4b46-3_amd64.deb ... 2020-11-20T12:03:54.905 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking dump (0.4b46-3) ... 2020-11-20T12:03:55.096 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package dump. 2020-11-20T12:03:55.113 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../03-dump_0.4b46-3_amd64.deb ... 2020-11-20T12:03:55.155 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking dump (0.4b46-3) ... 2020-11-20T12:03:55.175 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package indent. 2020-11-20T12:03:55.192 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../04-indent_2.2.11-5_amd64.deb ... 2020-11-20T12:03:55.225 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking indent (2.2.11-5) ... 2020-11-20T12:03:55.441 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package indent. 2020-11-20T12:03:55.457 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../04-indent_2.2.11-5_amd64.deb ... 2020-11-20T12:03:55.492 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking indent (2.2.11-5) ... 2020-11-20T12:03:55.520 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package iputils-arping. 2020-11-20T12:03:55.536 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../05-iputils-arping_3%3a20161105-1ubuntu3_amd64.deb ... 2020-11-20T12:03:55.578 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking iputils-arping (3:20161105-1ubuntu3) ... 2020-11-20T12:03:55.830 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libaio-dev:amd64. 2020-11-20T12:03:55.847 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../06-libaio-dev_0.3.110-5ubuntu0.1_amd64.deb ... 2020-11-20T12:03:56.306 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libaio-dev:amd64 (0.3.110-5ubuntu0.1) ... 2020-11-20T12:03:56.388 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package iputils-arping. 2020-11-20T12:03:56.400 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../05-iputils-arping_3%3a20161105-1ubuntu3_amd64.deb ... 2020-11-20T12:03:56.445 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking iputils-arping (3:20161105-1ubuntu3) ... 2020-11-20T12:03:56.550 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package zlib1g-dev:amd64. 2020-11-20T12:03:56.566 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../07-zlib1g-dev_1%3a1.2.11.dfsg-0ubuntu2_amd64.deb ... 2020-11-20T12:03:56.601 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking zlib1g-dev:amd64 (1:1.2.11.dfsg-0ubuntu2) ... 2020-11-20T12:03:56.697 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libaio-dev:amd64. 2020-11-20T12:03:56.714 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../06-libaio-dev_0.3.110-5ubuntu0.1_amd64.deb ... 2020-11-20T12:03:56.755 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libaio-dev:amd64 (0.3.110-5ubuntu0.1) ... 2020-11-20T12:03:56.887 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libelf-dev:amd64. 2020-11-20T12:03:56.902 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../08-libelf-dev_0.170-0.4ubuntu0.1_amd64.deb ... 2020-11-20T12:03:56.945 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libelf-dev:amd64 (0.170-0.4ubuntu0.1) ... 2020-11-20T12:03:57.008 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package zlib1g-dev:amd64. 2020-11-20T12:03:57.024 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../07-zlib1g-dev_1%3a1.2.11.dfsg-0ubuntu2_amd64.deb ... 2020-11-20T12:03:57.066 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking zlib1g-dev:amd64 (1:1.2.11.dfsg-0ubuntu2) ... 2020-11-20T12:03:57.281 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libfl2:amd64. 2020-11-20T12:03:57.298 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../09-libfl2_2.6.4-6_amd64.deb ... 2020-11-20T12:03:57.374 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libelf-dev:amd64. 2020-11-20T12:03:57.382 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libfl2:amd64 (2.6.4-6) ... 2020-11-20T12:03:57.385 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../08-libelf-dev_0.170-0.4ubuntu0.1_amd64.deb ... 2020-11-20T12:03:57.428 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libelf-dev:amd64 (0.170-0.4ubuntu0.1) ... 2020-11-20T12:03:57.642 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libfl-dev:amd64. 2020-11-20T12:03:57.659 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../10-libfl-dev_2.6.4-6_amd64.deb ... 2020-11-20T12:03:57.700 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libfl-dev:amd64 (2.6.4-6) ... 2020-11-20T12:03:57.767 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libfl2:amd64. 2020-11-20T12:03:57.780 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../09-libfl2_2.6.4-6_amd64.deb ... 2020-11-20T12:03:57.869 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libfl2:amd64 (2.6.4-6) ... 2020-11-20T12:03:57.973 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2020-11-20T12:03:57.989 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../11-libmbim-glib4_1.18.0-1~ubuntu18.04.1_amd64.deb ... 2020-11-20T12:03:58.021 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libmbim-glib4:amd64 (1.18.0-1~ubuntu18.04.1) ... 2020-11-20T12:03:58.117 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libfl-dev:amd64. 2020-11-20T12:03:58.135 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../10-libfl-dev_2.6.4-6_amd64.deb ... 2020-11-20T12:03:58.175 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libfl-dev:amd64 (2.6.4-6) ... 2020-11-20T12:03:58.290 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libmbim-proxy. 2020-11-20T12:03:58.305 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../12-libmbim-proxy_1.18.0-1~ubuntu18.04.1_amd64.deb ... 2020-11-20T12:03:58.356 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libmbim-proxy (1.18.0-1~ubuntu18.04.1) ... 2020-11-20T12:03:58.501 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2020-11-20T12:03:58.508 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../11-libmbim-glib4_1.18.0-1~ubuntu18.04.1_amd64.deb ... 2020-11-20T12:03:58.584 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libmbim-glib4:amd64 (1.18.0-1~ubuntu18.04.1) ... 2020-11-20T12:03:58.662 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libmm-glib0:amd64. 2020-11-20T12:03:58.677 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../13-libmm-glib0_1.10.0-1~ubuntu18.04.2_amd64.deb ... 2020-11-20T12:03:58.719 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libmm-glib0:amd64 (1.10.0-1~ubuntu18.04.2) ... 2020-11-20T12:03:58.857 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libmbim-proxy. 2020-11-20T12:03:58.871 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../12-libmbim-proxy_1.18.0-1~ubuntu18.04.1_amd64.deb ... 2020-11-20T12:03:58.914 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libmbim-proxy (1.18.0-1~ubuntu18.04.1) ... 2020-11-20T12:03:59.028 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libndp0:amd64. 2020-11-20T12:03:59.046 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../14-libndp0_1.6-1_amd64.deb ... 2020-11-20T12:03:59.079 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libndp0:amd64 (1.6-1) ... 2020-11-20T12:03:59.242 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libmm-glib0:amd64. 2020-11-20T12:03:59.255 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../13-libmm-glib0_1.10.0-1~ubuntu18.04.2_amd64.deb ... 2020-11-20T12:03:59.299 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libmm-glib0:amd64 (1.10.0-1~ubuntu18.04.2) ... 2020-11-20T12:03:59.415 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libnm0:amd64. 2020-11-20T12:03:59.430 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../15-libnm0_1.10.6-2ubuntu1.4_amd64.deb ... 2020-11-20T12:03:59.475 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libnm0:amd64 (1.10.6-2ubuntu1.4) ... 2020-11-20T12:03:59.627 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libndp0:amd64. 2020-11-20T12:03:59.641 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../14-libndp0_1.6-1_amd64.deb ... 2020-11-20T12:03:59.686 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libndp0:amd64 (1.6-1) ... 2020-11-20T12:03:59.851 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libpolkit-agent-1-0:amd64. 2020-11-20T12:03:59.866 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../16-libpolkit-agent-1-0_0.105-20ubuntu0.18.04.5_amd64.deb ... 2020-11-20T12:03:59.911 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libpolkit-agent-1-0:amd64 (0.105-20ubuntu0.18.04.5) ... 2020-11-20T12:04:00.022 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libnm0:amd64. 2020-11-20T12:04:00.035 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../15-libnm0_1.10.6-2ubuntu1.4_amd64.deb ... 2020-11-20T12:04:00.080 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libnm0:amd64 (1.10.6-2ubuntu1.4) ... 2020-11-20T12:04:00.203 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libpolkit-backend-1-0:amd64. 2020-11-20T12:04:00.225 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../17-libpolkit-backend-1-0_0.105-20ubuntu0.18.04.5_amd64.deb ... 2020-11-20T12:04:00.253 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libpolkit-backend-1-0:amd64 (0.105-20ubuntu0.18.04.5) ... 2020-11-20T12:04:00.458 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libpolkit-agent-1-0:amd64. 2020-11-20T12:04:00.473 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../16-libpolkit-agent-1-0_0.105-20ubuntu0.18.04.5_amd64.deb ... 2020-11-20T12:04:00.516 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libpolkit-agent-1-0:amd64 (0.105-20ubuntu0.18.04.5) ... 2020-11-20T12:04:00.565 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2020-11-20T12:04:00.579 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../18-libqmi-glib5_1.22.0-1.2~ubuntu18.04.1_amd64.deb ... 2020-11-20T12:04:00.623 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libqmi-glib5:amd64 (1.22.0-1.2~ubuntu18.04.1) ... 2020-11-20T12:04:00.835 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libpolkit-backend-1-0:amd64. 2020-11-20T12:04:00.851 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../17-libpolkit-backend-1-0_0.105-20ubuntu0.18.04.5_amd64.deb ... 2020-11-20T12:04:00.885 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libpolkit-backend-1-0:amd64 (0.105-20ubuntu0.18.04.5) ... 2020-11-20T12:04:00.950 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libqmi-proxy. 2020-11-20T12:04:00.966 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../19-libqmi-proxy_1.22.0-1.2~ubuntu18.04.1_amd64.deb ... 2020-11-20T12:04:01.076 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libqmi-proxy (1.22.0-1.2~ubuntu18.04.1) ... 2020-11-20T12:04:01.305 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2020-11-20T12:04:01.321 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../18-libqmi-glib5_1.22.0-1.2~ubuntu18.04.1_amd64.deb ... 2020-11-20T12:04:01.321 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libssl-dev:amd64. 2020-11-20T12:04:01.338 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../20-libssl-dev_1.1.1-1ubuntu2.1~18.04.6_amd64.deb ... 2020-11-20T12:04:01.365 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libqmi-glib5:amd64 (1.22.0-1.2~ubuntu18.04.1) ... 2020-11-20T12:04:01.370 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libssl-dev:amd64 (1.1.1-1ubuntu2.1~18.04.6) ... 2020-11-20T12:04:01.683 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libqmi-proxy. 2020-11-20T12:04:01.698 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../19-libqmi-proxy_1.22.0-1.2~ubuntu18.04.1_amd64.deb ... 2020-11-20T12:04:01.741 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libqmi-proxy (1.22.0-1.2~ubuntu18.04.1) ... 2020-11-20T12:04:01.882 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libteamdctl0:amd64. 2020-11-20T12:04:01.897 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../21-libteamdctl0_1.26-1_amd64.deb ... 2020-11-20T12:04:01.932 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libteamdctl0:amd64 (1.26-1) ... 2020-11-20T12:04:01.977 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libssl-dev:amd64. 2020-11-20T12:04:01.991 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../20-libssl-dev_1.1.1-1ubuntu2.1~18.04.6_amd64.deb ... 2020-11-20T12:04:02.035 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libssl-dev:amd64 (1.1.1-1ubuntu2.1~18.04.6) ... 2020-11-20T12:04:02.305 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package modemmanager. 2020-11-20T12:04:02.321 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../22-modemmanager_1.10.0-1~ubuntu18.04.2_amd64.deb ... 2020-11-20T12:04:02.369 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking modemmanager (1.10.0-1~ubuntu18.04.2) ... 2020-11-20T12:04:02.557 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package libteamdctl0:amd64. 2020-11-20T12:04:02.571 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../21-libteamdctl0_1.26-1_amd64.deb ... 2020-11-20T12:04:02.614 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking libteamdctl0:amd64 (1.26-1) ... 2020-11-20T12:04:02.990 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package wpasupplicant. 2020-11-20T12:04:02.991 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package modemmanager. 2020-11-20T12:04:03.011 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../22-modemmanager_1.10.0-1~ubuntu18.04.2_amd64.deb ... 2020-11-20T12:04:03.012 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../23-wpasupplicant_2%3a2.6-15ubuntu2.5_amd64.deb ... 2020-11-20T12:04:03.059 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking modemmanager (1.10.0-1~ubuntu18.04.2) ... 2020-11-20T12:04:03.174 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking wpasupplicant (2:2.6-15ubuntu2.5) ... 2020-11-20T12:04:03.630 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package wpasupplicant. 2020-11-20T12:04:03.649 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../23-wpasupplicant_2%3a2.6-15ubuntu2.5_amd64.deb ... 2020-11-20T12:04:03.662 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package policykit-1. 2020-11-20T12:04:03.678 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../24-policykit-1_0.105-20ubuntu0.18.04.5_amd64.deb ... 2020-11-20T12:04:03.724 INFO:teuthology.orchestra.run.smithi047.stdout:Unit polkit.service does not exist, proceeding anyway. 2020-11-20T12:04:03.725 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /run/systemd/system/polkit.service -> /dev/null. 2020-11-20T12:04:03.828 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking policykit-1 (0.105-20ubuntu0.18.04.5) ... 2020-11-20T12:04:03.828 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking wpasupplicant (2:2.6-15ubuntu2.5) ... 2020-11-20T12:04:04.237 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package network-manager. 2020-11-20T12:04:04.251 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../25-network-manager_1.10.6-2ubuntu1.4_amd64.deb ... 2020-11-20T12:04:04.318 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking network-manager (1.10.6-2ubuntu1.4) ... 2020-11-20T12:04:04.377 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package policykit-1. 2020-11-20T12:04:04.391 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../24-policykit-1_0.105-20ubuntu0.18.04.5_amd64.deb ... 2020-11-20T12:04:04.446 INFO:teuthology.orchestra.run.smithi189.stdout:Unit polkit.service does not exist, proceeding anyway. 2020-11-20T12:04:04.447 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /run/systemd/system/polkit.service -> /dev/null. 2020-11-20T12:04:04.564 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking policykit-1 (0.105-20ubuntu0.18.04.5) ... 2020-11-20T12:04:04.924 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ppp. 2020-11-20T12:04:04.940 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../26-ppp_2.4.7-2+2ubuntu1.3_amd64.deb ... 2020-11-20T12:04:04.949 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package network-manager. 2020-11-20T12:04:04.963 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../25-network-manager_1.10.6-2ubuntu1.4_amd64.deb ... 2020-11-20T12:04:05.036 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking network-manager (1.10.6-2ubuntu1.4) ... 2020-11-20T12:04:05.237 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ppp (2.4.7-2+2ubuntu1.3) ... 2020-11-20T12:04:05.586 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package pptp-linux. 2020-11-20T12:04:05.602 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../27-pptp-linux_1.9.0+ds-2_amd64.deb ... 2020-11-20T12:04:05.645 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking pptp-linux (1.9.0+ds-2) ... 2020-11-20T12:04:05.680 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package ppp. 2020-11-20T12:04:05.695 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../26-ppp_2.4.7-2+2ubuntu1.3_amd64.deb ... 2020-11-20T12:04:05.986 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking ppp (2.4.7-2+2ubuntu1.3) ... 2020-11-20T12:04:06.040 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package network-manager-pptp. 2020-11-20T12:04:06.054 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../28-network-manager-pptp_1.2.6-1_amd64.deb ... 2020-11-20T12:04:06.108 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking network-manager-pptp (1.2.6-1) ... 2020-11-20T12:04:06.460 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package usb-modeswitch-data. 2020-11-20T12:04:06.475 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../29-usb-modeswitch-data_20170806-2_all.deb ... 2020-11-20T12:04:06.502 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package pptp-linux. 2020-11-20T12:04:06.518 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../27-pptp-linux_1.9.0+ds-2_amd64.deb ... 2020-11-20T12:04:06.519 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking usb-modeswitch-data (20170806-2) ... 2020-11-20T12:04:06.560 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking pptp-linux (1.9.0+ds-2) ... 2020-11-20T12:04:06.939 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package network-manager-pptp. 2020-11-20T12:04:06.946 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package usb-modeswitch. 2020-11-20T12:04:06.954 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../28-network-manager-pptp_1.2.6-1_amd64.deb ... 2020-11-20T12:04:06.962 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../30-usb-modeswitch_2.5.2+repack0-2ubuntu1_amd64.deb ... 2020-11-20T12:04:07.006 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking network-manager-pptp (1.2.6-1) ... 2020-11-20T12:04:07.014 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking usb-modeswitch (2.5.2+repack0-2ubuntu1) ... 2020-11-20T12:04:07.258 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package usb-modeswitch-data. 2020-11-20T12:04:07.275 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../29-usb-modeswitch-data_20170806-2_all.deb ... 2020-11-20T12:04:07.308 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking usb-modeswitch-data (20170806-2) ... 2020-11-20T12:04:07.314 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libmbim-glib4:amd64 (1.18.0-1~ubuntu18.04.1) ... 2020-11-20T12:04:07.460 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ppp (2.4.7-2+2ubuntu1.3) ... 2020-11-20T12:04:07.653 INFO:teuthology.orchestra.run.smithi189.stdout:Selecting previously unselected package usb-modeswitch. 2020-11-20T12:04:07.667 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing to unpack .../30-usb-modeswitch_2.5.2+repack0-2ubuntu1_amd64.deb ... 2020-11-20T12:04:07.722 INFO:teuthology.orchestra.run.smithi189.stdout:Unpacking usb-modeswitch (2.5.2+repack0-2ubuntu1) ... 2020-11-20T12:04:08.015 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libmbim-glib4:amd64 (1.18.0-1~ubuntu18.04.1) ... 2020-11-20T12:04:08.160 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up ppp (2.4.7-2+2ubuntu1.3) ... 2020-11-20T12:04:08.371 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/pppd-dns.service -> /lib/systemd/system/pppd-dns.service. 2020-11-20T12:04:08.532 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up flex (2.6.4-6) ... 2020-11-20T12:04:08.659 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libpolkit-agent-1-0:amd64 (0.105-20ubuntu0.18.04.5) ... 2020-11-20T12:04:08.785 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up dump (0.4b46-3) ... 2020-11-20T12:04:08.905 INFO:teuthology.orchestra.run.smithi047.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2020-11-20T12:04:08.961 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libnm0:amd64 (1.10.6-2ubuntu1.4) ... 2020-11-20T12:04:08.966 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/pppd-dns.service -> /lib/systemd/system/pppd-dns.service. 2020-11-20T12:04:09.087 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libssl-dev:amd64 (1.1.1-1ubuntu2.1~18.04.6) ... 2020-11-20T12:04:09.133 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up flex (2.6.4-6) ... 2020-11-20T12:04:09.213 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up pptp-linux (1.9.0+ds-2) ... 2020-11-20T12:04:09.259 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libpolkit-agent-1-0:amd64 (0.105-20ubuntu0.18.04.5) ... 2020-11-20T12:04:09.368 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up dump (0.4b46-3) ... 2020-11-20T12:04:09.382 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up wpasupplicant (2:2.6-15ubuntu2.5) ... 2020-11-20T12:04:09.451 INFO:teuthology.orchestra.run.smithi189.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2020-11-20T12:04:09.511 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libnm0:amd64 (1.10.6-2ubuntu1.4) ... 2020-11-20T12:04:09.629 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libssl-dev:amd64 (1.1.1-1ubuntu2.1~18.04.6) ... 2020-11-20T12:04:09.757 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up pptp-linux (1.9.0+ds-2) ... 2020-11-20T12:04:09.774 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2020-11-20T12:04:09.774 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2020-11-20T12:04:09.924 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up wpasupplicant (2:2.6-15ubuntu2.5) ... 2020-11-20T12:04:10.068 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libmbim-proxy (1.18.0-1~ubuntu18.04.1) ... 2020-11-20T12:04:10.211 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up usb-modeswitch-data (20170806-2) ... 2020-11-20T12:04:10.335 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2020-11-20T12:04:10.336 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2020-11-20T12:04:10.353 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up usb-modeswitch (2.5.2+repack0-2ubuntu1) ... 2020-11-20T12:04:10.538 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libpolkit-backend-1-0:amd64 (0.105-20ubuntu0.18.04.5) ... 2020-11-20T12:04:10.643 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libmbim-proxy (1.18.0-1~ubuntu18.04.1) ... 2020-11-20T12:04:10.664 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libmm-glib0:amd64 (1.10.0-1~ubuntu18.04.2) ... 2020-11-20T12:04:10.744 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up usb-modeswitch-data (20170806-2) ... 2020-11-20T12:04:10.790 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up indent (2.2.11-5) ... 2020-11-20T12:04:10.845 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up usb-modeswitch (2.5.2+repack0-2ubuntu1) ... 2020-11-20T12:04:10.916 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libbison-dev:amd64 (2:3.0.4.dfsg-1build1) ... 2020-11-20T12:04:11.004 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libpolkit-backend-1-0:amd64 (0.105-20ubuntu0.18.04.5) ... 2020-11-20T12:04:11.042 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libfl2:amd64 (2.6.4-6) ... 2020-11-20T12:04:11.139 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libmm-glib0:amd64 (1.10.0-1~ubuntu18.04.2) ... 2020-11-20T12:04:11.168 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up bison (2:3.0.4.dfsg-1build1) ... 2020-11-20T12:04:11.265 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up indent (2.2.11-5) ... 2020-11-20T12:04:11.443 INFO:teuthology.orchestra.run.smithi047.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2020-11-20T12:04:11.513 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libaio-dev:amd64 (0.3.110-5ubuntu0.1) ... 2020-11-20T12:04:11.526 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libbison-dev:amd64 (2:3.0.4.dfsg-1build1) ... 2020-11-20T12:04:11.635 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libfl2:amd64 (2.6.4-6) ... 2020-11-20T12:04:11.639 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libndp0:amd64 (1.6-1) ... 2020-11-20T12:04:11.736 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up bison (2:3.0.4.dfsg-1build1) ... 2020-11-20T12:04:11.765 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up iputils-arping (3:20161105-1ubuntu3) ... 2020-11-20T12:04:11.809 INFO:teuthology.orchestra.run.smithi189.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2020-11-20T12:04:11.891 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libteamdctl0:amd64 (1.26-1) ... 2020-11-20T12:04:11.900 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libaio-dev:amd64 (0.3.110-5ubuntu0.1) ... 2020-11-20T12:04:12.016 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up zlib1g-dev:amd64 (1:1.2.11.dfsg-0ubuntu2) ... 2020-11-20T12:04:12.039 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libndp0:amd64 (1.6-1) ... 2020-11-20T12:04:12.142 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up policykit-1 (0.105-20ubuntu0.18.04.5) ... 2020-11-20T12:04:12.165 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up iputils-arping (3:20161105-1ubuntu3) ... 2020-11-20T12:04:12.291 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libteamdctl0:amd64 (1.26-1) ... 2020-11-20T12:04:12.417 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up zlib1g-dev:amd64 (1:1.2.11.dfsg-0ubuntu2) ... 2020-11-20T12:04:12.519 INFO:teuthology.orchestra.run.smithi047.stdout:Removed /run/systemd/system/polkit.service. 2020-11-20T12:04:12.542 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up policykit-1 (0.105-20ubuntu0.18.04.5) ... 2020-11-20T12:04:12.775 INFO:teuthology.orchestra.run.smithi047.stdout:polkit.service is a disabled or a static unit not running, not starting it. 2020-11-20T12:04:12.832 INFO:teuthology.orchestra.run.smithi189.stdout:Removed /run/systemd/system/polkit.service. 2020-11-20T12:04:12.915 INFO:teuthology.orchestra.run.smithi047.stdout:polkit.service is a disabled or a static unit, not starting it. 2020-11-20T12:04:12.963 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libqmi-glib5:amd64 (1.22.0-1.2~ubuntu18.04.1) ... 2020-11-20T12:04:13.089 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libelf-dev:amd64 (0.170-0.4ubuntu0.1) ... 2020-11-20T12:04:13.096 INFO:teuthology.orchestra.run.smithi189.stdout:polkit.service is a disabled or a static unit not running, not starting it. 2020-11-20T12:04:13.215 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libqmi-proxy (1.22.0-1.2~ubuntu18.04.1) ... 2020-11-20T12:04:13.233 INFO:teuthology.orchestra.run.smithi189.stdout:polkit.service is a disabled or a static unit, not starting it. 2020-11-20T12:04:13.279 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libqmi-glib5:amd64 (1.22.0-1.2~ubuntu18.04.1) ... 2020-11-20T12:04:13.341 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libfl-dev:amd64 (2.6.4-6) ... 2020-11-20T12:04:13.422 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libelf-dev:amd64 (0.170-0.4ubuntu0.1) ... 2020-11-20T12:04:13.467 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up network-manager (1.10.6-2ubuntu1.4) ... 2020-11-20T12:04:13.548 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libqmi-proxy (1.22.0-1.2~ubuntu18.04.1) ... 2020-11-20T12:04:13.674 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up libfl-dev:amd64 (2.6.4-6) ... 2020-11-20T12:04:13.816 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up network-manager (1.10.6-2ubuntu1.4) ... 2020-11-20T12:04:14.105 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:14.106 INFO:teuthology.orchestra.run.smithi047.stdout:The following network interfaces were found in /etc/network/interfaces 2020-11-20T12:04:14.106 INFO:teuthology.orchestra.run.smithi047.stdout:which means they are currently configured by ifupdown: 2020-11-20T12:04:14.106 INFO:teuthology.orchestra.run.smithi047.stdout:- ens1f0 2020-11-20T12:04:14.106 INFO:teuthology.orchestra.run.smithi047.stdout:If you want to manage those interfaces with NetworkManager instead 2020-11-20T12:04:14.106 INFO:teuthology.orchestra.run.smithi047.stdout:remove their configuration from /etc/network/interfaces. 2020-11-20T12:04:14.107 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:14.237 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2020-11-20T12:04:14.436 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:14.436 INFO:teuthology.orchestra.run.smithi189.stdout:The following network interfaces were found in /etc/network/interfaces 2020-11-20T12:04:14.436 INFO:teuthology.orchestra.run.smithi189.stdout:which means they are currently configured by ifupdown: 2020-11-20T12:04:14.437 INFO:teuthology.orchestra.run.smithi189.stdout:- enp3s0f1 2020-11-20T12:04:14.437 INFO:teuthology.orchestra.run.smithi189.stdout:If you want to manage those interfaces with NetworkManager instead 2020-11-20T12:04:14.437 INFO:teuthology.orchestra.run.smithi189.stdout:remove their configuration from /etc/network/interfaces. 2020-11-20T12:04:14.438 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:14.462 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2020-11-20T12:04:14.568 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2020-11-20T12:04:14.707 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2020-11-20T12:04:14.786 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2020-11-20T12:04:15.035 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2020-11-20T12:04:15.246 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up network-manager-pptp (1.2.6-1) ... 2020-11-20T12:04:15.422 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up modemmanager (1.10.0-1~ubuntu18.04.2) ... 2020-11-20T12:04:15.570 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up network-manager-pptp (1.2.6-1) ... 2020-11-20T12:04:15.682 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2020-11-20T12:04:15.682 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2020-11-20T12:04:15.764 INFO:teuthology.orchestra.run.smithi189.stdout:Setting up modemmanager (1.10.0-1~ubuntu18.04.2) ... 2020-11-20T12:04:16.025 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2020-11-20T12:04:16.026 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2020-11-20T12:04:16.058 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for systemd (237-3ubuntu10.42) ... 2020-11-20T12:04:16.268 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-11-20T12:04:16.375 INFO:teuthology.orchestra.run.smithi189.stdout:Processing triggers for systemd (237-3ubuntu10.42) ... 2020-11-20T12:04:16.593 INFO:teuthology.orchestra.run.smithi189.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-11-20T12:04:18.787 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for dbus (1.12.2-1ubuntu1.2) ... 2020-11-20T12:04:18.887 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2020-11-20T12:04:19.129 INFO:teuthology.orchestra.run.smithi189.stdout:Processing triggers for dbus (1.12.2-1ubuntu1.2) ... 2020-11-20T12:04:19.204 INFO:teuthology.orchestra.run.smithi189.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2020-11-20T12:04:19.214 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-11-20T12:04:19.314 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for install-info (6.5.0.dfsg.1-2) ... 2020-11-20T12:04:19.581 INFO:teuthology.orchestra.run.smithi189.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-11-20T12:04:19.591 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.3) ... 2020-11-20T12:04:19.698 INFO:teuthology.orchestra.run.smithi189.stdout:Processing triggers for install-info (6.5.0.dfsg.1-2) ... 2020-11-20T12:04:19.949 INFO:teuthology.orchestra.run.smithi189.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.3) ... 2020-11-20T12:04:20.020 INFO:teuthology.orchestra.run.smithi047.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-11-20T12:04:20.023 DEBUG:teuthology.parallel:result is None 2020-11-20T12:04:20.413 INFO:teuthology.orchestra.run.smithi189.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-11-20T12:04:20.416 DEBUG:teuthology.parallel:result is None 2020-11-20T12:04:20.416 INFO:teuthology.orchestra.run.smithi189:> sudo lsb_release -is 2020-11-20T12:04:20.462 INFO:teuthology.orchestra.run.smithi189.stdout:Ubuntu 2020-11-20T12:04:20.466 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-20T12:04:20.467 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=b86e691bf191566922a8f08e2a5abf64fde8cfcb 2020-11-20T12:04:20.631 INFO:teuthology.orchestra.run.smithi189:> dpkg-query -W -f '${Version}' ceph 2020-11-20T12:04:20.655 INFO:teuthology.orchestra.run.smithi189.stdout:16.0.0-7397-gb86e691b-1bionic 2020-11-20T12:04:20.656 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-7397-gb86e691b-1bionic 2020-11-20T12:04:20.656 INFO:teuthology.task.install:The correct ceph version 16.0.0-7397-gb86e691b-1bionic is installed. 2020-11-20T12:04:20.657 INFO:teuthology.orchestra.run.smithi047:> sudo lsb_release -is 2020-11-20T12:04:20.712 INFO:teuthology.orchestra.run.smithi047.stdout:Ubuntu 2020-11-20T12:04:20.717 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-11-20T12:04:20.717 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=b86e691bf191566922a8f08e2a5abf64fde8cfcb 2020-11-20T12:04:20.880 INFO:teuthology.orchestra.run.smithi047:> dpkg-query -W -f '${Version}' ceph 2020-11-20T12:04:20.906 INFO:teuthology.orchestra.run.smithi047.stdout:16.0.0-7397-gb86e691b-1bionic 2020-11-20T12:04:20.906 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-7397-gb86e691b-1bionic 2020-11-20T12:04:20.907 INFO:teuthology.task.install:The correct ceph version 16.0.0-7397-gb86e691b-1bionic is installed. 2020-11-20T12:04:20.907 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-11-20T12:04:20.908 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-20T12:04:20.908 INFO:teuthology.orchestra.run.smithi189:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-20T12:04:20.914 INFO:teuthology.orchestra.run.smithi189.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-20T12:04:20.924 INFO:teuthology.orchestra.run.smithi189.stderr:24+1 records in 2020-11-20T12:04:20.924 INFO:teuthology.orchestra.run.smithi189.stderr:24+1 records out 2020-11-20T12:04:20.924 INFO:teuthology.orchestra.run.smithi189.stderr:12667 bytes (13 kB, 12 KiB) copied, 0.000189395 s, 66.9 MB/s 2020-11-20T12:04:20.925 INFO:teuthology.orchestra.run.smithi047:> set -ex 2020-11-20T12:04:20.925 INFO:teuthology.orchestra.run.smithi047:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-20T12:04:20.952 INFO:teuthology.orchestra.run.smithi047.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-20T12:04:20.960 INFO:teuthology.orchestra.run.smithi047.stderr:24+1 records in 2020-11-20T12:04:20.960 INFO:teuthology.orchestra.run.smithi047.stderr:24+1 records out 2020-11-20T12:04:20.960 INFO:teuthology.orchestra.run.smithi047.stderr:12667 bytes (13 kB, 12 KiB) copied, 0.000242981 s, 52.1 MB/s 2020-11-20T12:04:20.961 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-11-20T12:04:20.962 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-20T12:04:20.962 INFO:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/daemon-helper 2020-11-20T12:04:20.972 INFO:teuthology.orchestra.run.smithi189.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-11-20T12:04:20.981 INFO:teuthology.orchestra.run.smithi189.stderr:5+1 records in 2020-11-20T12:04:20.982 INFO:teuthology.orchestra.run.smithi189.stderr:5+1 records out 2020-11-20T12:04:20.982 INFO:teuthology.orchestra.run.smithi189.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 9.481e-05 s, 32.3 MB/s 2020-11-20T12:04:20.983 INFO:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-11-20T12:04:21.038 INFO:teuthology.orchestra.run.smithi047:> set -ex 2020-11-20T12:04:21.038 INFO:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/daemon-helper 2020-11-20T12:04:21.043 INFO:teuthology.orchestra.run.smithi047.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-11-20T12:04:21.052 INFO:teuthology.orchestra.run.smithi047.stderr:5+1 records in 2020-11-20T12:04:21.053 INFO:teuthology.orchestra.run.smithi047.stderr:5+1 records out 2020-11-20T12:04:21.053 INFO:teuthology.orchestra.run.smithi047.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 9.3167e-05 s, 32.8 MB/s 2020-11-20T12:04:21.054 INFO:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-11-20T12:04:21.109 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-11-20T12:04:21.110 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-20T12:04:21.111 INFO:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/adjust-ulimits 2020-11-20T12:04:21.116 INFO:teuthology.orchestra.run.smithi189.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-11-20T12:04:21.126 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2020-11-20T12:04:21.126 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2020-11-20T12:04:21.126 INFO:teuthology.orchestra.run.smithi189.stderr:299 bytes copied, 6.7484e-05 s, 4.4 MB/s 2020-11-20T12:04:21.127 INFO:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-11-20T12:04:21.182 INFO:teuthology.orchestra.run.smithi047:> set -ex 2020-11-20T12:04:21.182 INFO:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/adjust-ulimits 2020-11-20T12:04:21.187 INFO:teuthology.orchestra.run.smithi047.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-11-20T12:04:21.196 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records in 2020-11-20T12:04:21.197 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records out 2020-11-20T12:04:21.197 INFO:teuthology.orchestra.run.smithi047.stderr:299 bytes copied, 6.7253e-05 s, 4.4 MB/s 2020-11-20T12:04:21.198 INFO:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-11-20T12:04:21.254 INFO:teuthology.run_tasks:Running task ceph... 2020-11-20T12:04:21.311 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2020-11-20T12:04:21.312 INFO:teuthology.orchestra.run.smithi047:> sudo chmod 777 /var/log/ceph 2020-11-20T12:04:21.314 INFO:teuthology.orchestra.run.smithi189:> sudo chmod 777 /var/log/ceph 2020-11-20T12:04:21.329 INFO:tasks.ceph:Disabling ceph logrotate... 2020-11-20T12:04:21.330 INFO:teuthology.orchestra.run.smithi047:> sudo rm -f -- /etc/logrotate.d/ceph 2020-11-20T12:04:21.369 INFO:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/logrotate.d/ceph 2020-11-20T12:04:21.386 INFO:tasks.ceph:Creating extra log directories... 2020-11-20T12:04:21.386 INFO:teuthology.orchestra.run.smithi047:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-11-20T12:04:21.425 INFO:teuthology.orchestra.run.smithi189:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-11-20T12:04:21.504 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2020-11-20T12:04:21.504 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2020-11-20T12:04:21.505 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2020-11-20T12:04:21.505 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-20T12:04:21.505 INFO:teuthology.orchestra.run.smithi189:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2020-11-20T12:04:21.510 INFO:teuthology.orchestra.run.smithi189.stderr:+ dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2020-11-20T12:04:21.512 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2020-11-20T12:04:21.512 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2020-11-20T12:04:21.512 INFO:teuthology.orchestra.run.smithi189.stderr:408 bytes copied, 6.7247e-05 s, 6.1 MB/s 2020-11-20T12:04:21.513 INFO:teuthology.orchestra.run.smithi189:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2020-11-20T12:04:21.590 INFO:teuthology.orchestra.run.smithi047:> set -ex 2020-11-20T12:04:21.591 INFO:teuthology.orchestra.run.smithi047:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2020-11-20T12:04:21.596 INFO:teuthology.orchestra.run.smithi047.stderr:+ dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2020-11-20T12:04:21.597 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records in 2020-11-20T12:04:21.597 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records out 2020-11-20T12:04:21.598 INFO:teuthology.orchestra.run.smithi047.stderr:408 bytes copied, 7.175e-05 s, 5.7 MB/s 2020-11-20T12:04:21.598 INFO:teuthology.orchestra.run.smithi047:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2020-11-20T12:04:21.681 INFO:tasks.ceph:Creating ceph cluster ceph... 2020-11-20T12:04:21.682 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'filestore flush min': 0, 'osd max markdown count': 1000, 'osd objectstore': 'filestore', 'osd op complaint time': 180, 'osd sloppy crc': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-11-20T12:04:21.682 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi/5640307', 'branch': 'wip-pdonnell-testing-20201120.053228', 'description': 'fs/thrash/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/filestore-xfs overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/ffsb}}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '5640307', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi', 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '18.04', 'overrides': {'admin_socket': {'branch': 'wip-pdonnell-testing-20201120.053228'}, 'ceph': {'cephfs': {'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'filestore flush min': 0, 'osd max markdown count': 1000, 'osd objectstore': 'filestore', 'osd op complaint time': 180, 'osd sloppy crc': True}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'filestore': True, 'fs': 'xfs'}, 'install': {'ceph': {'sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb'}}, 'workunit': {'branch': 'wip-pdonnell-testing-20201120.053228', 'sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 95, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'wip-pdonnell-testing-20201120.053228', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201120.053228/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb'}}, {'ceph': None}, {'ceph-fuse': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['suites/ffsb.sh']}}}], 'teuthology_branch': 'master', 'timestamp': '2020-11-20_08:39:42', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.21182', 'targets': {'smithi189.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7fiCLM94ZgBmaFMAXibiC0SEL9yXQUymMFlmebFjXuaZfc+uNILlmQpqiaXYFvmv894Yoz9Br8CoB65eJH3F8CCkocG/yO9D0oz/yJysH6rXiWhs7wNxKjzM7a6Vy+io00a5WDEP5+hpAci2C5d6K3TGgn3ldmi+u7FOkBfZATNjoHVIBXbixp9TaPEUCbAmbnm7JNdsISCvhy7aKPY56qeUklZ1CLAzmU3ZOY0HVPosKQduqA5g2z32K5ltuJNtUZWbjod64YjNG1rTKI+wSOR1Fk936drVyMPzN65wxfSTtg+6Rc+dbRX/CjnWsvOAncF751H7mivQ6WfasgAPz', 'smithi047.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFX3BDFEwaObOOpCQyvJwob5VQzCMqTMEbhuS6IulSBMirctkmFxZC7hPAuD0bK7UueGYa4ZrAcOGRq1Ua8S+lQQG8co0lx8/LsrTZmOsnqsSkMn+cG337gLxDLFGzUPAfs6l7ebIdq3ls/l56RVUwEDTkxeDjC6VXLvqoqWypI51W/SUqcXr+cdc/p+o76WLdTa0s6VtxOjUDZYwCI1IIwFd7c2BJbooBfBZd5pEfWiAwHz75DORHbyP02zSagtElOa+oetEXCbzek1BobGpRJG4MDYyhLVYx77xhJgq8+Ox7s4pdMbGAKIiccLRxucgYJHeEaCbqVZWxQVV+xEIp'}} 2020-11-20T12:04:21.683 INFO:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2020-11-20T12:04:21.721 INFO:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2020-11-20T12:04:21.729 INFO:teuthology.orchestra.run.smithi047:> sudo install -d -m0777 -- /var/run/ceph 2020-11-20T12:04:21.769 INFO:teuthology.orchestra.run.smithi189:> sudo install -d -m0777 -- /var/run/ceph 2020-11-20T12:04:21.787 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-20T12:04:21.787 INFO:teuthology.orchestra.run.smithi189:> dd if=/scratch_devs of=/dev/stdout 2020-11-20T12:04:21.833 INFO:teuthology.orchestra.run.smithi189.stderr:+ dd if=/scratch_devs of=/dev/stdout 2020-11-20T12:04:21.834 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/vg_nvme/lv_1 2020-11-20T12:04:21.834 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/vg_nvme/lv_2 2020-11-20T12:04:21.834 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/vg_nvme/lv_3 2020-11-20T12:04:21.834 INFO:teuthology.orchestra.run.smithi189.stdout:/dev/vg_nvme/lv_4 2020-11-20T12:04:21.835 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2020-11-20T12:04:21.835 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2020-11-20T12:04:21.835 INFO:teuthology.orchestra.run.smithi189.stderr:72 bytes copied, 0.000187306 s, 384 kB/s 2020-11-20T12:04:21.835 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-11-20T12:04:21.836 INFO:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-11-20T12:04:21.882 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-11-20T12:04:21.883 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-20T12:04:21.883 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 6h/6d Inode: 342 Links: 1 2020-11-20T12:04:21.883 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-20T12:04:21.883 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2020-11-20 12:01:22.403157179 +0000 2020-11-20T12:04:21.884 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2020-11-20 12:01:22.287157175 +0000 2020-11-20T12:04:21.884 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2020-11-20 12:01:22.287157175 +0000 2020-11-20T12:04:21.884 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: - 2020-11-20T12:04:21.892 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2020-11-20T12:04:21.892 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2020-11-20T12:04:21.893 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000388111 s, 1.3 MB/s 2020-11-20T12:04:21.897 INFO:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-11-20T12:04:21.947 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-11-20T12:04:21.947 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-20T12:04:21.947 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 6h/6d Inode: 349 Links: 1 2020-11-20T12:04:21.947 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-20T12:04:21.948 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2020-11-20 12:01:22.647157186 +0000 2020-11-20T12:04:21.948 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2020-11-20 12:01:22.543157183 +0000 2020-11-20T12:04:21.948 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2020-11-20 12:01:22.543157183 +0000 2020-11-20T12:04:21.948 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: - 2020-11-20T12:04:21.955 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2020-11-20T12:04:21.955 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2020-11-20T12:04:21.956 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000375182 s, 1.4 MB/s 2020-11-20T12:04:21.960 INFO:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-11-20T12:04:22.011 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-11-20T12:04:22.011 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-20T12:04:22.011 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 6h/6d Inode: 355 Links: 1 2020-11-20T12:04:22.012 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-20T12:04:22.012 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2020-11-20 12:01:22.879157194 +0000 2020-11-20T12:04:22.012 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2020-11-20 12:01:22.771157190 +0000 2020-11-20T12:04:22.012 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2020-11-20 12:01:22.771157190 +0000 2020-11-20T12:04:22.012 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: - 2020-11-20T12:04:22.020 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2020-11-20T12:04:22.020 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2020-11-20T12:04:22.021 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000392844 s, 1.3 MB/s 2020-11-20T12:04:22.025 INFO:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-11-20T12:04:22.079 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-11-20T12:04:22.079 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-20T12:04:22.079 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 6h/6d Inode: 361 Links: 1 2020-11-20T12:04:22.079 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-20T12:04:22.080 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2020-11-20 12:01:23.143157202 +0000 2020-11-20T12:04:22.080 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2020-11-20 12:01:23.015157198 +0000 2020-11-20T12:04:22.080 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2020-11-20 12:01:23.015157198 +0000 2020-11-20T12:04:22.080 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: - 2020-11-20T12:04:22.088 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2020-11-20T12:04:22.088 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2020-11-20T12:04:22.089 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000399988 s, 1.3 MB/s 2020-11-20T12:04:22.093 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2020-11-20T12:04:22.093 INFO:teuthology.orchestra.run.smithi047:> set -ex 2020-11-20T12:04:22.094 INFO:teuthology.orchestra.run.smithi047:> dd if=/scratch_devs of=/dev/stdout 2020-11-20T12:04:22.099 INFO:teuthology.orchestra.run.smithi047.stderr:+ dd if=/scratch_devs of=/dev/stdout 2020-11-20T12:04:22.100 INFO:teuthology.orchestra.run.smithi047.stdout:/dev/vg_nvme/lv_1 2020-11-20T12:04:22.100 INFO:teuthology.orchestra.run.smithi047.stdout:/dev/vg_nvme/lv_2 2020-11-20T12:04:22.101 INFO:teuthology.orchestra.run.smithi047.stdout:/dev/vg_nvme/lv_3 2020-11-20T12:04:22.101 INFO:teuthology.orchestra.run.smithi047.stdout:/dev/vg_nvme/lv_4 2020-11-20T12:04:22.101 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records in 2020-11-20T12:04:22.101 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records out 2020-11-20T12:04:22.102 INFO:teuthology.orchestra.run.smithi047.stderr:72 bytes copied, 0.000175684 s, 410 kB/s 2020-11-20T12:04:22.102 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-11-20T12:04:22.102 INFO:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-11-20T12:04:22.150 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-11-20T12:04:22.151 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-20T12:04:22.151 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 6h/6d Inode: 345 Links: 1 2020-11-20T12:04:22.151 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-20T12:04:22.151 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2020-11-20 12:01:22.244815794 +0000 2020-11-20T12:04:22.151 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2020-11-20 12:01:22.160773797 +0000 2020-11-20T12:04:22.152 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2020-11-20 12:01:22.160773797 +0000 2020-11-20T12:04:22.152 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: - 2020-11-20T12:04:22.161 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2020-11-20T12:04:22.161 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2020-11-20T12:04:22.161 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000285174 s, 1.8 MB/s 2020-11-20T12:04:22.165 INFO:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-11-20T12:04:22.214 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-11-20T12:04:22.215 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-20T12:04:22.216 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 6h/6d Inode: 351 Links: 1 2020-11-20T12:04:22.216 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-20T12:04:22.216 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2020-11-20 12:01:22.484935789 +0000 2020-11-20T12:04:22.216 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2020-11-20 12:01:22.372879791 +0000 2020-11-20T12:04:22.217 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2020-11-20 12:01:22.372879791 +0000 2020-11-20T12:04:22.217 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: - 2020-11-20T12:04:22.224 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2020-11-20T12:04:22.224 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2020-11-20T12:04:22.225 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000308923 s, 1.7 MB/s 2020-11-20T12:04:22.229 INFO:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-11-20T12:04:22.278 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-11-20T12:04:22.279 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-20T12:04:22.279 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 6h/6d Inode: 358 Links: 1 2020-11-20T12:04:22.279 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-20T12:04:22.279 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2020-11-20 12:01:22.705045784 +0000 2020-11-20T12:04:22.280 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2020-11-20 12:01:22.596991786 +0000 2020-11-20T12:04:22.280 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2020-11-20 12:01:22.596991786 +0000 2020-11-20T12:04:22.280 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: - 2020-11-20T12:04:22.288 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2020-11-20T12:04:22.289 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2020-11-20T12:04:22.289 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000316962 s, 1.6 MB/s 2020-11-20T12:04:22.293 INFO:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-11-20T12:04:22.359 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-11-20T12:04:22.359 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-20T12:04:22.359 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 6h/6d Inode: 364 Links: 1 2020-11-20T12:04:22.360 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-20T12:04:22.360 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2020-11-20 12:01:22.981183779 +0000 2020-11-20T12:04:22.360 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2020-11-20 12:01:22.837111781 +0000 2020-11-20T12:04:22.360 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2020-11-20 12:01:22.837111781 +0000 2020-11-20T12:04:22.361 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: - 2020-11-20T12:04:22.361 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2020-11-20T12:04:22.361 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2020-11-20T12:04:22.362 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000217546 s, 2.4 MB/s 2020-11-20T12:04:22.365 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2020-11-20T12:04:22.365 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi189.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi047.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2020-11-20T12:04:22.366 INFO:tasks.ceph:Generating config... 2020-11-20T12:04:22.367 INFO:tasks.ceph:[client] client mount timeout = 600 2020-11-20T12:04:22.368 INFO:tasks.ceph:[client] debug client = 20 2020-11-20T12:04:22.368 INFO:tasks.ceph:[client] debug ms = 1 2020-11-20T12:04:22.369 INFO:tasks.ceph:[client] rados mon op timeout = 15m 2020-11-20T12:04:22.369 INFO:tasks.ceph:[client] rados osd op timeout = 15m 2020-11-20T12:04:22.370 INFO:tasks.ceph:[mds] debug mds = 20 2020-11-20T12:04:22.370 INFO:tasks.ceph:[mds] debug ms = 1 2020-11-20T12:04:22.371 INFO:tasks.ceph:[mds] mds bal frag = True 2020-11-20T12:04:22.371 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2020-11-20T12:04:22.372 INFO:tasks.ceph:[mds] mds bal merge size = 5 2020-11-20T12:04:22.372 INFO:tasks.ceph:[mds] mds bal split bits = 3 2020-11-20T12:04:22.372 INFO:tasks.ceph:[mds] mds bal split size = 100 2020-11-20T12:04:22.373 INFO:tasks.ceph:[mds] mds debug frag = True 2020-11-20T12:04:22.373 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2020-11-20T12:04:22.374 INFO:tasks.ceph:[mds] mds op complaint time = 180 2020-11-20T12:04:22.374 INFO:tasks.ceph:[mds] mds verify scatter = True 2020-11-20T12:04:22.374 INFO:tasks.ceph:[mds] osd op complaint time = 180 2020-11-20T12:04:22.375 INFO:tasks.ceph:[mds] rados mon op timeout = 15m 2020-11-20T12:04:22.375 INFO:tasks.ceph:[mds] rados osd op timeout = 15m 2020-11-20T12:04:22.376 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-11-20T12:04:22.376 INFO:tasks.ceph:[mgr] debug ms = 1 2020-11-20T12:04:22.376 INFO:tasks.ceph:[mon] debug mon = 20 2020-11-20T12:04:22.377 INFO:tasks.ceph:[mon] debug ms = 1 2020-11-20T12:04:22.377 INFO:tasks.ceph:[mon] debug paxos = 20 2020-11-20T12:04:22.378 INFO:tasks.ceph:[mon] mon op complaint time = 120 2020-11-20T12:04:22.378 INFO:tasks.ceph:[osd] debug filestore = 20 2020-11-20T12:04:22.378 INFO:tasks.ceph:[osd] debug journal = 20 2020-11-20T12:04:22.379 INFO:tasks.ceph:[osd] debug ms = 20 2020-11-20T12:04:22.379 INFO:tasks.ceph:[osd] debug osd = 25 2020-11-20T12:04:22.380 INFO:tasks.ceph:[osd] filestore flush min = 0 2020-11-20T12:04:22.380 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2020-11-20T12:04:22.381 INFO:tasks.ceph:[osd] osd objectstore = filestore 2020-11-20T12:04:22.381 INFO:tasks.ceph:[osd] osd op complaint time = 180 2020-11-20T12:04:22.382 INFO:tasks.ceph:[osd] osd sloppy crc = True 2020-11-20T12:04:22.382 INFO:tasks.ceph:Setting up mon.a... 2020-11-20T12:04:22.383 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2020-11-20T12:04:22.423 INFO:teuthology.orchestra.run.smithi189.stdout:creating /etc/ceph/ceph.keyring 2020-11-20T12:04:22.426 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2020-11-20T12:04:22.525 INFO:teuthology.orchestra.run.smithi189:> sudo chmod 0644 /etc/ceph/ceph.keyring 2020-11-20T12:04:22.597 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.189'), ('mon.b', '172.21.15.47'), ('mon.c', '[v2:172.21.15.47:3301,v1:172.21.15.47:6790]')] 2020-11-20T12:04:22.598 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.189,172.21.15.47,[v2:172.21.15.47:3301,v1:172.21.15.47:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'filestore flush min': 0, 'osd max markdown count': 1000, 'osd objectstore': 'filestore', 'osd op complaint time': 180, 'osd sloppy crc': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.b': {}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}} 2020-11-20T12:04:22.601 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-20T12:04:22.601 INFO:teuthology.orchestra.run.smithi189:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-20T12:04:22.660 INFO:teuthology.orchestra.run.smithi189.stderr:+ dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-20T12:04:22.670 INFO:teuthology.orchestra.run.smithi189.stderr:6+1 records in 2020-11-20T12:04:22.670 INFO:teuthology.orchestra.run.smithi189.stderr:6+1 records out 2020-11-20T12:04:22.670 INFO:teuthology.orchestra.run.smithi189.stderr:3242 bytes (3.2 kB, 3.2 KiB) copied, 0.00820022 s, 395 kB/s 2020-11-20T12:04:22.671 INFO:teuthology.orchestra.run.smithi189:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.189 --add b 172.21.15.47 --addv c '[v2:172.21.15.47:3301,v1:172.21.15.47:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:04:22.763 INFO:teuthology.orchestra.run.smithi189.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:04:22.764 INFO:teuthology.orchestra.run.smithi189.stdout:monmaptool: generated fsid cda18d0e-64b3-41bb-964a-084069c3e648 2020-11-20T12:04:22.764 INFO:teuthology.orchestra.run.smithi189.stdout:epoch 0 2020-11-20T12:04:22.764 INFO:teuthology.orchestra.run.smithi189.stdout:fsid cda18d0e-64b3-41bb-964a-084069c3e648 2020-11-20T12:04:22.764 INFO:teuthology.orchestra.run.smithi189.stdout:last_changed 2020-11-20T12:04:22.764778+0000 2020-11-20T12:04:22.765 INFO:teuthology.orchestra.run.smithi189.stdout:created 2020-11-20T12:04:22.764778+0000 2020-11-20T12:04:22.765 INFO:teuthology.orchestra.run.smithi189.stdout:min_mon_release 0 (unknown) 2020-11-20T12:04:22.765 INFO:teuthology.orchestra.run.smithi189.stdout:election_strategy: 1 2020-11-20T12:04:22.765 INFO:teuthology.orchestra.run.smithi189.stdout:0: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.a 2020-11-20T12:04:22.766 INFO:teuthology.orchestra.run.smithi189.stdout:1: [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] mon.b 2020-11-20T12:04:22.766 INFO:teuthology.orchestra.run.smithi189.stdout:2: [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] mon.c 2020-11-20T12:04:22.766 INFO:teuthology.orchestra.run.smithi189.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2020-11-20T12:04:22.767 INFO:teuthology.orchestra.run.smithi189:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2020-11-20T12:04:22.816 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID cda18d0e-64b3-41bb-964a-084069c3e648... 2020-11-20T12:04:22.817 INFO:teuthology.orchestra.run.smithi047:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:04:22.825 INFO:teuthology.orchestra.run.smithi189:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T12:04:22.868 INFO:teuthology.orchestra.run.smithi047.stdout:[global] 2020-11-20T12:04:22.868 INFO:teuthology.orchestra.run.smithi047.stdout: chdir = "" 2020-11-20T12:04:22.869 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:04:22.869 INFO:teuthology.orchestra.run.smithi047.stdout: auth supported = cephx 2020-11-20T12:04:22.869 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:22.869 INFO:teuthology.orchestra.run.smithi047.stdout: filestore xattr use omap = true 2020-11-20T12:04:22.869 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:22.870 INFO:teuthology.orchestra.run.smithi047.stdout: mon clock drift allowed = 1.000 2020-11-20T12:04:22.870 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:22.870 INFO:teuthology.orchestra.run.smithi047.stdout: osd crush chooseleaf type = 0 2020-11-20T12:04:22.870 INFO:teuthology.orchestra.run.smithi047.stdout: auth debug = true 2020-11-20T12:04:22.870 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:22.871 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on old message = true 2020-11-20T12:04:22.871 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on bug = true 2020-11-20T12:04:22.871 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:22.871 INFO:teuthology.orchestra.run.smithi047.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:04:22.871 INFO:teuthology.orchestra.run.smithi047.stdout: mon pg warn max object skew = 0 2020-11-20T12:04:22.872 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:22.872 INFO:teuthology.orchestra.run.smithi047.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:04:22.872 INFO:teuthology.orchestra.run.smithi047.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:04:22.872 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:22.872 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default size = 2 2020-11-20T12:04:22.873 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:22.873 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow primary affinity = true 2020-11-20T12:04:22.873 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow pg remap = true 2020-11-20T12:04:22.873 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:04:22.873 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:04:22.874 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on no sortbitwise = false 2020-11-20T12:04:22.874 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:04:22.874 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on too few osds = false 2020-11-20T12:04:22.874 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:04:22.874 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:04:22.875 INFO:teuthology.orchestra.run.smithi047.stdout: mon_allow_pool_size_one = true 2020-11-20T12:04:22.875 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:22.875 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:04:22.875 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:22.875 INFO:teuthology.orchestra.run.smithi047.stdout: osd default data pool replay window = 5 2020-11-20T12:04:22.876 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:22.876 INFO:teuthology.orchestra.run.smithi047.stdout: mon allow pool delete = true 2020-11-20T12:04:22.876 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:22.876 INFO:teuthology.orchestra.run.smithi047.stdout: mon cluster log file level = debug 2020-11-20T12:04:22.876 INFO:teuthology.orchestra.run.smithi047.stdout: debug asserts on shutdown = true 2020-11-20T12:04:22.877 INFO:teuthology.orchestra.run.smithi047.stdout: mon health detail to clog = false 2020-11-20T12:04:22.877 INFO:teuthology.orchestra.run.smithi047.stdout: mon host = "172.21.15.189,172.21.15.47,[v2:172.21.15.47:3301,v1:172.21.15.47:6790]" 2020-11-20T12:04:22.877 INFO:teuthology.orchestra.run.smithi047.stdout: fsid = cda18d0e-64b3-41bb-964a-084069c3e648 2020-11-20T12:04:22.877 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:22.877 INFO:teuthology.orchestra.run.smithi047.stdout:[osd] 2020-11-20T12:04:22.878 INFO:teuthology.orchestra.run.smithi047.stdout: osd journal size = 100 2020-11-20T12:04:22.878 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:22.878 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub load threshold = 5.0 2020-11-20T12:04:22.878 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub max interval = 600 2020-11-20T12:04:22.878 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:22.879 INFO:teuthology.orchestra.run.smithi047.stdout: osd recover clone overlap = true 2020-11-20T12:04:22.879 INFO:teuthology.orchestra.run.smithi047.stdout: osd recovery max chunk = 1048576 2020-11-20T12:04:22.879 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:22.879 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug shutdown = true 2020-11-20T12:04:22.879 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug op order = true 2020-11-20T12:04:22.880 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug verify stray on activate = true 2020-11-20T12:04:22.880 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:22.880 INFO:teuthology.orchestra.run.smithi047.stdout: osd open classes on start = true 2020-11-20T12:04:22.880 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug pg log writeout = true 2020-11-20T12:04:22.880 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:22.881 INFO:teuthology.orchestra.run.smithi047.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:04:22.881 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:22.881 INFO:teuthology.orchestra.run.smithi047.stdout: osd map max advance = 10 2020-11-20T12:04:22.881 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:22.882 INFO:teuthology.orchestra.run.smithi047.stdout: journal zero on create = true 2020-11-20T12:04:22.882 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:22.882 INFO:teuthology.orchestra.run.smithi047.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:04:22.882 INFO:teuthology.orchestra.run.smithi047.stdout: filestore apply finisher threads = 3 2020-11-20T12:04:22.882 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:22.882 INFO:teuthology.orchestra.run.smithi047.stdout: bdev debug aio = true 2020-11-20T12:04:22.883 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug misdirected ops = true 2020-11-20T12:04:22.883 INFO:teuthology.orchestra.run.smithi047.stdout: debug filestore = 20 2020-11-20T12:04:22.883 INFO:teuthology.orchestra.run.smithi047.stdout: debug journal = 20 2020-11-20T12:04:22.883 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 20 2020-11-20T12:04:22.883 INFO:teuthology.orchestra.run.smithi047.stdout: debug osd = 25 2020-11-20T12:04:22.884 INFO:teuthology.orchestra.run.smithi047.stdout: filestore flush min = 0 2020-11-20T12:04:22.884 INFO:teuthology.orchestra.run.smithi047.stdout: osd max markdown count = 1000 2020-11-20T12:04:22.884 INFO:teuthology.orchestra.run.smithi047.stdout: osd objectstore = filestore 2020-11-20T12:04:22.884 INFO:teuthology.orchestra.run.smithi047.stdout: osd op complaint time = 180 2020-11-20T12:04:22.884 INFO:teuthology.orchestra.run.smithi047.stdout: osd sloppy crc = True 2020-11-20T12:04:22.885 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:22.885 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr] 2020-11-20T12:04:22.885 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2020-11-20T12:04:22.885 INFO:teuthology.orchestra.run.smithi047.stdout: debug mgr = 20 2020-11-20T12:04:22.885 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2020-11-20T12:04:22.886 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2020-11-20T12:04:22.886 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:04:22.886 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:04:22.886 INFO:teuthology.orchestra.run.smithi047.stdout: mgr/telemetry/nag = false 2020-11-20T12:04:22.886 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:22.887 INFO:teuthology.orchestra.run.smithi047.stdout:[mon] 2020-11-20T12:04:22.887 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2020-11-20T12:04:22.887 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2020-11-20T12:04:22.887 INFO:teuthology.orchestra.run.smithi047.stdout: debug paxos = 20 2020-11-20T12:04:22.887 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2020-11-20T12:04:22.888 INFO:teuthology.orchestra.run.smithi047.stdout: mon data avail warn = 5 2020-11-20T12:04:22.888 INFO:teuthology.orchestra.run.smithi047.stdout: mon mgr mkfs grace = 240 2020-11-20T12:04:22.888 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:04:22.888 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd reporter subtree level = osd 2020-11-20T12:04:22.888 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd prime pg temp = true 2020-11-20T12:04:22.889 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:04:22.889 INFO:teuthology.orchestra.run.smithi047.stdout: mon op complaint time = 120 2020-11-20T12:04:22.889 INFO:teuthology.orchestra.run.smithi047.stdout: 2020-11-20T12:04:22.889 INFO:teuthology.orchestra.run.smithi047.stdout:[client] 2020-11-20T12:04:22.889 INFO:teuthology.orchestra.run.smithi047.stdout: rgw cache enabled = true 2020-11-20T12:04:22.890 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable ops log = true 2020-11-20T12:04:22.890 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable usage log = true 2020-11-20T12:04:22.890 INFO:teuthology.orchestra.run.smithi047.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:04:22.890 INFO:teuthology.orchestra.run.smithi047.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:04:22.890 INFO:teuthology.orchestra.run.smithi047.stdout: client mount timeout = 600 2020-11-20T12:04:22.891 INFO:teuthology.orchestra.run.smithi047.stdout: debug client = 20 2020-11-20T12:04:22.891 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2020-11-20T12:04:22.891 INFO:teuthology.orchestra.run.smithi047.stdout: rados mon op timeout = 15m 2020-11-20T12:04:22.891 INFO:teuthology.orchestra.run.smithi047.stdout: rados osd op timeout = 15m 2020-11-20T12:04:22.891 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.a] 2020-11-20T12:04:22.892 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.b] 2020-11-20T12:04:22.892 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.c] 2020-11-20T12:04:22.892 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.a] 2020-11-20T12:04:22.892 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.b] 2020-11-20T12:04:22.892 INFO:teuthology.orchestra.run.smithi047.stdout:[mds] 2020-11-20T12:04:22.893 INFO:teuthology.orchestra.run.smithi047.stdout: debug mds = 20 2020-11-20T12:04:22.893 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2020-11-20T12:04:22.893 INFO:teuthology.orchestra.run.smithi047.stdout: mds bal frag = True 2020-11-20T12:04:22.893 INFO:teuthology.orchestra.run.smithi047.stdout: mds bal fragment size max = 10000 2020-11-20T12:04:22.893 INFO:teuthology.orchestra.run.smithi047.stdout: mds bal merge size = 5 2020-11-20T12:04:22.894 INFO:teuthology.orchestra.run.smithi047.stdout: mds bal split bits = 3 2020-11-20T12:04:22.894 INFO:teuthology.orchestra.run.smithi047.stdout: mds bal split size = 100 2020-11-20T12:04:22.894 INFO:teuthology.orchestra.run.smithi047.stdout: mds debug frag = True 2020-11-20T12:04:22.894 INFO:teuthology.orchestra.run.smithi047.stdout: mds debug scatterstat = True 2020-11-20T12:04:22.894 INFO:teuthology.orchestra.run.smithi047.stdout: mds op complaint time = 180 2020-11-20T12:04:22.895 INFO:teuthology.orchestra.run.smithi047.stdout: mds verify scatter = True 2020-11-20T12:04:22.895 INFO:teuthology.orchestra.run.smithi047.stdout: osd op complaint time = 180 2020-11-20T12:04:22.895 INFO:teuthology.orchestra.run.smithi047.stdout: rados mon op timeout = 15m 2020-11-20T12:04:22.895 INFO:teuthology.orchestra.run.smithi047.stdout: rados osd op timeout = 15m 2020-11-20T12:04:22.896 INFO:teuthology.orchestra.run.smithi189.stdout:[global] 2020-11-20T12:04:22.897 INFO:teuthology.orchestra.run.smithi189.stdout: chdir = "" 2020-11-20T12:04:22.897 INFO:teuthology.orchestra.run.smithi189.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T12:04:22.897 INFO:teuthology.orchestra.run.smithi189.stdout: auth supported = cephx 2020-11-20T12:04:22.897 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:22.897 INFO:teuthology.orchestra.run.smithi189.stdout: filestore xattr use omap = true 2020-11-20T12:04:22.898 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:22.898 INFO:teuthology.orchestra.run.smithi189.stdout: mon clock drift allowed = 1.000 2020-11-20T12:04:22.898 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:22.898 INFO:teuthology.orchestra.run.smithi189.stdout: osd crush chooseleaf type = 0 2020-11-20T12:04:22.898 INFO:teuthology.orchestra.run.smithi189.stdout: auth debug = true 2020-11-20T12:04:22.899 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:22.899 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on old message = true 2020-11-20T12:04:22.899 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on bug = true 2020-11-20T12:04:22.899 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:22.899 INFO:teuthology.orchestra.run.smithi189.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T12:04:22.900 INFO:teuthology.orchestra.run.smithi189.stdout: mon pg warn max object skew = 0 2020-11-20T12:04:22.900 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:22.900 INFO:teuthology.orchestra.run.smithi189.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T12:04:22.900 INFO:teuthology.orchestra.run.smithi189.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T12:04:22.900 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:22.901 INFO:teuthology.orchestra.run.smithi189.stdout: osd pool default size = 2 2020-11-20T12:04:22.901 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:22.901 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd allow primary affinity = true 2020-11-20T12:04:22.901 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd allow pg remap = true 2020-11-20T12:04:22.901 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on legacy crush tunables = false 2020-11-20T12:04:22.902 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on crush straw calc version zero = false 2020-11-20T12:04:22.902 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on no sortbitwise = false 2020-11-20T12:04:22.902 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on osd down out interval zero = false 2020-11-20T12:04:22.902 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on too few osds = false 2020-11-20T12:04:22.903 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T12:04:22.903 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T12:04:22.903 INFO:teuthology.orchestra.run.smithi189.stdout: mon_allow_pool_size_one = true 2020-11-20T12:04:22.903 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:22.903 INFO:teuthology.orchestra.run.smithi189.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T12:04:22.904 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:22.904 INFO:teuthology.orchestra.run.smithi189.stdout: osd default data pool replay window = 5 2020-11-20T12:04:22.904 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:22.904 INFO:teuthology.orchestra.run.smithi189.stdout: mon allow pool delete = true 2020-11-20T12:04:22.904 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:22.905 INFO:teuthology.orchestra.run.smithi189.stdout: mon cluster log file level = debug 2020-11-20T12:04:22.905 INFO:teuthology.orchestra.run.smithi189.stdout: debug asserts on shutdown = true 2020-11-20T12:04:22.905 INFO:teuthology.orchestra.run.smithi189.stdout: mon health detail to clog = false 2020-11-20T12:04:22.905 INFO:teuthology.orchestra.run.smithi189.stdout: mon host = "172.21.15.189,172.21.15.47,[v2:172.21.15.47:3301,v1:172.21.15.47:6790]" 2020-11-20T12:04:22.905 INFO:teuthology.orchestra.run.smithi189.stdout: fsid = cda18d0e-64b3-41bb-964a-084069c3e648 2020-11-20T12:04:22.906 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:22.906 INFO:teuthology.orchestra.run.smithi189.stdout:[osd] 2020-11-20T12:04:22.906 INFO:teuthology.orchestra.run.smithi189.stdout: osd journal size = 100 2020-11-20T12:04:22.906 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:22.906 INFO:teuthology.orchestra.run.smithi189.stdout: osd scrub load threshold = 5.0 2020-11-20T12:04:22.907 INFO:teuthology.orchestra.run.smithi189.stdout: osd scrub max interval = 600 2020-11-20T12:04:22.907 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:22.907 INFO:teuthology.orchestra.run.smithi189.stdout: osd recover clone overlap = true 2020-11-20T12:04:22.907 INFO:teuthology.orchestra.run.smithi189.stdout: osd recovery max chunk = 1048576 2020-11-20T12:04:22.907 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:22.908 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug shutdown = true 2020-11-20T12:04:22.908 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug op order = true 2020-11-20T12:04:22.908 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug verify stray on activate = true 2020-11-20T12:04:22.908 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:22.908 INFO:teuthology.orchestra.run.smithi189.stdout: osd open classes on start = true 2020-11-20T12:04:22.909 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug pg log writeout = true 2020-11-20T12:04:22.909 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:22.909 INFO:teuthology.orchestra.run.smithi189.stdout: osd deep scrub update digest min age = 30 2020-11-20T12:04:22.909 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:22.909 INFO:teuthology.orchestra.run.smithi189.stdout: osd map max advance = 10 2020-11-20T12:04:22.910 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:22.910 INFO:teuthology.orchestra.run.smithi189.stdout: journal zero on create = true 2020-11-20T12:04:22.910 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:22.910 INFO:teuthology.orchestra.run.smithi189.stdout: filestore ondisk finisher threads = 3 2020-11-20T12:04:22.910 INFO:teuthology.orchestra.run.smithi189.stdout: filestore apply finisher threads = 3 2020-11-20T12:04:22.911 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:22.911 INFO:teuthology.orchestra.run.smithi189.stdout: bdev debug aio = true 2020-11-20T12:04:22.911 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug misdirected ops = true 2020-11-20T12:04:22.911 INFO:teuthology.orchestra.run.smithi189.stdout: debug filestore = 20 2020-11-20T12:04:22.911 INFO:teuthology.orchestra.run.smithi189.stdout: debug journal = 20 2020-11-20T12:04:22.912 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 20 2020-11-20T12:04:22.912 INFO:teuthology.orchestra.run.smithi189.stdout: debug osd = 25 2020-11-20T12:04:22.912 INFO:teuthology.orchestra.run.smithi189.stdout: filestore flush min = 0 2020-11-20T12:04:22.912 INFO:teuthology.orchestra.run.smithi189.stdout: osd max markdown count = 1000 2020-11-20T12:04:22.912 INFO:teuthology.orchestra.run.smithi189.stdout: osd objectstore = filestore 2020-11-20T12:04:22.913 INFO:teuthology.orchestra.run.smithi189.stdout: osd op complaint time = 180 2020-11-20T12:04:22.913 INFO:teuthology.orchestra.run.smithi189.stdout: osd sloppy crc = True 2020-11-20T12:04:22.913 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:22.913 INFO:teuthology.orchestra.run.smithi189.stdout:[mgr] 2020-11-20T12:04:22.913 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2020-11-20T12:04:22.914 INFO:teuthology.orchestra.run.smithi189.stdout: debug mgr = 20 2020-11-20T12:04:22.914 INFO:teuthology.orchestra.run.smithi189.stdout: debug mon = 20 2020-11-20T12:04:22.914 INFO:teuthology.orchestra.run.smithi189.stdout: debug auth = 20 2020-11-20T12:04:22.914 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:04:22.914 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:04:22.915 INFO:teuthology.orchestra.run.smithi189.stdout: mgr/telemetry/nag = false 2020-11-20T12:04:22.915 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:22.915 INFO:teuthology.orchestra.run.smithi189.stdout:[mon] 2020-11-20T12:04:22.915 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2020-11-20T12:04:22.915 INFO:teuthology.orchestra.run.smithi189.stdout: debug mon = 20 2020-11-20T12:04:22.916 INFO:teuthology.orchestra.run.smithi189.stdout: debug paxos = 20 2020-11-20T12:04:22.916 INFO:teuthology.orchestra.run.smithi189.stdout: debug auth = 20 2020-11-20T12:04:22.916 INFO:teuthology.orchestra.run.smithi189.stdout: mon data avail warn = 5 2020-11-20T12:04:22.916 INFO:teuthology.orchestra.run.smithi189.stdout: mon mgr mkfs grace = 240 2020-11-20T12:04:22.916 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min pgs per osd = 4 2020-11-20T12:04:22.917 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd reporter subtree level = osd 2020-11-20T12:04:22.917 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd prime pg temp = true 2020-11-20T12:04:22.917 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min bytes per osd = 10 2020-11-20T12:04:22.917 INFO:teuthology.orchestra.run.smithi189.stdout: mon op complaint time = 120 2020-11-20T12:04:22.917 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:22.918 INFO:teuthology.orchestra.run.smithi189.stdout:[client] 2020-11-20T12:04:22.918 INFO:teuthology.orchestra.run.smithi189.stdout: rgw cache enabled = true 2020-11-20T12:04:22.918 INFO:teuthology.orchestra.run.smithi189.stdout: rgw enable ops log = true 2020-11-20T12:04:22.918 INFO:teuthology.orchestra.run.smithi189.stdout: rgw enable usage log = true 2020-11-20T12:04:22.918 INFO:teuthology.orchestra.run.smithi189.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T12:04:22.919 INFO:teuthology.orchestra.run.smithi189.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T12:04:22.919 INFO:teuthology.orchestra.run.smithi189.stdout: client mount timeout = 600 2020-11-20T12:04:22.919 INFO:teuthology.orchestra.run.smithi189.stdout: debug client = 20 2020-11-20T12:04:22.919 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2020-11-20T12:04:22.919 INFO:teuthology.orchestra.run.smithi189.stdout: rados mon op timeout = 15m 2020-11-20T12:04:22.920 INFO:teuthology.orchestra.run.smithi189.stdout: rados osd op timeout = 15m 2020-11-20T12:04:22.920 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.a] 2020-11-20T12:04:22.920 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.b] 2020-11-20T12:04:22.920 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.c] 2020-11-20T12:04:22.920 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.a] 2020-11-20T12:04:22.921 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.b] 2020-11-20T12:04:22.921 INFO:teuthology.orchestra.run.smithi189.stdout:[mds] 2020-11-20T12:04:22.921 INFO:teuthology.orchestra.run.smithi189.stdout: debug mds = 20 2020-11-20T12:04:22.921 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2020-11-20T12:04:22.922 INFO:teuthology.orchestra.run.smithi189.stdout: mds bal frag = True 2020-11-20T12:04:22.922 INFO:teuthology.orchestra.run.smithi189.stdout: mds bal fragment size max = 10000 2020-11-20T12:04:22.922 INFO:teuthology.orchestra.run.smithi189.stdout: mds bal merge size = 5 2020-11-20T12:04:22.923 INFO:teuthology.orchestra.run.smithi189.stdout: mds bal split bits = 3 2020-11-20T12:04:22.923 INFO:teuthology.orchestra.run.smithi189.stdout: mds bal split size = 100 2020-11-20T12:04:22.923 INFO:teuthology.orchestra.run.smithi189.stdout: mds debug frag = True 2020-11-20T12:04:22.923 INFO:teuthology.orchestra.run.smithi189.stdout: mds debug scatterstat = True 2020-11-20T12:04:22.923 INFO:teuthology.orchestra.run.smithi189.stdout: mds op complaint time = 180 2020-11-20T12:04:22.924 INFO:teuthology.orchestra.run.smithi189.stdout: mds verify scatter = True 2020-11-20T12:04:22.924 INFO:teuthology.orchestra.run.smithi189.stdout: osd op complaint time = 180 2020-11-20T12:04:22.924 INFO:teuthology.orchestra.run.smithi189.stdout: rados mon op timeout = 15m 2020-11-20T12:04:22.924 INFO:teuthology.orchestra.run.smithi189.stdout: rados osd op timeout = 15m 2020-11-20T12:04:22.925 INFO:tasks.ceph:Creating admin key on mon.a... 2020-11-20T12:04:22.926 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2020-11-20T12:04:23.021 INFO:tasks.ceph:Copying monmap to all nodes... 2020-11-20T12:04:23.021 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-20T12:04:23.021 INFO:teuthology.orchestra.run.smithi189:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2020-11-20T12:04:23.076 INFO:teuthology.orchestra.run.smithi189.stderr:+ dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2020-11-20T12:04:23.077 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.] 2020-11-20T12:04:23.078 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDGsLdfHIBIHxAAh8K/jppH9Fbuh9dK7aPqkw== 2020-11-20T12:04:23.078 INFO:teuthology.orchestra.run.smithi189.stdout:[client.admin] 2020-11-20T12:04:23.079 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDHsLdfMrgvARAAH4HzTyxHSx7hWlopxFbCsg== 2020-11-20T12:04:23.079 INFO:teuthology.orchestra.run.smithi189.stdout: caps mds = "allow *" 2020-11-20T12:04:23.079 INFO:teuthology.orchestra.run.smithi189.stdout: caps mgr = "allow *" 2020-11-20T12:04:23.080 INFO:teuthology.orchestra.run.smithi189.stdout: caps mon = "allow *" 2020-11-20T12:04:23.080 INFO:teuthology.orchestra.run.smithi189.stdout: caps osd = "allow *" 2020-11-20T12:04:23.080 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2020-11-20T12:04:23.080 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2020-11-20T12:04:23.081 INFO:teuthology.orchestra.run.smithi189.stderr:206 bytes copied, 0.00017589 s, 1.2 MB/s 2020-11-20T12:04:23.081 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-20T12:04:23.082 INFO:teuthology.orchestra.run.smithi189:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2020-11-20T12:04:23.129 INFO:teuthology.orchestra.run.smithi189.stderr:+ dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2020-11-20T12:04:23.130 INFO:teuthology.orchestra.run.smithi189.stdout: �͡�d�A��J@i��Hư�_˘�-ư�_˘�-�aXa �����bXb �/��/cXc �/��/abc 2020-11-20T12:04:23.131 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2020-11-20T12:04:23.131 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2020-11-20T12:04:23.131 INFO:teuthology.orchestra.run.smithi189.stderr:409 bytes copied, 0.000157347 s, 2.6 MB/s 2020-11-20T12:04:23.131 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi189.front.sepia.ceph.com 2020-11-20T12:04:23.132 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-20T12:04:23.132 INFO:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.keyring 2020-11-20T12:04:23.132 INFO:teuthology.orchestra.run.smithi189:> sudo chmod 0644 /etc/ceph/ceph.keyring 2020-11-20T12:04:23.187 INFO:teuthology.orchestra.run.smithi189.stderr:+ sudo dd of=/etc/ceph/ceph.keyring 2020-11-20T12:04:23.238 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2020-11-20T12:04:23.238 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2020-11-20T12:04:23.239 INFO:teuthology.orchestra.run.smithi189.stderr:206 bytes copied, 0.0416415 s, 4.9 kB/s 2020-11-20T12:04:23.239 INFO:teuthology.orchestra.run.smithi189.stderr:+ sudo chmod 0644 /etc/ceph/ceph.keyring 2020-11-20T12:04:23.249 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-20T12:04:23.250 INFO:teuthology.orchestra.run.smithi189:> dd of=/home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:04:23.310 INFO:teuthology.orchestra.run.smithi189.stderr:+ dd of=/home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:04:23.362 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2020-11-20T12:04:23.363 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2020-11-20T12:04:23.364 INFO:teuthology.orchestra.run.smithi189.stderr:409 bytes copied, 0.0481409 s, 8.5 kB/s 2020-11-20T12:04:23.364 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi047.front.sepia.ceph.com 2020-11-20T12:04:23.366 INFO:teuthology.orchestra.run.smithi047:> set -ex 2020-11-20T12:04:23.367 INFO:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/ceph/ceph.keyring 2020-11-20T12:04:23.368 INFO:teuthology.orchestra.run.smithi047:> sudo chmod 0644 /etc/ceph/ceph.keyring 2020-11-20T12:04:23.380 INFO:teuthology.orchestra.run.smithi047.stderr:+ sudo dd of=/etc/ceph/ceph.keyring 2020-11-20T12:04:23.389 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records in 2020-11-20T12:04:23.389 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records out 2020-11-20T12:04:23.390 INFO:teuthology.orchestra.run.smithi047.stderr:206 bytes copied, 6.8813e-05 s, 3.0 MB/s 2020-11-20T12:04:23.390 INFO:teuthology.orchestra.run.smithi047.stderr:+ sudo chmod 0644 /etc/ceph/ceph.keyring 2020-11-20T12:04:23.400 INFO:teuthology.orchestra.run.smithi047:> set -ex 2020-11-20T12:04:23.400 INFO:teuthology.orchestra.run.smithi047:> dd of=/home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:04:23.454 INFO:teuthology.orchestra.run.smithi047.stderr:+ dd of=/home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:04:23.498 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records in 2020-11-20T12:04:23.498 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records out 2020-11-20T12:04:23.499 INFO:teuthology.orchestra.run.smithi047.stderr:409 bytes copied, 0.0425003 s, 9.6 kB/s 2020-11-20T12:04:23.499 INFO:tasks.ceph:Setting up mon nodes... 2020-11-20T12:04:23.499 INFO:tasks.ceph:Setting up mgr nodes... 2020-11-20T12:04:23.500 INFO:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2020-11-20T12:04:23.544 INFO:teuthology.orchestra.run.smithi189.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2020-11-20T12:04:23.545 INFO:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2020-11-20T12:04:23.593 INFO:teuthology.orchestra.run.smithi047.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2020-11-20T12:04:23.595 INFO:tasks.ceph:Setting up mds nodes... 2020-11-20T12:04:23.596 INFO:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2020-11-20T12:04:23.640 INFO:teuthology.orchestra.run.smithi189.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2020-11-20T12:04:23.642 INFO:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2020-11-20T12:04:23.710 INFO:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2020-11-20T12:04:23.762 INFO:teuthology.orchestra.run.smithi047.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2020-11-20T12:04:23.765 INFO:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2020-11-20T12:04:23.838 INFO:tasks.ceph_client:Setting up client nodes... 2020-11-20T12:04:23.840 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2020-11-20T12:04:23.877 INFO:teuthology.orchestra.run.smithi189.stdout:creating /etc/ceph/ceph.client.0.keyring 2020-11-20T12:04:23.889 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-11-20T12:04:23.890 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi189.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi047.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2020-11-20T12:04:23.891 INFO:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2020-11-20T12:04:23.952 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2020-11-20T12:04:23.954 INFO:tasks.ceph:role: osd.0 2020-11-20T12:04:23.954 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi189.front.sepia.ceph.com 2020-11-20T12:04:23.954 INFO:teuthology.orchestra.run.smithi189:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-11-20T12:04:24.484 INFO:teuthology.orchestra.run.smithi189.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-11-20T12:04:24.485 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-20T12:04:24.485 INFO:teuthology.orchestra.run.smithi189.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-20T12:04:24.485 INFO:teuthology.orchestra.run.smithi189.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-20T12:04:24.485 INFO:teuthology.orchestra.run.smithi189.stdout: = sunit=0 swidth=0 blks 2020-11-20T12:04:24.486 INFO:teuthology.orchestra.run.smithi189.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-20T12:04:24.486 INFO:teuthology.orchestra.run.smithi189.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-20T12:04:24.486 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-20T12:04:24.487 INFO:teuthology.orchestra.run.smithi189.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-20T12:04:24.488 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi189.front.sepia.ceph.com -o noatime 2020-11-20T12:04:24.488 INFO:teuthology.orchestra.run.smithi189:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2020-11-20T12:04:24.563 INFO:teuthology.orchestra.run.smithi189:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2020-11-20T12:04:24.576 INFO:teuthology.orchestra.run.smithi189.stderr:sudo: /sbin/restorecon: command not found 2020-11-20T12:04:24.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-20T12:04:24.578 INFO:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2020-11-20T12:04:24.636 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2020-11-20T12:04:24.636 INFO:tasks.ceph:role: osd.1 2020-11-20T12:04:24.636 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi189.front.sepia.ceph.com 2020-11-20T12:04:24.637 INFO:teuthology.orchestra.run.smithi189:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-11-20T12:04:25.150 INFO:teuthology.orchestra.run.smithi189.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-11-20T12:04:25.150 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-20T12:04:25.151 INFO:teuthology.orchestra.run.smithi189.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-20T12:04:25.151 INFO:teuthology.orchestra.run.smithi189.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-20T12:04:25.152 INFO:teuthology.orchestra.run.smithi189.stdout: = sunit=0 swidth=0 blks 2020-11-20T12:04:25.152 INFO:teuthology.orchestra.run.smithi189.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-20T12:04:25.152 INFO:teuthology.orchestra.run.smithi189.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-20T12:04:25.153 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-20T12:04:25.153 INFO:teuthology.orchestra.run.smithi189.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-20T12:04:25.154 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi189.front.sepia.ceph.com -o noatime 2020-11-20T12:04:25.154 INFO:teuthology.orchestra.run.smithi189:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2020-11-20T12:04:25.223 INFO:teuthology.orchestra.run.smithi189:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2020-11-20T12:04:25.235 INFO:teuthology.orchestra.run.smithi189.stderr:sudo: /sbin/restorecon: command not found 2020-11-20T12:04:25.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-20T12:04:25.237 INFO:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2020-11-20T12:04:25.295 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2020-11-20T12:04:25.295 INFO:tasks.ceph:role: osd.2 2020-11-20T12:04:25.295 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi189.front.sepia.ceph.com 2020-11-20T12:04:25.296 INFO:teuthology.orchestra.run.smithi189:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-11-20T12:04:25.789 INFO:teuthology.orchestra.run.smithi189.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-11-20T12:04:25.790 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-20T12:04:25.790 INFO:teuthology.orchestra.run.smithi189.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-20T12:04:25.791 INFO:teuthology.orchestra.run.smithi189.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-20T12:04:25.791 INFO:teuthology.orchestra.run.smithi189.stdout: = sunit=0 swidth=0 blks 2020-11-20T12:04:25.791 INFO:teuthology.orchestra.run.smithi189.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-20T12:04:25.792 INFO:teuthology.orchestra.run.smithi189.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-20T12:04:25.792 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-20T12:04:25.792 INFO:teuthology.orchestra.run.smithi189.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-20T12:04:25.794 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi189.front.sepia.ceph.com -o noatime 2020-11-20T12:04:25.795 INFO:teuthology.orchestra.run.smithi189:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2020-11-20T12:04:25.834 INFO:teuthology.orchestra.run.smithi189:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2020-11-20T12:04:25.886 INFO:teuthology.orchestra.run.smithi189.stderr:sudo: /sbin/restorecon: command not found 2020-11-20T12:04:25.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-20T12:04:25.887 INFO:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2020-11-20T12:04:25.942 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2020-11-20T12:04:25.943 INFO:tasks.ceph:role: osd.3 2020-11-20T12:04:25.943 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi189.front.sepia.ceph.com 2020-11-20T12:04:25.943 INFO:teuthology.orchestra.run.smithi189:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2020-11-20T12:04:26.500 INFO:teuthology.orchestra.run.smithi189.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2020-11-20T12:04:26.500 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-20T12:04:26.500 INFO:teuthology.orchestra.run.smithi189.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-20T12:04:26.501 INFO:teuthology.orchestra.run.smithi189.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-20T12:04:26.501 INFO:teuthology.orchestra.run.smithi189.stdout: = sunit=0 swidth=0 blks 2020-11-20T12:04:26.501 INFO:teuthology.orchestra.run.smithi189.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-20T12:04:26.501 INFO:teuthology.orchestra.run.smithi189.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-20T12:04:26.502 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-20T12:04:26.502 INFO:teuthology.orchestra.run.smithi189.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-20T12:04:26.503 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi189.front.sepia.ceph.com -o noatime 2020-11-20T12:04:26.503 INFO:teuthology.orchestra.run.smithi189:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2020-11-20T12:04:26.549 INFO:teuthology.orchestra.run.smithi189:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2020-11-20T12:04:26.604 INFO:teuthology.orchestra.run.smithi189.stderr:sudo: /sbin/restorecon: command not found 2020-11-20T12:04:26.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-20T12:04:26.605 INFO:teuthology.orchestra.run.smithi189:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:04:26.708 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:26.705+0000 7f0bcd20ad80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2020-11-20T12:04:26.710 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:26.709+0000 7f0bcd20ad80 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2020-11-20T12:04:26.746 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:26.745+0000 7f0bcd20ad80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-20T12:04:26.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:26.745+0000 7f0bcd20ad80 -1 journal FileJournal::_open_file : zeroing journal 2020-11-20T12:04:26.984 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:26.981+0000 7f0bcd20ad80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-20T12:04:26.988 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:26.985+0000 7f0bcd20ad80 -1 journal do_read_entry(4096): bad header magic 2020-11-20T12:04:26.988 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:26.985+0000 7f0bcd20ad80 -1 journal do_read_entry(4096): bad header magic 2020-11-20T12:04:27.015 INFO:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2020-11-20T12:04:27.030 INFO:teuthology.orchestra.run.smithi189:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:04:27.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:27.109+0000 7fd852b4bd80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2020-11-20T12:04:27.109 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:27.109+0000 7fd852b4bd80 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2020-11-20T12:04:27.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:27.141+0000 7fd852b4bd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-20T12:04:27.144 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:27.141+0000 7fd852b4bd80 -1 journal FileJournal::_open_file : zeroing journal 2020-11-20T12:04:27.387 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:27.385+0000 7fd852b4bd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-20T12:04:27.389 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:27.385+0000 7fd852b4bd80 -1 journal do_read_entry(4096): bad header magic 2020-11-20T12:04:27.389 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:27.385+0000 7fd852b4bd80 -1 journal do_read_entry(4096): bad header magic 2020-11-20T12:04:27.415 INFO:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2020-11-20T12:04:27.472 INFO:teuthology.orchestra.run.smithi189:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:04:27.554 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:27.553+0000 7f41dc3f4d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2020-11-20T12:04:27.555 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:27.553+0000 7f41dc3f4d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2020-11-20T12:04:27.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:27.565+0000 7f41dc3f4d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-20T12:04:27.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:27.565+0000 7f41dc3f4d80 -1 journal FileJournal::_open_file : zeroing journal 2020-11-20T12:04:27.708 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:27.705+0000 7f41dc3f4d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-20T12:04:27.709 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:27.709+0000 7f41dc3f4d80 -1 journal do_read_entry(4096): bad header magic 2020-11-20T12:04:27.709 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:27.709+0000 7f41dc3f4d80 -1 journal do_read_entry(4096): bad header magic 2020-11-20T12:04:27.718 INFO:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2020-11-20T12:04:27.775 INFO:teuthology.orchestra.run.smithi189:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:04:27.853 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:27.853+0000 7fd2a5c91d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2020-11-20T12:04:27.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:27.853+0000 7fd2a5c91d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2020-11-20T12:04:27.865 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:27.865+0000 7fd2a5c91d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-20T12:04:27.866 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:27.865+0000 7fd2a5c91d80 -1 journal FileJournal::_open_file : zeroing journal 2020-11-20T12:04:27.990 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:27.989+0000 7fd2a5c91d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-20T12:04:27.993 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:27.993+0000 7fd2a5c91d80 -1 journal do_read_entry(4096): bad header magic 2020-11-20T12:04:27.994 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:27.993+0000 7fd2a5c91d80 -1 journal do_read_entry(4096): bad header magic 2020-11-20T12:04:28.002 INFO:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2020-11-20T12:04:28.060 INFO:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2020-11-20T12:04:28.076 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2020-11-20T12:04:28.076 INFO:tasks.ceph:role: osd.4 2020-11-20T12:04:28.077 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi047.front.sepia.ceph.com 2020-11-20T12:04:28.077 INFO:teuthology.orchestra.run.smithi047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-11-20T12:04:28.160 INFO:teuthology.orchestra.run.smithi047.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-11-20T12:04:28.160 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-20T12:04:28.161 INFO:teuthology.orchestra.run.smithi047.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-20T12:04:28.161 INFO:teuthology.orchestra.run.smithi047.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-20T12:04:28.161 INFO:teuthology.orchestra.run.smithi047.stdout: = sunit=0 swidth=0 blks 2020-11-20T12:04:28.161 INFO:teuthology.orchestra.run.smithi047.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-20T12:04:28.162 INFO:teuthology.orchestra.run.smithi047.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-20T12:04:28.162 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-20T12:04:28.162 INFO:teuthology.orchestra.run.smithi047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-20T12:04:28.163 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi047.front.sepia.ceph.com -o noatime 2020-11-20T12:04:28.163 INFO:teuthology.orchestra.run.smithi047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2020-11-20T12:04:28.224 INFO:teuthology.orchestra.run.smithi047:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2020-11-20T12:04:28.281 INFO:teuthology.orchestra.run.smithi047.stderr:sudo: /sbin/restorecon: command not found 2020-11-20T12:04:28.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-20T12:04:28.283 INFO:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2020-11-20T12:04:28.339 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2020-11-20T12:04:28.339 INFO:tasks.ceph:role: osd.5 2020-11-20T12:04:28.339 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi047.front.sepia.ceph.com 2020-11-20T12:04:28.340 INFO:teuthology.orchestra.run.smithi047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-11-20T12:04:28.420 INFO:teuthology.orchestra.run.smithi047.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-11-20T12:04:28.421 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-20T12:04:28.421 INFO:teuthology.orchestra.run.smithi047.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-20T12:04:28.421 INFO:teuthology.orchestra.run.smithi047.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-20T12:04:28.421 INFO:teuthology.orchestra.run.smithi047.stdout: = sunit=0 swidth=0 blks 2020-11-20T12:04:28.422 INFO:teuthology.orchestra.run.smithi047.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-20T12:04:28.422 INFO:teuthology.orchestra.run.smithi047.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-20T12:04:28.422 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-20T12:04:28.422 INFO:teuthology.orchestra.run.smithi047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-20T12:04:28.423 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi047.front.sepia.ceph.com -o noatime 2020-11-20T12:04:28.424 INFO:teuthology.orchestra.run.smithi047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2020-11-20T12:04:28.486 INFO:teuthology.orchestra.run.smithi047:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2020-11-20T12:04:28.538 INFO:teuthology.orchestra.run.smithi047.stderr:sudo: /sbin/restorecon: command not found 2020-11-20T12:04:28.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-20T12:04:28.539 INFO:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2020-11-20T12:04:28.594 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2020-11-20T12:04:28.595 INFO:tasks.ceph:role: osd.6 2020-11-20T12:04:28.595 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi047.front.sepia.ceph.com 2020-11-20T12:04:28.595 INFO:teuthology.orchestra.run.smithi047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-11-20T12:04:28.676 INFO:teuthology.orchestra.run.smithi047.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-11-20T12:04:28.676 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-20T12:04:28.677 INFO:teuthology.orchestra.run.smithi047.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-20T12:04:28.677 INFO:teuthology.orchestra.run.smithi047.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-20T12:04:28.677 INFO:teuthology.orchestra.run.smithi047.stdout: = sunit=0 swidth=0 blks 2020-11-20T12:04:28.677 INFO:teuthology.orchestra.run.smithi047.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-20T12:04:28.677 INFO:teuthology.orchestra.run.smithi047.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-20T12:04:28.678 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-20T12:04:28.678 INFO:teuthology.orchestra.run.smithi047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-20T12:04:28.679 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi047.front.sepia.ceph.com -o noatime 2020-11-20T12:04:28.679 INFO:teuthology.orchestra.run.smithi047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2020-11-20T12:04:28.740 INFO:teuthology.orchestra.run.smithi047:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2020-11-20T12:04:28.794 INFO:teuthology.orchestra.run.smithi047.stderr:sudo: /sbin/restorecon: command not found 2020-11-20T12:04:28.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-20T12:04:28.796 INFO:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2020-11-20T12:04:28.855 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2020-11-20T12:04:28.855 INFO:tasks.ceph:role: osd.7 2020-11-20T12:04:28.856 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi047.front.sepia.ceph.com 2020-11-20T12:04:28.857 INFO:teuthology.orchestra.run.smithi047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2020-11-20T12:04:28.936 INFO:teuthology.orchestra.run.smithi047.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2020-11-20T12:04:28.937 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-20T12:04:28.937 INFO:teuthology.orchestra.run.smithi047.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-11-20T12:04:28.937 INFO:teuthology.orchestra.run.smithi047.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-20T12:04:28.937 INFO:teuthology.orchestra.run.smithi047.stdout: = sunit=0 swidth=0 blks 2020-11-20T12:04:28.937 INFO:teuthology.orchestra.run.smithi047.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-11-20T12:04:28.938 INFO:teuthology.orchestra.run.smithi047.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-20T12:04:28.938 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-20T12:04:28.938 INFO:teuthology.orchestra.run.smithi047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-20T12:04:28.939 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi047.front.sepia.ceph.com -o noatime 2020-11-20T12:04:28.940 INFO:teuthology.orchestra.run.smithi047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2020-11-20T12:04:28.998 INFO:teuthology.orchestra.run.smithi047:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2020-11-20T12:04:29.051 INFO:teuthology.orchestra.run.smithi047.stderr:sudo: /sbin/restorecon: command not found 2020-11-20T12:04:29.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-20T12:04:29.052 INFO:teuthology.orchestra.run.smithi047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:04:29.155 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.149+0000 7f7d891e3d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2020-11-20T12:04:29.155 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.149+0000 7f7d891e3d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2020-11-20T12:04:29.166 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.161+0000 7f7d891e3d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-20T12:04:29.166 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.161+0000 7f7d891e3d80 -1 journal FileJournal::_open_file : zeroing journal 2020-11-20T12:04:29.228 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.221+0000 7f7d891e3d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-20T12:04:29.228 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.221+0000 7f7d891e3d80 -1 journal do_read_entry(4096): bad header magic 2020-11-20T12:04:29.228 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.221+0000 7f7d891e3d80 -1 journal do_read_entry(4096): bad header magic 2020-11-20T12:04:29.236 INFO:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2020-11-20T12:04:29.245 INFO:teuthology.orchestra.run.smithi047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:04:29.325 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.321+0000 7fcdf9716d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2020-11-20T12:04:29.326 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.321+0000 7fcdf9716d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2020-11-20T12:04:29.338 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.333+0000 7fcdf9716d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-20T12:04:29.339 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.333+0000 7fcdf9716d80 -1 journal FileJournal::_open_file : zeroing journal 2020-11-20T12:04:29.400 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.393+0000 7fcdf9716d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-20T12:04:29.400 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.393+0000 7fcdf9716d80 -1 journal do_read_entry(4096): bad header magic 2020-11-20T12:04:29.401 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.393+0000 7fcdf9716d80 -1 journal do_read_entry(4096): bad header magic 2020-11-20T12:04:29.408 INFO:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2020-11-20T12:04:29.468 INFO:teuthology.orchestra.run.smithi047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:04:29.550 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.545+0000 7f7100eb8d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2020-11-20T12:04:29.551 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.545+0000 7f7100eb8d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2020-11-20T12:04:29.563 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.557+0000 7f7100eb8d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-20T12:04:29.563 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.557+0000 7f7100eb8d80 -1 journal FileJournal::_open_file : zeroing journal 2020-11-20T12:04:29.625 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.621+0000 7f7100eb8d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-20T12:04:29.625 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.621+0000 7f7100eb8d80 -1 journal do_read_entry(4096): bad header magic 2020-11-20T12:04:29.625 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.621+0000 7f7100eb8d80 -1 journal do_read_entry(4096): bad header magic 2020-11-20T12:04:29.632 INFO:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2020-11-20T12:04:29.692 INFO:teuthology.orchestra.run.smithi047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:04:29.774 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.769+0000 7f4c80007d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2020-11-20T12:04:29.774 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.769+0000 7f4c80007d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2020-11-20T12:04:29.786 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.781+0000 7f4c80007d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-20T12:04:29.786 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.781+0000 7f4c80007d80 -1 journal FileJournal::_open_file : zeroing journal 2020-11-20T12:04:29.846 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.841+0000 7f4c80007d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-20T12:04:29.846 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.841+0000 7f4c80007d80 -1 journal do_read_entry(4096): bad header magic 2020-11-20T12:04:29.847 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:29.841+0000 7f4c80007d80 -1 journal do_read_entry(4096): bad header magic 2020-11-20T12:04:29.854 INFO:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2020-11-20T12:04:29.912 INFO:tasks.ceph:Reading keys from all nodes... 2020-11-20T12:04:29.913 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-20T12:04:29.913 INFO:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2020-11-20T12:04:29.918 INFO:teuthology.orchestra.run.smithi189.stderr:+ sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2020-11-20T12:04:29.928 INFO:teuthology.orchestra.run.smithi189.stdout:[mgr.y] 2020-11-20T12:04:29.928 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDHsLdfPRl5IBAAYYAMBbphBlIjttwWYZ6CHQ== 2020-11-20T12:04:29.928 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2020-11-20T12:04:29.929 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2020-11-20T12:04:29.929 INFO:teuthology.orchestra.run.smithi189.stderr:56 bytes copied, 0.000171637 s, 326 kB/s 2020-11-20T12:04:29.929 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-20T12:04:29.930 INFO:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2020-11-20T12:04:29.976 INFO:teuthology.orchestra.run.smithi189.stderr:+ sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2020-11-20T12:04:29.985 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.a] 2020-11-20T12:04:29.986 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDHsLdfzuZEJhAATAlwaL0m9MdydSN0ayIreA== 2020-11-20T12:04:29.986 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2020-11-20T12:04:29.986 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2020-11-20T12:04:29.986 INFO:teuthology.orchestra.run.smithi189.stderr:56 bytes copied, 0.000166621 s, 336 kB/s 2020-11-20T12:04:29.987 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-20T12:04:29.988 INFO:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2020-11-20T12:04:30.032 INFO:teuthology.orchestra.run.smithi189.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2020-11-20T12:04:30.041 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.0] 2020-11-20T12:04:30.042 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDKsLdf3pdKKhAAD2Ms0pN7mFpoGsx6dv9HdA== 2020-11-20T12:04:30.042 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2020-11-20T12:04:30.042 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2020-11-20T12:04:30.042 INFO:teuthology.orchestra.run.smithi189.stderr:56 bytes copied, 0.000160431 s, 349 kB/s 2020-11-20T12:04:30.043 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-20T12:04:30.044 INFO:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2020-11-20T12:04:30.089 INFO:teuthology.orchestra.run.smithi189.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2020-11-20T12:04:30.097 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.1] 2020-11-20T12:04:30.097 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDLsLdfZOeQBhAAGiAijgKaHiCc24RGtlR10g== 2020-11-20T12:04:30.098 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2020-11-20T12:04:30.098 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2020-11-20T12:04:30.098 INFO:teuthology.orchestra.run.smithi189.stderr:56 bytes copied, 0.000174793 s, 320 kB/s 2020-11-20T12:04:30.099 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-20T12:04:30.099 INFO:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2020-11-20T12:04:30.144 INFO:teuthology.orchestra.run.smithi189.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2020-11-20T12:04:30.153 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.2] 2020-11-20T12:04:30.153 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDLsLdfZ5gfIRAAzhwGyhv3TmVPVeT6pbvuVg== 2020-11-20T12:04:30.153 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2020-11-20T12:04:30.154 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2020-11-20T12:04:30.154 INFO:teuthology.orchestra.run.smithi189.stderr:56 bytes copied, 0.000156984 s, 357 kB/s 2020-11-20T12:04:30.155 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-20T12:04:30.155 INFO:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2020-11-20T12:04:30.201 INFO:teuthology.orchestra.run.smithi189.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2020-11-20T12:04:30.209 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.3] 2020-11-20T12:04:30.210 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDLsLdfsA3yMhAAjS49fwgVmrYeS8gg+1gygg== 2020-11-20T12:04:30.210 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2020-11-20T12:04:30.210 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2020-11-20T12:04:30.210 INFO:teuthology.orchestra.run.smithi189.stderr:56 bytes copied, 0.000156954 s, 357 kB/s 2020-11-20T12:04:30.211 INFO:teuthology.orchestra.run.smithi047:> set -ex 2020-11-20T12:04:30.212 INFO:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2020-11-20T12:04:30.217 INFO:teuthology.orchestra.run.smithi047.stderr:+ sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2020-11-20T12:04:30.227 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr.x] 2020-11-20T12:04:30.227 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDHsLdfl8xkIxAAchO38ceJjz7t3tyPvszbqg== 2020-11-20T12:04:30.228 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records in 2020-11-20T12:04:30.228 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records out 2020-11-20T12:04:30.228 INFO:teuthology.orchestra.run.smithi047.stderr:56 bytes copied, 0.000175096 s, 320 kB/s 2020-11-20T12:04:30.229 INFO:teuthology.orchestra.run.smithi047:> set -ex 2020-11-20T12:04:30.229 INFO:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2020-11-20T12:04:30.272 INFO:teuthology.orchestra.run.smithi047.stderr:+ sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2020-11-20T12:04:30.281 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.b] 2020-11-20T12:04:30.282 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDHsLdffKl3LRAAjO8NFP67PxAsPoKpp4K+VA== 2020-11-20T12:04:30.282 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records in 2020-11-20T12:04:30.282 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records out 2020-11-20T12:04:30.282 INFO:teuthology.orchestra.run.smithi047.stderr:56 bytes copied, 0.00022817 s, 245 kB/s 2020-11-20T12:04:30.283 INFO:teuthology.orchestra.run.smithi047:> set -ex 2020-11-20T12:04:30.284 INFO:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2020-11-20T12:04:30.328 INFO:teuthology.orchestra.run.smithi047.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2020-11-20T12:04:30.337 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.4] 2020-11-20T12:04:30.338 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDNsLdfuohLCRAAKm461NAL91RyH6BWyND2og== 2020-11-20T12:04:30.338 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records in 2020-11-20T12:04:30.338 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records out 2020-11-20T12:04:30.338 INFO:teuthology.orchestra.run.smithi047.stderr:56 bytes copied, 0.000210904 s, 266 kB/s 2020-11-20T12:04:30.339 INFO:teuthology.orchestra.run.smithi047:> set -ex 2020-11-20T12:04:30.340 INFO:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2020-11-20T12:04:30.388 INFO:teuthology.orchestra.run.smithi047.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2020-11-20T12:04:30.397 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.5] 2020-11-20T12:04:30.397 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDNsLdfO3h0ExAAtJQW0thTmLxHJ1EVikCSbg== 2020-11-20T12:04:30.398 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records in 2020-11-20T12:04:30.398 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records out 2020-11-20T12:04:30.398 INFO:teuthology.orchestra.run.smithi047.stderr:56 bytes copied, 0.000172576 s, 324 kB/s 2020-11-20T12:04:30.399 INFO:teuthology.orchestra.run.smithi047:> set -ex 2020-11-20T12:04:30.400 INFO:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2020-11-20T12:04:30.444 INFO:teuthology.orchestra.run.smithi047.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2020-11-20T12:04:30.453 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.6] 2020-11-20T12:04:30.453 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDNsLdf5y7WIBAA5V6UGB5k+eDAh70UWXfSJA== 2020-11-20T12:04:30.454 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records in 2020-11-20T12:04:30.454 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records out 2020-11-20T12:04:30.454 INFO:teuthology.orchestra.run.smithi047.stderr:56 bytes copied, 0.000158727 s, 353 kB/s 2020-11-20T12:04:30.455 INFO:teuthology.orchestra.run.smithi047:> set -ex 2020-11-20T12:04:30.456 INFO:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2020-11-20T12:04:30.500 INFO:teuthology.orchestra.run.smithi047.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2020-11-20T12:04:30.509 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.7] 2020-11-20T12:04:30.509 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDNsLdfTDgtLhAAAyvIuEr/e93t+g9Wsc2pgA== 2020-11-20T12:04:30.510 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records in 2020-11-20T12:04:30.510 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records out 2020-11-20T12:04:30.510 INFO:teuthology.orchestra.run.smithi047.stderr:56 bytes copied, 0.0002066 s, 271 kB/s 2020-11-20T12:04:30.511 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-20T12:04:30.512 INFO:teuthology.orchestra.run.smithi189:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2020-11-20T12:04:30.516 INFO:teuthology.orchestra.run.smithi189.stderr:+ dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2020-11-20T12:04:30.518 INFO:teuthology.orchestra.run.smithi189.stdout:[client.0] 2020-11-20T12:04:30.518 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDHsLdfWJxeNBAAEZ/eAJsaE2f6ltAUiPLTrA== 2020-11-20T12:04:30.518 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2020-11-20T12:04:30.519 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2020-11-20T12:04:30.519 INFO:teuthology.orchestra.run.smithi189.stderr:59 bytes copied, 0.00016306 s, 362 kB/s 2020-11-20T12:04:30.519 INFO:tasks.ceph:Adding keys to all mons... 2020-11-20T12:04:30.519 INFO:teuthology.orchestra.run.smithi047:> sudo tee -a /etc/ceph/ceph.keyring 2020-11-20T12:04:30.554 INFO:teuthology.orchestra.run.smithi189:> sudo tee -a /etc/ceph/ceph.keyring 2020-11-20T12:04:30.565 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr.y] 2020-11-20T12:04:30.565 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDHsLdfPRl5IBAAYYAMBbphBlIjttwWYZ6CHQ== 2020-11-20T12:04:30.566 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.a] 2020-11-20T12:04:30.566 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDHsLdfzuZEJhAATAlwaL0m9MdydSN0ayIreA== 2020-11-20T12:04:30.566 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.0] 2020-11-20T12:04:30.567 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDKsLdf3pdKKhAAD2Ms0pN7mFpoGsx6dv9HdA== 2020-11-20T12:04:30.567 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.1] 2020-11-20T12:04:30.567 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDLsLdfZOeQBhAAGiAijgKaHiCc24RGtlR10g== 2020-11-20T12:04:30.568 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.2] 2020-11-20T12:04:30.568 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDLsLdfZ5gfIRAAzhwGyhv3TmVPVeT6pbvuVg== 2020-11-20T12:04:30.569 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.3] 2020-11-20T12:04:30.569 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDLsLdfsA3yMhAAjS49fwgVmrYeS8gg+1gygg== 2020-11-20T12:04:30.570 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr.x] 2020-11-20T12:04:30.570 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDHsLdfl8xkIxAAchO38ceJjz7t3tyPvszbqg== 2020-11-20T12:04:30.570 INFO:teuthology.orchestra.run.smithi047.stdout:[mds.b] 2020-11-20T12:04:30.571 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDHsLdffKl3LRAAjO8NFP67PxAsPoKpp4K+VA== 2020-11-20T12:04:30.571 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.4] 2020-11-20T12:04:30.571 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDNsLdfuohLCRAAKm461NAL91RyH6BWyND2og== 2020-11-20T12:04:30.572 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.5] 2020-11-20T12:04:30.572 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDNsLdfO3h0ExAAtJQW0thTmLxHJ1EVikCSbg== 2020-11-20T12:04:30.573 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.6] 2020-11-20T12:04:30.573 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDNsLdf5y7WIBAA5V6UGB5k+eDAh70UWXfSJA== 2020-11-20T12:04:30.573 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.7] 2020-11-20T12:04:30.574 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDNsLdfTDgtLhAAAyvIuEr/e93t+g9Wsc2pgA== 2020-11-20T12:04:30.574 INFO:teuthology.orchestra.run.smithi047.stdout:[client.0] 2020-11-20T12:04:30.574 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDHsLdfWJxeNBAAEZ/eAJsaE2f6ltAUiPLTrA== 2020-11-20T12:04:30.576 INFO:teuthology.orchestra.run.smithi189.stdout:[mgr.y] 2020-11-20T12:04:30.576 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDHsLdfPRl5IBAAYYAMBbphBlIjttwWYZ6CHQ== 2020-11-20T12:04:30.577 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.a] 2020-11-20T12:04:30.577 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDHsLdfzuZEJhAATAlwaL0m9MdydSN0ayIreA== 2020-11-20T12:04:30.577 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.0] 2020-11-20T12:04:30.577 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDKsLdf3pdKKhAAD2Ms0pN7mFpoGsx6dv9HdA== 2020-11-20T12:04:30.578 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.1] 2020-11-20T12:04:30.578 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDLsLdfZOeQBhAAGiAijgKaHiCc24RGtlR10g== 2020-11-20T12:04:30.578 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.2] 2020-11-20T12:04:30.578 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDLsLdfZ5gfIRAAzhwGyhv3TmVPVeT6pbvuVg== 2020-11-20T12:04:30.579 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.3] 2020-11-20T12:04:30.579 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDLsLdfsA3yMhAAjS49fwgVmrYeS8gg+1gygg== 2020-11-20T12:04:30.579 INFO:teuthology.orchestra.run.smithi189.stdout:[mgr.x] 2020-11-20T12:04:30.579 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDHsLdfl8xkIxAAchO38ceJjz7t3tyPvszbqg== 2020-11-20T12:04:30.579 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.b] 2020-11-20T12:04:30.580 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDHsLdffKl3LRAAjO8NFP67PxAsPoKpp4K+VA== 2020-11-20T12:04:30.580 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.4] 2020-11-20T12:04:30.580 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDNsLdfuohLCRAAKm461NAL91RyH6BWyND2og== 2020-11-20T12:04:30.580 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.5] 2020-11-20T12:04:30.580 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDNsLdfO3h0ExAAtJQW0thTmLxHJ1EVikCSbg== 2020-11-20T12:04:30.581 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.6] 2020-11-20T12:04:30.581 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDNsLdf5y7WIBAA5V6UGB5k+eDAh70UWXfSJA== 2020-11-20T12:04:30.581 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.7] 2020-11-20T12:04:30.581 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDNsLdfTDgtLhAAAyvIuEr/e93t+g9Wsc2pgA== 2020-11-20T12:04:30.581 INFO:teuthology.orchestra.run.smithi189.stdout:[client.0] 2020-11-20T12:04:30.582 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQDHsLdfWJxeNBAAEZ/eAJsaE2f6ltAUiPLTrA== 2020-11-20T12:04:30.582 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-11-20T12:04:30.621 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-11-20T12:04:30.665 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2020-11-20T12:04:30.705 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2020-11-20T12:04:30.752 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:30.754 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:30.838 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:30.841 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:30.926 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:30.929 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:31.012 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:31.015 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:31.098 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-11-20T12:04:31.101 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-11-20T12:04:31.182 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2020-11-20T12:04:31.185 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2020-11-20T12:04:31.269 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:31.273 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:31.314 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:31.357 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:31.401 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:31.441 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:31.486 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:31.525 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T12:04:31.569 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-20T12:04:31.609 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-20T12:04:31.656 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-11-20T12:04:31.657 INFO:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2020-11-20T12:04:31.715 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2020-11-20T12:04:31.827 INFO:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2020-11-20T12:04:31.844 INFO:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2020-11-20T12:04:31.860 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2020-11-20T12:04:31.976 INFO:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2020-11-20T12:04:31.992 INFO:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2020-11-20T12:04:32.050 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2020-11-20T12:04:32.169 INFO:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2020-11-20T12:04:32.215 INFO:teuthology.orchestra.run.smithi047:> rm -- /home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:04:32.281 INFO:teuthology.orchestra.run.smithi189:> rm -- /home/ubuntu/cephtest/ceph.monmap 2020-11-20T12:04:32.294 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2020-11-20T12:04:32.294 INFO:tasks.ceph.mon.a:Restarting daemon 2020-11-20T12:04:32.295 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2020-11-20T12:04:32.343 INFO:tasks.ceph.mon.a:Started 2020-11-20T12:04:32.344 INFO:tasks.ceph.mon.b:Restarting daemon 2020-11-20T12:04:32.345 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2020-11-20T12:04:32.362 INFO:tasks.ceph.mon.b:Started 2020-11-20T12:04:32.363 INFO:tasks.ceph.mon.c:Restarting daemon 2020-11-20T12:04:32.364 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2020-11-20T12:04:32.374 INFO:tasks.ceph.mon.c:Started 2020-11-20T12:04:32.374 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2020-11-20T12:04:32.374 INFO:tasks.ceph.mgr.y:Restarting daemon 2020-11-20T12:04:32.375 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2020-11-20T12:04:32.382 INFO:tasks.ceph.mgr.y:Started 2020-11-20T12:04:32.383 INFO:tasks.ceph.mgr.x:Restarting daemon 2020-11-20T12:04:32.383 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2020-11-20T12:04:32.394 INFO:tasks.ceph.mgr.x:Started 2020-11-20T12:04:32.395 INFO:tasks.ceph:Setting crush tunables to default 2020-11-20T12:04:32.395 INFO:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd crush tunables default 2020-11-20T12:04:32.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.653+0000 7f012492f700 1 Processor -- start 2020-11-20T12:04:32.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.653+0000 7f012492f700 1 -- start start 2020-11-20T12:04:32.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.653+0000 7f012492f700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f01200b19e0 0x7f012013a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:32.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.653+0000 7f012492f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0120066a30 0x7f012013ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:32.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.653+0000 7f012492f700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0120066fb0 con 0x7f0120062930 2020-11-20T12:04:32.656 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.653+0000 7f012492f700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01200666b0 con 0x7f01200b19e0 2020-11-20T12:04:32.656 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.653+0000 7f012492f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0120067250 con 0x7f0120066a30 2020-11-20T12:04:32.656 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.653+0000 7f011f59e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0120066a30 0x7f012013ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:32.656 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.653+0000 7f011f59e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0120066a30 0x7f012013ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47664/0 (socket says 172.21.15.189:47664) 2020-11-20T12:04:32.657 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.653+0000 7f011f59e700 1 -- 172.21.15.189:0/3328460488 learned_addr learned my addr 172.21.15.189:0/3328460488 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:32.657 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.653+0000 7f011f59e700 1 -- 172.21.15.189:0/3328460488 >> v2:172.21.15.189:3300/0 conn(0x7f0120066a30 msgr2=0x7f012013ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T12:04:32.657 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.653+0000 7f011f59e700 1 -- 172.21.15.189:0/3328460488 >> v2:172.21.15.189:3300/0 conn(0x7f0120066a30 msgr2=0x7f012013ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:32.657 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.653+0000 7f011f59e700 1 --2- 172.21.15.189:0/3328460488 >> v2:172.21.15.189:3300/0 conn(0x7f0120066a30 0x7f012013ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:32.658 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.653+0000 7f011e59c700 1 --2- 172.21.15.189:0/3328460488 >> v2:172.21.15.47:3300/0 conn(0x7f01200b19e0 0x7f012013a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:32.658 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.653+0000 7f011f59e700 1 --2- 172.21.15.189:0/3328460488 >> v2:172.21.15.189:3300/0 conn(0x7f0120066a30 0x7f012013ace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T12:04:32.658 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.653+0000 7f011e59c700 1 -- 172.21.15.189:0/3328460488 >> v2:172.21.15.47:3300/0 conn(0x7f01200b19e0 msgr2=0x7f012013a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T12:04:32.658 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.653+0000 7f011e59c700 1 -- 172.21.15.189:0/3328460488 >> v2:172.21.15.47:3300/0 conn(0x7f01200b19e0 msgr2=0x7f012013a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:32.658 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.653+0000 7f011e59c700 1 --2- 172.21.15.189:0/3328460488 >> v2:172.21.15.47:3300/0 conn(0x7f01200b19e0 0x7f012013a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:32.659 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.653+0000 7f011e59c700 1 --2- 172.21.15.189:0/3328460488 >> v2:172.21.15.47:3300/0 conn(0x7f01200b19e0 0x7f012013a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T12:04:32.855 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.853+0000 7f011f59e700 1 --2- 172.21.15.189:0/3328460488 >> v2:172.21.15.189:3300/0 conn(0x7f0120066a30 0x7f012013ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:32.855 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.853+0000 7f011f59e700 1 -- 172.21.15.189:0/3328460488 >> v2:172.21.15.189:3300/0 conn(0x7f0120066a30 msgr2=0x7f012013ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T12:04:32.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.853+0000 7f011f59e700 1 -- 172.21.15.189:0/3328460488 >> v2:172.21.15.189:3300/0 conn(0x7f0120066a30 msgr2=0x7f012013ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:32.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.853+0000 7f011f59e700 1 --2- 172.21.15.189:0/3328460488 >> v2:172.21.15.189:3300/0 conn(0x7f0120066a30 0x7f012013ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:32.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.853+0000 7f011f59e700 1 --2- 172.21.15.189:0/3328460488 >> v2:172.21.15.189:3300/0 conn(0x7f0120066a30 0x7f012013ace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T12:04:32.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.853+0000 7f011e59c700 1 --2- 172.21.15.189:0/3328460488 >> v2:172.21.15.47:3300/0 conn(0x7f01200b19e0 0x7f012013a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:32.857 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.853+0000 7f011e59c700 1 -- 172.21.15.189:0/3328460488 >> v2:172.21.15.47:3300/0 conn(0x7f01200b19e0 msgr2=0x7f012013a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T12:04:32.857 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.853+0000 7f011e59c700 1 -- 172.21.15.189:0/3328460488 >> v2:172.21.15.47:3300/0 conn(0x7f01200b19e0 msgr2=0x7f012013a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:32.857 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.853+0000 7f011e59c700 1 --2- 172.21.15.189:0/3328460488 >> v2:172.21.15.47:3300/0 conn(0x7f01200b19e0 0x7f012013a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:32.857 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:32.853+0000 7f011e59c700 1 --2- 172.21.15.189:0/3328460488 >> v2:172.21.15.47:3300/0 conn(0x7f01200b19e0 0x7f012013a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T12:04:33.256 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:33.253+0000 7f011f59e700 1 --2- 172.21.15.189:0/3328460488 >> v2:172.21.15.189:3300/0 conn(0x7f0120066a30 0x7f012013ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:33.257 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:33.253+0000 7f011f59e700 1 -- 172.21.15.189:0/3328460488 >> v2:172.21.15.189:3300/0 conn(0x7f0120066a30 msgr2=0x7f012013ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T12:04:33.257 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:33.253+0000 7f011f59e700 1 -- 172.21.15.189:0/3328460488 >> v2:172.21.15.189:3300/0 conn(0x7f0120066a30 msgr2=0x7f012013ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:33.257 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:33.253+0000 7f011f59e700 1 --2- 172.21.15.189:0/3328460488 >> v2:172.21.15.189:3300/0 conn(0x7f0120066a30 0x7f012013ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:33.257 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:33.253+0000 7f011f59e700 1 --2- 172.21.15.189:0/3328460488 >> v2:172.21.15.189:3300/0 conn(0x7f0120066a30 0x7f012013ace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T12:04:33.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:33.257+0000 7f011e59c700 1 --2- 172.21.15.189:0/3328460488 >> v2:172.21.15.47:3300/0 conn(0x7f01200b19e0 0x7f012013a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:33.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:33.257+0000 7f011e59c700 1 -- 172.21.15.189:0/3328460488 >> v2:172.21.15.47:3300/0 conn(0x7f01200b19e0 msgr2=0x7f012013a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T12:04:33.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:33.257+0000 7f011e59c700 1 -- 172.21.15.189:0/3328460488 >> v2:172.21.15.47:3300/0 conn(0x7f01200b19e0 msgr2=0x7f012013a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:33.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:33.257+0000 7f011e59c700 1 --2- 172.21.15.189:0/3328460488 >> v2:172.21.15.47:3300/0 conn(0x7f01200b19e0 0x7f012013a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:33.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:33.257+0000 7f011e59c700 1 --2- 172.21.15.189:0/3328460488 >> v2:172.21.15.47:3300/0 conn(0x7f01200b19e0 0x7f012013a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T12:04:34.058 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:34.057+0000 7f011f59e700 1 --2- 172.21.15.189:0/3328460488 >> v2:172.21.15.189:3300/0 conn(0x7f0120066a30 0x7f012013ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:34.058 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:34.057+0000 7f011f59e700 1 -- 172.21.15.189:0/3328460488 >> v2:172.21.15.189:3300/0 conn(0x7f0120066a30 msgr2=0x7f012013ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T12:04:34.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:34.057+0000 7f011f59e700 1 -- 172.21.15.189:0/3328460488 >> v2:172.21.15.189:3300/0 conn(0x7f0120066a30 msgr2=0x7f012013ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:34.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:34.057+0000 7f011f59e700 1 --2- 172.21.15.189:0/3328460488 >> v2:172.21.15.189:3300/0 conn(0x7f0120066a30 0x7f012013ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:34.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:34.057+0000 7f011f59e700 1 --2- 172.21.15.189:0/3328460488 >> v2:172.21.15.189:3300/0 conn(0x7f0120066a30 0x7f012013ace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T12:04:34.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:34.057+0000 7f011e59c700 1 --2- 172.21.15.189:0/3328460488 >> v2:172.21.15.47:3300/0 conn(0x7f01200b19e0 0x7f012013a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:34.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:34.057+0000 7f011e59c700 1 -- 172.21.15.189:0/3328460488 >> v2:172.21.15.47:3300/0 conn(0x7f01200b19e0 msgr2=0x7f012013a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T12:04:34.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:34.057+0000 7f011e59c700 1 -- 172.21.15.189:0/3328460488 >> v2:172.21.15.47:3300/0 conn(0x7f01200b19e0 msgr2=0x7f012013a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:34.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:34.057+0000 7f011e59c700 1 --2- 172.21.15.189:0/3328460488 >> v2:172.21.15.47:3300/0 conn(0x7f01200b19e0 0x7f012013a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:34.061 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:34.057+0000 7f011e59c700 1 --2- 172.21.15.189:0/3328460488 >> v2:172.21.15.47:3300/0 conn(0x7f01200b19e0 0x7f012013a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T12:04:35.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:35.653+0000 7f011cd99700 1 -- 172.21.15.189:0/3328460488 >> v2:172.21.15.189:3300/0 conn(0x7f0120066a30 msgr2=0x7f012013ace0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T12:04:35.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:35.653+0000 7f011cd99700 1 --2- 172.21.15.189:0/3328460488 >> v2:172.21.15.189:3300/0 conn(0x7f0120066a30 0x7f012013ace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:35.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:35.653+0000 7f011cd99700 1 -- 172.21.15.189:0/3328460488 >> v2:172.21.15.47:3300/0 conn(0x7f01200b19e0 msgr2=0x7f012013a7f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T12:04:35.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:35.653+0000 7f011cd99700 1 --2- 172.21.15.189:0/3328460488 >> v2:172.21.15.47:3300/0 conn(0x7f01200b19e0 0x7f012013a7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:35.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:35.653+0000 7f011cd99700 1 -- 172.21.15.189:0/3328460488 >> v1:172.21.15.47:6789/0 conn(0x7f0120062930 legacy=0x7f01200b12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:35.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:35.653+0000 7f011cd99700 1 --2- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0104003380 0x7f01040057d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:35.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:35.653+0000 7f011cd99700 1 -- 172.21.15.189:0/3328460488 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01200b0e30 con 0x7f0104005df0 2020-11-20T12:04:35.656 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:35.653+0000 7f011cd99700 1 -- 172.21.15.189:0/3328460488 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0120067250 con 0x7f0104003380 2020-11-20T12:04:35.656 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:35.653+0000 7f011ed9d700 1 --2- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0104003380 0x7f01040057d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:35.657 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:35.653+0000 7f011ed9d700 1 -- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0104003380 msgr2=0x7f01040057d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T12:04:35.657 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:35.653+0000 7f011ed9d700 1 -- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0104003380 msgr2=0x7f01040057d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:35.657 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:35.653+0000 7f011ed9d700 1 --2- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0104003380 0x7f01040057d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:35.658 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:35.653+0000 7f011ed9d700 1 --2- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0104003380 0x7f01040057d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T12:04:35.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:35.853+0000 7f011ed9d700 1 --2- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0104003380 0x7f01040057d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:35.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:35.853+0000 7f011ed9d700 1 -- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0104003380 msgr2=0x7f01040057d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T12:04:35.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:35.853+0000 7f011ed9d700 1 -- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0104003380 msgr2=0x7f01040057d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:35.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:35.853+0000 7f011ed9d700 1 --2- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0104003380 0x7f01040057d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:35.857 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:35.853+0000 7f011ed9d700 1 --2- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0104003380 0x7f01040057d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T12:04:36.257 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:36.257+0000 7f011ed9d700 1 --2- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0104003380 0x7f01040057d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:36.257 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:36.257+0000 7f011ed9d700 1 -- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0104003380 msgr2=0x7f01040057d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T12:04:36.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:36.257+0000 7f011ed9d700 1 -- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0104003380 msgr2=0x7f01040057d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:36.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:36.257+0000 7f011ed9d700 1 --2- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0104003380 0x7f01040057d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:36.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:36.257+0000 7f011ed9d700 1 --2- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0104003380 0x7f01040057d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T12:04:37.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.057+0000 7f011ed9d700 1 --2- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0104003380 0x7f01040057d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:37.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.057+0000 7f011ed9d700 1 -- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0104003380 msgr2=0x7f01040057d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T12:04:37.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.057+0000 7f011ed9d700 1 -- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0104003380 msgr2=0x7f01040057d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T12:04:37.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.057+0000 7f011ed9d700 1 --2- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0104003380 0x7f01040057d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T12:04:37.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.057+0000 7f011ed9d700 1 --2- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0104003380 0x7f01040057d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T12:04:37.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.565+0000 7f011dd9b700 1 -- 172.21.15.189:0/3328460488 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818809606 0 0) 0x7f01200b0e30 con 0x7f0104005df0 2020-11-20T12:04:37.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.565+0000 7f011dd9b700 1 -- 172.21.15.189:0/3328460488 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0100003240 con 0x7f0104005df0 2020-11-20T12:04:37.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.569+0000 7f011dd9b700 1 -- 172.21.15.189:0/3328460488 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 729973969 0 0) 0x7f0100003240 con 0x7f0104005df0 2020-11-20T12:04:37.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.569+0000 7f011dd9b700 1 -- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0104003380 msgr2=0x7f01040057d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T12:04:37.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.569+0000 7f011dd9b700 1 --2- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0104003380 0x7f01040057d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:37.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.569+0000 7f011dd9b700 1 -- 172.21.15.189:0/3328460488 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f010400abd0 con 0x7f0104005df0 2020-11-20T12:04:37.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.569+0000 7f011dd9b700 1 -- 172.21.15.189:0/3328460488 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f010800d1c0 con 0x7f0104005df0 2020-11-20T12:04:37.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.569+0000 7f011dd9b700 1 -- 172.21.15.189:0/3328460488 >> v1:172.21.15.189:6789/0 conn(0x7f0104005df0 legacy=0x7f0104008290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:37.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.569+0000 7f011f59e700 1 -- 172.21.15.189:0/3328460488 reap_dead start 2020-11-20T12:04:37.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.569+0000 7f011dd9b700 1 --2- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0100003730 0x7f0100003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:37.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.569+0000 7f011dd9b700 1 --2- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01000044e0 0x7f0100004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:37.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.569+0000 7f011dd9b700 1 --2- 172.21.15.189:0/3328460488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0100004e50 0x7f010000b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:37.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.569+0000 7f011dd9b700 1 -- 172.21.15.189:0/3328460488 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0120067250 con 0x7f01000044e0 2020-11-20T12:04:37.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.569+0000 7f011dd9b700 1 -- 172.21.15.189:0/3328460488 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0100005360 con 0x7f0100004e50 2020-11-20T12:04:37.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.569+0000 7f011dd9b700 1 -- 172.21.15.189:0/3328460488 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f010000bc50 con 0x7f0100003730 2020-11-20T12:04:37.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.569+0000 7f011ed9d700 1 --2- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0100003730 0x7f0100003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:37.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.569+0000 7f011f59e700 1 --2- 172.21.15.189:0/3328460488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0100004e50 0x7f010000b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:37.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.569+0000 7f011f59e700 1 -- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0100003730 msgr2=0x7f0100003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:37.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.569+0000 7f011f59e700 1 --2- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0100003730 0x7f0100003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:37.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.569+0000 7f011f59e700 1 -- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01000044e0 msgr2=0x7f0100004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:37.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.569+0000 7f011f59e700 1 --2- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01000044e0 0x7f0100004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:04:37.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.569+0000 7f011f59e700 1 -- 172.21.15.189:0/3328460488 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0100003240 con 0x7f0100004e50 2020-11-20T12:04:37.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.569+0000 7f011f59e700 1 --2- 172.21.15.189:0/3328460488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0100004e50 0x7f010000b730 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f012006de20 tx=0x7f0118008040).ready entity=mon.1 client_cookie=ece40c7a2c51ce64 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:37.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.573+0000 7f011dd9b700 1 -- 172.21.15.189:0/3328460488 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0118014a80 con 0x7f0100004e50 2020-11-20T12:04:37.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.573+0000 7f011dd9b700 1 -- 172.21.15.189:0/3328460488 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f011800c040 con 0x7f0100004e50 2020-11-20T12:04:37.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.573+0000 7f011dd9b700 1 -- 172.21.15.189:0/3328460488 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0118014df0 con 0x7f0100004e50 2020-11-20T12:04:37.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.573+0000 7f012492f700 1 -- 172.21.15.189:0/3328460488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0100004e50 msgr2=0x7f010000b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:37.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.573+0000 7f012492f700 1 --2- 172.21.15.189:0/3328460488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0100004e50 0x7f010000b730 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f012006de20 tx=0x7f0118008040).stop 2020-11-20T12:04:37.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.573+0000 7f012492f700 1 -- 172.21.15.189:0/3328460488 shutdown_connections 2020-11-20T12:04:37.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.573+0000 7f012492f700 1 --2- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01000044e0 0x7f0100004960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:04:37.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.573+0000 7f012492f700 1 --2- 172.21.15.189:0/3328460488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0100004e50 0x7f010000b730 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:37.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.573+0000 7f012492f700 1 --2- 172.21.15.189:0/3328460488 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0100003730 0x7f0100003b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:37.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.573+0000 7f012492f700 1 -- 172.21.15.189:0/3328460488 >> 172.21.15.189:0/3328460488 conn(0x7f0120065080 msgr2=0x7f0120062080 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:37.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.573+0000 7f012492f700 1 -- 172.21.15.189:0/3328460488 shutdown_connections 2020-11-20T12:04:37.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.573+0000 7f012492f700 1 -- 172.21.15.189:0/3328460488 wait complete. 2020-11-20T12:04:37.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.577+0000 7f012492f700 1 Processor -- start 2020-11-20T12:04:37.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.577+0000 7f012492f700 1 -- start start 2020-11-20T12:04:37.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.581+0000 7f012492f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0100003730 0x7f012013edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:37.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.581+0000 7f012492f700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f01000044e0 0x7f0120149b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:37.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.581+0000 7f012492f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0100004e50 0x7f01200a8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:37.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.581+0000 7f012492f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01200a7c50 con 0x7f0100003730 2020-11-20T12:04:37.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.581+0000 7f012492f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012014a320 con 0x7f0100004e50 2020-11-20T12:04:37.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.581+0000 7f012492f700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01200a89e0 con 0x7f01000044e0 2020-11-20T12:04:37.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.581+0000 7f011f59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0100004e50 0x7f01200a8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:37.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.581+0000 7f011ed9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0100003730 0x7f012013edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:37.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.581+0000 7f011e59c700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f01000044e0 0x7f0120149b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:37.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.581+0000 7f011f59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0100004e50 0x7f01200a8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47734/0 (socket says 172.21.15.189:47734) 2020-11-20T12:04:37.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.581+0000 7f011f59e700 1 -- 172.21.15.189:0/4160678454 learned_addr learned my addr 172.21.15.189:0/4160678454 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:37.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.581+0000 7f011f59e700 1 -- 172.21.15.189:0/4160678454 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f01000044e0 msgr2=0x7f0120149b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:37.588 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.581+0000 7f011f59e700 1 --2- 172.21.15.189:0/4160678454 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f01000044e0 0x7f0120149b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:37.588 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.581+0000 7f011f59e700 1 -- 172.21.15.189:0/4160678454 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0100003730 msgr2=0x7f012013edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:37.588 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.581+0000 7f011f59e700 1 --2- 172.21.15.189:0/4160678454 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0100003730 0x7f012013edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:37.588 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.581+0000 7f011f59e700 1 -- 172.21.15.189:0/4160678454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012014a050 con 0x7f0100004e50 2020-11-20T12:04:37.589 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.581+0000 7f011f59e700 1 --2- 172.21.15.189:0/4160678454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0100004e50 0x7f01200a8e80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f0120063040 tx=0x7f0118008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:37.589 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.581+0000 7f0116ffd700 1 -- 172.21.15.189:0/4160678454 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01180032b0 con 0x7f0100004e50 2020-11-20T12:04:37.589 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.581+0000 7f0116ffd700 1 -- 172.21.15.189:0/4160678454 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0118002030 con 0x7f0100004e50 2020-11-20T12:04:37.589 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.581+0000 7f012492f700 1 -- 172.21.15.189:0/4160678454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01200aa630 con 0x7f0100004e50 2020-11-20T12:04:37.589 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.581+0000 7f0116ffd700 1 -- 172.21.15.189:0/4160678454 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0118020890 con 0x7f0100004e50 2020-11-20T12:04:37.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.581+0000 7f012492f700 1 -- 172.21.15.189:0/4160678454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01200a9590 con 0x7f0100004e50 2020-11-20T12:04:37.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.585+0000 7f0116ffd700 1 -- 172.21.15.189:0/4160678454 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f0118012030 con 0x7f0100004e50 2020-11-20T12:04:37.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.585+0000 7f012492f700 1 -- 172.21.15.189:0/4160678454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01200a8ba0 con 0x7f0100004e50 2020-11-20T12:04:37.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.585+0000 7f0116ffd700 1 -- 172.21.15.189:0/4160678454 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f011802b700 con 0x7f0100004e50 2020-11-20T12:04:37.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.589+0000 7f0116ffd700 1 -- 172.21.15.189:0/4160678454 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7f0118030060 con 0x7f0100004e50 2020-11-20T12:04:37.667 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.665+0000 7f012492f700 1 -- 172.21.15.189:0/4160678454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f0120062d10 con 0x7f0100004e50 2020-11-20T12:04:37.675 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.673+0000 7f0116ffd700 1 -- 172.21.15.189:0/4160678454 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f011801b020 con 0x7f0100004e50 2020-11-20T12:04:37.678 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.677+0000 7f012492f700 1 -- 172.21.15.189:0/4160678454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0100004e50 msgr2=0x7f01200a8e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:37.678 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.677+0000 7f012492f700 1 --2- 172.21.15.189:0/4160678454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0100004e50 0x7f01200a8e80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f0120063040 tx=0x7f0118008040).stop 2020-11-20T12:04:37.678 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.677+0000 7f012492f700 1 -- 172.21.15.189:0/4160678454 shutdown_connections 2020-11-20T12:04:37.679 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.677+0000 7f012492f700 1 --2- 172.21.15.189:0/4160678454 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0100003730 0x7f012013edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:37.679 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.677+0000 7f012492f700 1 --2- 172.21.15.189:0/4160678454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0100004e50 0x7f01200a8e80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:37.679 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.677+0000 7f012492f700 1 --2- 172.21.15.189:0/4160678454 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f01000044e0 0x7f0120149b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:37.679 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.677+0000 7f012492f700 1 -- 172.21.15.189:0/4160678454 >> 172.21.15.189:0/4160678454 conn(0x7f0120065080 msgr2=0x7f01200608c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:37.680 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.677+0000 7f012492f700 1 -- 172.21.15.189:0/4160678454 shutdown_connections 2020-11-20T12:04:37.680 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:37.677+0000 7f012492f700 1 -- 172.21.15.189:0/4160678454 wait complete. 2020-11-20T12:04:37.680 INFO:teuthology.orchestra.run.smithi189.stderr:adjusted tunables profile to default 2020-11-20T12:04:37.694 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2020-11-20T12:04:37.694 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-20T12:04:37.695 INFO:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2020-11-20T12:04:37.700 INFO:teuthology.orchestra.run.smithi189.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2020-11-20T12:04:37.708 INFO:teuthology.orchestra.run.smithi189.stdout:c81214e1-4f84-4aaa-8ceb-99c398b2a89a 2020-11-20T12:04:37.709 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2020-11-20T12:04:37.709 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2020-11-20T12:04:37.709 INFO:teuthology.orchestra.run.smithi189.stderr:37 bytes copied, 0.000154133 s, 240 kB/s 2020-11-20T12:04:37.710 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-20T12:04:37.710 INFO:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2020-11-20T12:04:37.756 INFO:teuthology.orchestra.run.smithi189.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2020-11-20T12:04:37.765 INFO:teuthology.orchestra.run.smithi189.stdout:d2f3ec7d-d403-484f-b5ed-b2e6b079d63d 2020-11-20T12:04:37.766 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2020-11-20T12:04:37.766 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2020-11-20T12:04:37.766 INFO:teuthology.orchestra.run.smithi189.stderr:37 bytes copied, 0.00020714 s, 179 kB/s 2020-11-20T12:04:37.767 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-20T12:04:37.768 INFO:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2020-11-20T12:04:37.812 INFO:teuthology.orchestra.run.smithi189.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2020-11-20T12:04:37.819 INFO:teuthology.orchestra.run.smithi189.stdout:e6cf4790-000c-43eb-8844-175f04c60503 2020-11-20T12:04:37.820 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2020-11-20T12:04:37.820 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2020-11-20T12:04:37.820 INFO:teuthology.orchestra.run.smithi189.stderr:37 bytes copied, 0.000156837 s, 236 kB/s 2020-11-20T12:04:37.821 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-20T12:04:37.821 INFO:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2020-11-20T12:04:37.868 INFO:teuthology.orchestra.run.smithi189.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2020-11-20T12:04:37.876 INFO:teuthology.orchestra.run.smithi189.stdout:ba49dfac-f2b9-476e-853d-42f05e21a77a 2020-11-20T12:04:37.876 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2020-11-20T12:04:37.876 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2020-11-20T12:04:37.876 INFO:teuthology.orchestra.run.smithi189.stderr:37 bytes copied, 0.00015651 s, 236 kB/s 2020-11-20T12:04:37.877 INFO:teuthology.orchestra.run.smithi047:> set -ex 2020-11-20T12:04:37.878 INFO:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2020-11-20T12:04:37.883 INFO:teuthology.orchestra.run.smithi047.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2020-11-20T12:04:37.893 INFO:teuthology.orchestra.run.smithi047.stdout:3ca7b438-38b2-4689-8b18-fb5a8ddf6aee 2020-11-20T12:04:37.893 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records in 2020-11-20T12:04:37.893 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records out 2020-11-20T12:04:37.894 INFO:teuthology.orchestra.run.smithi047.stderr:37 bytes copied, 0.0001654 s, 224 kB/s 2020-11-20T12:04:37.894 INFO:teuthology.orchestra.run.smithi047:> set -ex 2020-11-20T12:04:37.895 INFO:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2020-11-20T12:04:37.940 INFO:teuthology.orchestra.run.smithi047.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2020-11-20T12:04:37.948 INFO:teuthology.orchestra.run.smithi047.stdout:6a45fca7-83d3-44fd-a1d4-b01e4201a101 2020-11-20T12:04:37.949 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records in 2020-11-20T12:04:37.950 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records out 2020-11-20T12:04:37.951 INFO:teuthology.orchestra.run.smithi047.stderr:37 bytes copied, 0.000137483 s, 269 kB/s 2020-11-20T12:04:37.952 INFO:teuthology.orchestra.run.smithi047:> set -ex 2020-11-20T12:04:37.953 INFO:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2020-11-20T12:04:37.999 INFO:teuthology.orchestra.run.smithi047.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2020-11-20T12:04:38.011 INFO:teuthology.orchestra.run.smithi047.stdout:c4b1b9ed-61f8-4677-9755-dff59a30e447 2020-11-20T12:04:38.011 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records in 2020-11-20T12:04:38.011 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records out 2020-11-20T12:04:38.012 INFO:teuthology.orchestra.run.smithi047.stderr:37 bytes copied, 0.000159503 s, 232 kB/s 2020-11-20T12:04:38.012 INFO:teuthology.orchestra.run.smithi047:> set -ex 2020-11-20T12:04:38.013 INFO:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2020-11-20T12:04:38.055 INFO:teuthology.orchestra.run.smithi047.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2020-11-20T12:04:38.064 INFO:teuthology.orchestra.run.smithi047.stdout:4608e4ed-cc40-4c5c-ba8b-3779bbc62f65 2020-11-20T12:04:38.065 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records in 2020-11-20T12:04:38.065 INFO:teuthology.orchestra.run.smithi047.stderr:0+1 records out 2020-11-20T12:04:38.065 INFO:teuthology.orchestra.run.smithi047.stderr:37 bytes copied, 0.000152735 s, 242 kB/s 2020-11-20T12:04:38.066 INFO:teuthology.orchestra.run.smithi047:> sudo ceph --cluster ceph osd new c81214e1-4f84-4aaa-8ceb-99c398b2a89a 0 2020-11-20T12:04:38.286 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.281+0000 7fe921d91700 1 Processor -- start 2020-11-20T12:04:38.287 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.281+0000 7fe921d91700 1 -- start start 2020-11-20T12:04:38.288 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.281+0000 7fe921d91700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe91c060310 0x7fe91c13a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:38.289 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.281+0000 7fe921d91700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe91c066ba0 0x7fe91c0a0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:38.289 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.281+0000 7fe921d91700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe91c0a0a10 0x7fe91c137360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:38.289 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.281+0000 7fe921d91700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe91c0674b0 con 0x7fe91c0a0a10 2020-11-20T12:04:38.289 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.281+0000 7fe921d91700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe91c0a8fc0 con 0x7fe91c060310 2020-11-20T12:04:38.289 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.281+0000 7fe921d91700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe91c0a9260 con 0x7fe91c066ba0 2020-11-20T12:04:38.290 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.285+0000 7fe921590700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe91c0a0a10 0x7fe91c137360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:38.290 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.285+0000 7fe921590700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe91c0a0a10 0x7fe91c137360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51622/0 (socket says 172.21.15.47:51622) 2020-11-20T12:04:38.290 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.285+0000 7fe921590700 1 -- 172.21.15.47:0/2124755857 learned_addr learned my addr 172.21.15.47:0/2124755857 (peer_addr_for_me v2:172.21.15.47:0/0) 2020-11-20T12:04:38.290 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.285+0000 7fe91bfff700 1 --2- 172.21.15.47:0/2124755857 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe91c066ba0 0x7fe91c0a0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:38.290 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.285+0000 7fe920d8f700 1 --2- 172.21.15.47:0/2124755857 >> v2:172.21.15.189:3300/0 conn(0x7fe91c060310 0x7fe91c13a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:38.291 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.285+0000 7fe921590700 1 -- 172.21.15.47:0/2124755857 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe91c066ba0 msgr2=0x7fe91c0a0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:38.291 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.285+0000 7fe921590700 1 --2- 172.21.15.47:0/2124755857 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe91c066ba0 0x7fe91c0a0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.291 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.285+0000 7fe921590700 1 -- 172.21.15.47:0/2124755857 >> v2:172.21.15.189:3300/0 conn(0x7fe91c060310 msgr2=0x7fe91c13a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:38.291 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.285+0000 7fe921590700 1 --2- 172.21.15.47:0/2124755857 >> v2:172.21.15.189:3300/0 conn(0x7fe91c060310 0x7fe91c13a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.291 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.285+0000 7fe921590700 1 -- 172.21.15.47:0/2124755857 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91c068460 con 0x7fe91c0a0a10 2020-11-20T12:04:38.292 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.285+0000 7fe91bfff700 1 --2- 172.21.15.47:0/2124755857 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe91c066ba0 0x7fe91c0a0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:04:38.292 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.285+0000 7fe921590700 1 --2- 172.21.15.47:0/2124755857 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe91c0a0a10 0x7fe91c137360 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fe91c0ae2f0 tx=0x7fe914009290).ready entity=mon.0 client_cookie=a700c6e32c025f1c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:38.295 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.285+0000 7fe91b7fe700 1 -- 172.21.15.47:0/2124755857 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91400bba0 con 0x7fe91c0a0a10 2020-11-20T12:04:38.295 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.285+0000 7fe91b7fe700 1 -- 172.21.15.47:0/2124755857 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe91400bd00 con 0x7fe91c0a0a10 2020-11-20T12:04:38.295 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.285+0000 7fe91b7fe700 1 -- 172.21.15.47:0/2124755857 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe914011610 con 0x7fe91c0a0a10 2020-11-20T12:04:38.295 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.285+0000 7fe921d91700 1 -- 172.21.15.47:0/2124755857 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe91c0a0a10 msgr2=0x7fe91c137360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:38.296 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.285+0000 7fe921d91700 1 --2- 172.21.15.47:0/2124755857 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe91c0a0a10 0x7fe91c137360 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fe91c0ae2f0 tx=0x7fe914009290).stop 2020-11-20T12:04:38.296 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.285+0000 7fe921d91700 1 -- 172.21.15.47:0/2124755857 shutdown_connections 2020-11-20T12:04:38.296 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.285+0000 7fe921d91700 1 --2- 172.21.15.47:0/2124755857 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe91c0a0a10 0x7fe91c137360 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.296 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.285+0000 7fe921d91700 1 --2- 172.21.15.47:0/2124755857 >> v2:172.21.15.189:3300/0 conn(0x7fe91c060310 0x7fe91c13a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.296 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.285+0000 7fe921d91700 1 --2- 172.21.15.47:0/2124755857 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe91c066ba0 0x7fe91c0a0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.297 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.285+0000 7fe921d91700 1 -- 172.21.15.47:0/2124755857 >> 172.21.15.47:0/2124755857 conn(0x7fe91c065910 msgr2=0x7fe91c060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:38.297 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.285+0000 7fe921d91700 1 -- 172.21.15.47:0/2124755857 shutdown_connections 2020-11-20T12:04:38.297 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.285+0000 7fe921d91700 1 -- 172.21.15.47:0/2124755857 wait complete. 2020-11-20T12:04:38.297 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.285+0000 7fe921d91700 1 Processor -- start 2020-11-20T12:04:38.297 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.285+0000 7fe921d91700 1 -- start start 2020-11-20T12:04:38.298 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.289+0000 7fe921d91700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe91c060310 0x7fe91c145400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:38.298 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.289+0000 7fe921d91700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe91c066ba0 0x7fe91c0a3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:38.298 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.289+0000 7fe921d91700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe91c0a0a10 0x7fe91c13be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:38.298 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.289+0000 7fe921d91700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91c0a33a0 con 0x7fe91c0a0a10 2020-11-20T12:04:38.298 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.289+0000 7fe921d91700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91c0a35d0 con 0x7fe91c060310 2020-11-20T12:04:38.299 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.289+0000 7fe921d91700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe91c0a3870 con 0x7fe91c066ba0 2020-11-20T12:04:38.299 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.289+0000 7fe91bfff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe91c066ba0 0x7fe91c0a3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:38.299 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.289+0000 7fe91bfff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe91c066ba0 0x7fe91c0a3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:51084/0 (socket says 172.21.15.47:51084) 2020-11-20T12:04:38.299 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.289+0000 7fe91bfff700 1 -- 172.21.15.47:0/3544367549 learned_addr learned my addr 172.21.15.47:0/3544367549 (peer_addr_for_me v2:172.21.15.47:0/0) 2020-11-20T12:04:38.300 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.289+0000 7fe921590700 1 --2- 172.21.15.47:0/3544367549 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe91c0a0a10 0x7fe91c13be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:38.300 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.289+0000 7fe920d8f700 1 --2- 172.21.15.47:0/3544367549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe91c060310 0x7fe91c145400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:38.300 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.289+0000 7fe91bfff700 1 -- 172.21.15.47:0/3544367549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe91c060310 msgr2=0x7fe91c145400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:38.300 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.289+0000 7fe91bfff700 1 --2- 172.21.15.47:0/3544367549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe91c060310 0x7fe91c145400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.301 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.289+0000 7fe91bfff700 1 -- 172.21.15.47:0/3544367549 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe91c0a0a10 msgr2=0x7fe91c13be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:38.301 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.289+0000 7fe91bfff700 1 --2- 172.21.15.47:0/3544367549 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe91c0a0a10 0x7fe91c13be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.301 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.289+0000 7fe91bfff700 1 -- 172.21.15.47:0/3544367549 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91c139d10 con 0x7fe91c066ba0 2020-11-20T12:04:38.301 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.289+0000 7fe91bfff700 1 --2- 172.21.15.47:0/3544367549 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe91c066ba0 0x7fe91c0a3d10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fe90800ef50 tx=0x7fe908006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:38.302 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.289+0000 7fe918ff9700 1 -- 172.21.15.47:0/3544367549 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90800cb30 con 0x7fe91c066ba0 2020-11-20T12:04:38.302 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.289+0000 7fe921d91700 1 -- 172.21.15.47:0/3544367549 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe91c13b280 con 0x7fe91c066ba0 2020-11-20T12:04:38.302 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.289+0000 7fe918ff9700 1 -- 172.21.15.47:0/3544367549 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe90800cc90 con 0x7fe91c066ba0 2020-11-20T12:04:38.302 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.289+0000 7fe918ff9700 1 -- 172.21.15.47:0/3544367549 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe908012600 con 0x7fe91c066ba0 2020-11-20T12:04:38.302 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.289+0000 7fe921d91700 1 -- 172.21.15.47:0/3544367549 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe91c13b7f0 con 0x7fe91c066ba0 2020-11-20T12:04:38.303 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.289+0000 7fe918ff9700 1 -- 172.21.15.47:0/3544367549 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fe908012bd0 con 0x7fe91c066ba0 2020-11-20T12:04:38.303 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.289+0000 7fe918ff9700 1 -- 172.21.15.47:0/3544367549 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 754+0+0 (secure 0 0 0) 0x7fe90801cb60 con 0x7fe91c066ba0 2020-11-20T12:04:38.303 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.289+0000 7fe921d91700 1 -- 172.21.15.47:0/3544367549 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe91c0a3a30 con 0x7fe91c066ba0 2020-11-20T12:04:38.304 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.293+0000 7fe918ff9700 1 -- 172.21.15.47:0/3544367549 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7fe90800ce00 con 0x7fe91c066ba0 2020-11-20T12:04:38.379 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.373+0000 7fe921d91700 1 -- 172.21.15.47:0/3544367549 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c81214e1-4f84-4aaa-8ceb-99c398b2a89a", "id": 0} v 0) v1 -- 0x7fe91c13ba10 con 0x7fe91c066ba0 2020-11-20T12:04:38.385 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.381+0000 7fe918ff9700 1 -- 172.21.15.47:0/3544367549 <== mon.2 v2:172.21.15.47:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c81214e1-4f84-4aaa-8ceb-99c398b2a89a", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe90801cda0 con 0x7fe91c066ba0 2020-11-20T12:04:38.385 INFO:teuthology.orchestra.run.smithi047.stdout:0 2020-11-20T12:04:38.386 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.381+0000 7fe921d91700 1 -- 172.21.15.47:0/3544367549 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe91c066ba0 msgr2=0x7fe91c0a3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:38.387 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.381+0000 7fe921d91700 1 --2- 172.21.15.47:0/3544367549 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe91c066ba0 0x7fe91c0a3d10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fe90800ef50 tx=0x7fe908006df0).stop 2020-11-20T12:04:38.387 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.381+0000 7fe921d91700 1 -- 172.21.15.47:0/3544367549 shutdown_connections 2020-11-20T12:04:38.387 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.381+0000 7fe921d91700 1 --2- 172.21.15.47:0/3544367549 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe91c0a0a10 0x7fe91c13be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.387 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.381+0000 7fe921d91700 1 --2- 172.21.15.47:0/3544367549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe91c060310 0x7fe91c145400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.387 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.381+0000 7fe921d91700 1 --2- 172.21.15.47:0/3544367549 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe91c066ba0 0x7fe91c0a3d10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.388 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.381+0000 7fe921d91700 1 -- 172.21.15.47:0/3544367549 >> 172.21.15.47:0/3544367549 conn(0x7fe91c065910 msgr2=0x7fe91c0a7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:38.388 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.381+0000 7fe921d91700 1 -- 172.21.15.47:0/3544367549 shutdown_connections 2020-11-20T12:04:38.388 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.381+0000 7fe921d91700 1 -- 172.21.15.47:0/3544367549 wait complete. 2020-11-20T12:04:38.398 INFO:teuthology.orchestra.run.smithi047:> sudo ceph --cluster ceph osd new d2f3ec7d-d403-484f-b5ed-b2e6b079d63d 1 2020-11-20T12:04:38.515 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.509+0000 7fb054126700 1 Processor -- start 2020-11-20T12:04:38.515 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.509+0000 7fb054126700 1 -- start start 2020-11-20T12:04:38.518 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.509+0000 7fb054126700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb04c0ae4f0 0x7fb04c1377d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:38.518 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb054126700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb04c137cc0 0x7fb04c063760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:38.518 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb054126700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb04c0a8fb0 con 0x7fb04c063c50 2020-11-20T12:04:38.518 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb054126700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb04c138140 con 0x7fb04c137cc0 2020-11-20T12:04:38.518 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb054126700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c067dd0 con 0x7fb04c0ae4f0 2020-11-20T12:04:38.519 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb051ec2700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb04c0ae4f0 0x7fb04c1377d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:38.519 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb051ec2700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb04c0ae4f0 0x7fb04c1377d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:51088/0 (socket says 172.21.15.47:51088) 2020-11-20T12:04:38.519 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb051ec2700 1 -- 172.21.15.47:0/2823539146 learned_addr learned my addr 172.21.15.47:0/2823539146 (peer_addr_for_me v2:172.21.15.47:0/0) 2020-11-20T12:04:38.519 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb0516c1700 1 --2- 172.21.15.47:0/2823539146 >> v2:172.21.15.189:3300/0 conn(0x7fb04c137cc0 0x7fb04c063760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:38.520 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb051ec2700 1 -- 172.21.15.47:0/2823539146 >> v2:172.21.15.189:3300/0 conn(0x7fb04c137cc0 msgr2=0x7fb04c063760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:38.520 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb051ec2700 1 --2- 172.21.15.47:0/2823539146 >> v2:172.21.15.189:3300/0 conn(0x7fb04c137cc0 0x7fb04c063760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.520 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb051ec2700 1 -- 172.21.15.47:0/2823539146 >> v1:172.21.15.47:6789/0 conn(0x7fb04c063c50 legacy=0x7fb04c060a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:38.520 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb051ec2700 1 -- 172.21.15.47:0/2823539146 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04c064100 con 0x7fb04c0ae4f0 2020-11-20T12:04:38.520 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb050ec0700 1 -- 172.21.15.47:0/2823539146 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2161436442 0 0) 0x7fb04c0a8fb0 con 0x7fb04c063c50 2020-11-20T12:04:38.521 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb051ec2700 1 --2- 172.21.15.47:0/2823539146 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb04c0ae4f0 0x7fb04c1377d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fb044009d50 tx=0x7fb044009290).ready entity=mon.2 client_cookie=870a06ee2a7420db server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:38.521 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb050ec0700 1 -- 172.21.15.47:0/2823539146 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04400bc80 con 0x7fb04c0ae4f0 2020-11-20T12:04:38.521 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb050ec0700 1 -- 172.21.15.47:0/2823539146 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb04400bde0 con 0x7fb04c0ae4f0 2020-11-20T12:04:38.521 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb050ec0700 1 -- 172.21.15.47:0/2823539146 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb044011610 con 0x7fb04c0ae4f0 2020-11-20T12:04:38.522 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb054126700 1 -- 172.21.15.47:0/2823539146 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb04c0ae4f0 msgr2=0x7fb04c1377d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:38.522 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb054126700 1 --2- 172.21.15.47:0/2823539146 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb04c0ae4f0 0x7fb04c1377d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fb044009d50 tx=0x7fb044009290).stop 2020-11-20T12:04:38.522 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb054126700 1 -- 172.21.15.47:0/2823539146 shutdown_connections 2020-11-20T12:04:38.522 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb054126700 1 --2- 172.21.15.47:0/2823539146 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb04c0ae4f0 0x7fb04c1377d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.522 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb054126700 1 --2- 172.21.15.47:0/2823539146 >> v2:172.21.15.189:3300/0 conn(0x7fb04c137cc0 0x7fb04c063760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.523 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb054126700 1 -- 172.21.15.47:0/2823539146 >> 172.21.15.47:0/2823539146 conn(0x7fb04c062580 msgr2=0x7fb04c0adc40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:38.523 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb054126700 1 -- 172.21.15.47:0/2823539146 shutdown_connections 2020-11-20T12:04:38.523 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb054126700 1 -- 172.21.15.47:0/2823539146 wait complete. 2020-11-20T12:04:38.525 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb054126700 1 Processor -- start 2020-11-20T12:04:38.526 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb054126700 1 -- start start 2020-11-20T12:04:38.526 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb054126700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb04c063c50 0x7fb04c0a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:38.526 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb054126700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb04c0ae4f0 0x7fb04c0ad050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:38.526 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb054126700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb04c137cc0 0x7fb04c0a7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:38.527 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb054126700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c0a6370 con 0x7fb04c063c50 2020-11-20T12:04:38.527 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb054126700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c0a11c0 con 0x7fb04c137cc0 2020-11-20T12:04:38.527 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb054126700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c14cef0 con 0x7fb04c0ae4f0 2020-11-20T12:04:38.527 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb051ec2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb04c063c50 0x7fb04c0a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:38.527 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb0516c1700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb04c0ae4f0 0x7fb04c0ad050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:38.528 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb0516c1700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb04c0ae4f0 0x7fb04c0ad050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:51096/0 (socket says 172.21.15.47:51096) 2020-11-20T12:04:38.528 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb051ec2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb04c063c50 0x7fb04c0a08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51636/0 (socket says 172.21.15.47:51636) 2020-11-20T12:04:38.528 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb0516c1700 1 -- 172.21.15.47:0/2320987889 learned_addr learned my addr 172.21.15.47:0/2320987889 (peer_addr_for_me v2:172.21.15.47:0/0) 2020-11-20T12:04:38.528 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.513+0000 7fb0526c3700 1 --2- 172.21.15.47:0/2320987889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb04c137cc0 0x7fb04c0a7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:38.528 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.517+0000 7fb0516c1700 1 -- 172.21.15.47:0/2320987889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb04c137cc0 msgr2=0x7fb04c0a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:38.529 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.517+0000 7fb0516c1700 1 --2- 172.21.15.47:0/2320987889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb04c137cc0 0x7fb04c0a7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.529 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.517+0000 7fb0516c1700 1 -- 172.21.15.47:0/2320987889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb04c063c50 msgr2=0x7fb04c0a08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:38.529 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.517+0000 7fb0516c1700 1 --2- 172.21.15.47:0/2320987889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb04c063c50 0x7fb04c0a08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.529 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.517+0000 7fb0516c1700 1 -- 172.21.15.47:0/2320987889 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04c0a0f20 con 0x7fb04c0ae4f0 2020-11-20T12:04:38.529 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.517+0000 7fb051ec2700 1 --2- 172.21.15.47:0/2320987889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb04c063c50 0x7fb04c0a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:04:38.530 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.517+0000 7fb0516c1700 1 --2- 172.21.15.47:0/2320987889 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb04c0ae4f0 0x7fb04c0ad050 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fb04000d930 tx=0x7fb040008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:38.530 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.517+0000 7fb049ffb700 1 -- 172.21.15.47:0/2320987889 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0400096d0 con 0x7fb04c0ae4f0 2020-11-20T12:04:38.530 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.517+0000 7fb049ffb700 1 -- 172.21.15.47:0/2320987889 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb040009830 con 0x7fb04c0ae4f0 2020-11-20T12:04:38.530 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.517+0000 7fb049ffb700 1 -- 172.21.15.47:0/2320987889 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb040009bd0 con 0x7fb04c0ae4f0 2020-11-20T12:04:38.531 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.517+0000 7fb054126700 1 -- 172.21.15.47:0/2320987889 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb04c0a6e80 con 0x7fb04c0ae4f0 2020-11-20T12:04:38.531 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.517+0000 7fb054126700 1 -- 172.21.15.47:0/2320987889 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb04c0a79b0 con 0x7fb04c0ae4f0 2020-11-20T12:04:38.531 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.517+0000 7fb049ffb700 1 -- 172.21.15.47:0/2320987889 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fb04001c6c0 con 0x7fb04c0ae4f0 2020-11-20T12:04:38.531 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.517+0000 7fb049ffb700 1 -- 172.21.15.47:0/2320987889 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 869+0+0 (secure 0 0 0) 0x7fb040012800 con 0x7fb04c0ae4f0 2020-11-20T12:04:38.531 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.517+0000 7fb054126700 1 -- 172.21.15.47:0/2320987889 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb04c044f60 con 0x7fb04c0ae4f0 2020-11-20T12:04:38.532 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.521+0000 7fb049ffb700 1 -- 172.21.15.47:0/2320987889 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7fb04001cb30 con 0x7fb04c0ae4f0 2020-11-20T12:04:38.609 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.605+0000 7fb054126700 1 -- 172.21.15.47:0/2320987889 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d2f3ec7d-d403-484f-b5ed-b2e6b079d63d", "id": 1} v 0) v1 -- 0x7fb04c0aa450 con 0x7fb04c0ae4f0 2020-11-20T12:04:38.615 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.609+0000 7fb049ffb700 1 -- 172.21.15.47:0/2320987889 <== mon.2 v2:172.21.15.47:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d2f3ec7d-d403-484f-b5ed-b2e6b079d63d", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb04001ccd0 con 0x7fb04c0ae4f0 2020-11-20T12:04:38.616 INFO:teuthology.orchestra.run.smithi047.stdout:1 2020-11-20T12:04:38.617 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.613+0000 7fb054126700 1 -- 172.21.15.47:0/2320987889 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb04c0ae4f0 msgr2=0x7fb04c0ad050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:38.617 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.613+0000 7fb054126700 1 --2- 172.21.15.47:0/2320987889 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb04c0ae4f0 0x7fb04c0ad050 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fb04000d930 tx=0x7fb040008040).stop 2020-11-20T12:04:38.617 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.613+0000 7fb054126700 1 -- 172.21.15.47:0/2320987889 shutdown_connections 2020-11-20T12:04:38.618 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.613+0000 7fb054126700 1 --2- 172.21.15.47:0/2320987889 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb04c063c50 0x7fb04c0a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.618 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.613+0000 7fb054126700 1 --2- 172.21.15.47:0/2320987889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb04c137cc0 0x7fb04c0a7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.618 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.613+0000 7fb054126700 1 --2- 172.21.15.47:0/2320987889 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb04c0ae4f0 0x7fb04c0ad050 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.618 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.613+0000 7fb054126700 1 -- 172.21.15.47:0/2320987889 >> 172.21.15.47:0/2320987889 conn(0x7fb04c062580 msgr2=0x7fb04c066d80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:38.619 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.613+0000 7fb054126700 1 -- 172.21.15.47:0/2320987889 shutdown_connections 2020-11-20T12:04:38.619 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.613+0000 7fb054126700 1 -- 172.21.15.47:0/2320987889 wait complete. 2020-11-20T12:04:38.628 INFO:teuthology.orchestra.run.smithi047:> sudo ceph --cluster ceph osd new e6cf4790-000c-43eb-8844-175f04c60503 2 2020-11-20T12:04:38.744 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.741+0000 7f846c785700 1 Processor -- start 2020-11-20T12:04:38.744 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.741+0000 7f846c785700 1 -- start start 2020-11-20T12:04:38.746 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.741+0000 7f846c785700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f84640ae730 0x7f84640640d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:38.746 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.741+0000 7f846c785700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8464134e50 con 0x7f84640a70a0 2020-11-20T12:04:38.746 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.741+0000 7f846c785700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84640aa060 con 0x7f84640ae350 2020-11-20T12:04:38.746 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.741+0000 7f846c785700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f846413b080 con 0x7f84640ae730 2020-11-20T12:04:38.747 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.741+0000 7f8469d20700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f84640ae730 0x7f84640640d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:38.747 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.741+0000 7f846ad22700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f84640a70a0 0x7f8464141d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43178/0 (socket says 172.21.15.47:43178) 2020-11-20T12:04:38.747 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.741+0000 7f846ad22700 1 -- 172.21.15.47:0/1605029596 learned_addr learned my addr 172.21.15.47:0/1605029596 (peer_addr_for_me v1:172.21.15.47:0/0) 2020-11-20T12:04:38.747 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.741+0000 7f8469d20700 1 -- 172.21.15.47:0/1605029596 >> v1:172.21.15.189:6789/0 conn(0x7f84640ae350 legacy=0x7f846413a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:38.748 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.741+0000 7f8469d20700 1 -- 172.21.15.47:0/1605029596 >> v1:172.21.15.47:6789/0 conn(0x7f84640a70a0 legacy=0x7f8464141d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:38.748 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.741+0000 7f8469d20700 1 -- 172.21.15.47:0/1605029596 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84640a85e0 con 0x7f84640ae730 2020-11-20T12:04:38.748 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.741+0000 7f8469d20700 1 --2- 172.21.15.47:0/1605029596 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84640ae730 0x7f84640640d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f846413b1c0 tx=0x7f8454006df0).ready entity=mon.0 client_cookie=7432436fcf0b8dcf server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:38.748 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.741+0000 7f846951f700 1 -- 172.21.15.47:0/1605029596 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8454009800 con 0x7f84640ae730 2020-11-20T12:04:38.748 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.741+0000 7f846951f700 1 -- 172.21.15.47:0/1605029596 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8454009960 con 0x7f84640ae730 2020-11-20T12:04:38.749 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.741+0000 7f846951f700 1 -- 172.21.15.47:0/1605029596 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8454009ce0 con 0x7f84640ae730 2020-11-20T12:04:38.752 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.741+0000 7f846c785700 1 -- 172.21.15.47:0/1605029596 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84640ae730 msgr2=0x7f84640640d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:38.752 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.741+0000 7f846c785700 1 --2- 172.21.15.47:0/1605029596 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84640ae730 0x7f84640640d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f846413b1c0 tx=0x7f8454006df0).stop 2020-11-20T12:04:38.752 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.741+0000 7f846c785700 1 -- 172.21.15.47:0/1605029596 shutdown_connections 2020-11-20T12:04:38.752 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.741+0000 7f846c785700 1 --2- 172.21.15.47:0/1605029596 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84640ae730 0x7f84640640d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.753 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.741+0000 7f846c785700 1 -- 172.21.15.47:0/1605029596 >> 172.21.15.47:0/1605029596 conn(0x7f8464062580 msgr2=0x7f84640adc40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:38.753 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.741+0000 7f846c785700 1 -- 172.21.15.47:0/1605029596 shutdown_connections 2020-11-20T12:04:38.753 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.741+0000 7f846c785700 1 -- 172.21.15.47:0/1605029596 wait complete. 2020-11-20T12:04:38.753 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.745+0000 7f846c785700 1 Processor -- start 2020-11-20T12:04:38.753 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.745+0000 7f846c785700 1 -- start start 2020-11-20T12:04:38.754 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.745+0000 7f846c785700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84640a70a0 0x7f846409d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:38.754 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.745+0000 7f846c785700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f84640ae350 0x7f846409fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:38.754 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.745+0000 7f846c785700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84640ae730 0x7f846409cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:38.754 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.745+0000 7f846c785700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846409df50 con 0x7f84640ae730 2020-11-20T12:04:38.755 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.745+0000 7f846c785700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846409e1f0 con 0x7f84640a70a0 2020-11-20T12:04:38.755 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.745+0000 7f846c785700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8464137a90 con 0x7f84640ae350 2020-11-20T12:04:38.755 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.745+0000 7f8469d20700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f84640ae350 0x7f846409fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:38.755 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.745+0000 7f8469d20700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f84640ae350 0x7f846409fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:51108/0 (socket says 172.21.15.47:51108) 2020-11-20T12:04:38.755 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.745+0000 7f8469d20700 1 -- 172.21.15.47:0/419349503 learned_addr learned my addr 172.21.15.47:0/419349503 (peer_addr_for_me v2:172.21.15.47:0/0) 2020-11-20T12:04:38.756 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.745+0000 7f846ad22700 1 --2- 172.21.15.47:0/419349503 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84640ae730 0x7f846409cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:38.756 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.745+0000 7f846a521700 1 --2- 172.21.15.47:0/419349503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84640a70a0 0x7f846409d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:38.756 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.745+0000 7f8469d20700 1 -- 172.21.15.47:0/419349503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84640a70a0 msgr2=0x7f846409d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:38.756 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.745+0000 7f8469d20700 1 --2- 172.21.15.47:0/419349503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84640a70a0 0x7f846409d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.756 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.745+0000 7f8469d20700 1 -- 172.21.15.47:0/419349503 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84640ae730 msgr2=0x7f846409cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:38.757 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.745+0000 7f8469d20700 1 --2- 172.21.15.47:0/419349503 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84640ae730 0x7f846409cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.757 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.745+0000 7f8469d20700 1 -- 172.21.15.47:0/419349503 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8464137c50 con 0x7f84640ae350 2020-11-20T12:04:38.757 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.745+0000 7f846ad22700 1 --2- 172.21.15.47:0/419349503 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84640ae730 0x7f846409cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:04:38.757 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.745+0000 7f8469d20700 1 --2- 172.21.15.47:0/419349503 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f84640ae350 0x7f846409fa50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f8454010000 tx=0x7f8454015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:38.758 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.745+0000 7f84527fc700 1 -- 172.21.15.47:0/419349503 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845401e460 con 0x7f84640ae350 2020-11-20T12:04:38.758 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.745+0000 7f84527fc700 1 -- 172.21.15.47:0/419349503 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f845401e5c0 con 0x7f84640ae350 2020-11-20T12:04:38.758 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.745+0000 7f846c785700 1 -- 172.21.15.47:0/419349503 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f846409d000 con 0x7f84640ae350 2020-11-20T12:04:38.758 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.745+0000 7f84527fc700 1 -- 172.21.15.47:0/419349503 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845401e960 con 0x7f84640ae350 2020-11-20T12:04:38.758 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.745+0000 7f846c785700 1 -- 172.21.15.47:0/419349503 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f846409d470 con 0x7f84640ae350 2020-11-20T12:04:38.759 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.749+0000 7f84527fc700 1 -- 172.21.15.47:0/419349503 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f8454025030 con 0x7f84640ae350 2020-11-20T12:04:38.759 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.749+0000 7f84527fc700 1 -- 172.21.15.47:0/419349503 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 984+0+0 (secure 0 0 0) 0x7f845401d810 con 0x7f84640ae350 2020-11-20T12:04:38.759 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.749+0000 7f846c785700 1 -- 172.21.15.47:0/419349503 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8464044f60 con 0x7f84640ae350 2020-11-20T12:04:38.759 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.749+0000 7f84527fc700 1 -- 172.21.15.47:0/419349503 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7f8454023070 con 0x7f84640ae350 2020-11-20T12:04:38.839 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.833+0000 7f846c785700 1 -- 172.21.15.47:0/419349503 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e6cf4790-000c-43eb-8844-175f04c60503", "id": 2} v 0) v1 -- 0x7f84640a4ac0 con 0x7f84640ae350 2020-11-20T12:04:38.844 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.841+0000 7f84527fc700 1 -- 172.21.15.47:0/419349503 <== mon.2 v2:172.21.15.47:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e6cf4790-000c-43eb-8844-175f04c60503", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f845401d3c0 con 0x7f84640ae350 2020-11-20T12:04:38.845 INFO:teuthology.orchestra.run.smithi047.stdout:2 2020-11-20T12:04:38.846 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.841+0000 7f846c785700 1 -- 172.21.15.47:0/419349503 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f84640ae350 msgr2=0x7f846409fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:38.846 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.841+0000 7f846c785700 1 --2- 172.21.15.47:0/419349503 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f84640ae350 0x7f846409fa50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f8454010000 tx=0x7f8454015000).stop 2020-11-20T12:04:38.846 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.841+0000 7f846c785700 1 -- 172.21.15.47:0/419349503 shutdown_connections 2020-11-20T12:04:38.847 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.841+0000 7f846c785700 1 --2- 172.21.15.47:0/419349503 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84640ae730 0x7f846409cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.847 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.841+0000 7f846c785700 1 --2- 172.21.15.47:0/419349503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84640a70a0 0x7f846409d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.847 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.841+0000 7f846c785700 1 --2- 172.21.15.47:0/419349503 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f84640ae350 0x7f846409fa50 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.848 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.841+0000 7f846c785700 1 -- 172.21.15.47:0/419349503 >> 172.21.15.47:0/419349503 conn(0x7f8464062580 msgr2=0x7f84640a5520 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:38.848 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.841+0000 7f846c785700 1 -- 172.21.15.47:0/419349503 shutdown_connections 2020-11-20T12:04:38.848 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.841+0000 7f846c785700 1 -- 172.21.15.47:0/419349503 wait complete. 2020-11-20T12:04:38.857 INFO:teuthology.orchestra.run.smithi047:> sudo ceph --cluster ceph osd new ba49dfac-f2b9-476e-853d-42f05e21a77a 3 2020-11-20T12:04:38.973 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac2f73700 1 Processor -- start 2020-11-20T12:04:38.973 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac2f73700 1 -- start start 2020-11-20T12:04:38.975 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac2f73700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7faabc0a6f30 0x7faabc138180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:38.975 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac2f73700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faabc0a7310 0x7faabc064450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:38.975 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac2f73700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7faabc0669b0 0x7faabc138960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:38.975 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac2f73700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faabc135c30 con 0x7faabc0a6f30 2020-11-20T12:04:38.976 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac2f73700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faabc063d10 con 0x7faabc0a7310 2020-11-20T12:04:38.976 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac2f73700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faabc063fb0 con 0x7faabc0669b0 2020-11-20T12:04:38.976 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac1f71700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7faabc0a6f30 0x7faabc138180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:38.976 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac2772700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7faabc0669b0 0x7faabc138960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:38.976 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac1f71700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7faabc0a6f30 0x7faabc138180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51654/0 (socket says 172.21.15.47:51654) 2020-11-20T12:04:38.977 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac2772700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7faabc0669b0 0x7faabc138960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:51116/0 (socket says 172.21.15.47:51116) 2020-11-20T12:04:38.977 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac1f71700 1 -- 172.21.15.47:0/3711363930 learned_addr learned my addr 172.21.15.47:0/3711363930 (peer_addr_for_me v2:172.21.15.47:0/0) 2020-11-20T12:04:38.977 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac1770700 1 --2- 172.21.15.47:0/3711363930 >> v2:172.21.15.189:3300/0 conn(0x7faabc0a7310 0x7faabc064450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:38.977 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac2772700 1 -- 172.21.15.47:0/3711363930 >> v2:172.21.15.189:3300/0 conn(0x7faabc0a7310 msgr2=0x7faabc064450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:38.977 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac2772700 1 --2- 172.21.15.47:0/3711363930 >> v2:172.21.15.189:3300/0 conn(0x7faabc0a7310 0x7faabc064450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.978 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac2772700 1 -- 172.21.15.47:0/3711363930 >> v2:172.21.15.47:3300/0 conn(0x7faabc0a6f30 msgr2=0x7faabc138180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:38.978 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac2772700 1 --2- 172.21.15.47:0/3711363930 >> v2:172.21.15.47:3300/0 conn(0x7faabc0a6f30 0x7faabc138180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.978 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac2772700 1 -- 172.21.15.47:0/3711363930 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faabc068550 con 0x7faabc0669b0 2020-11-20T12:04:38.978 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac1f71700 1 --2- 172.21.15.47:0/3711363930 >> v2:172.21.15.47:3300/0 conn(0x7faabc0a6f30 0x7faabc138180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:04:38.978 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac2772700 1 --2- 172.21.15.47:0/3711363930 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7faabc0669b0 0x7faabc138960 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7faabc068050 tx=0x7faaac0091d0).ready entity=mon.2 client_cookie=597abd00ac3387ac server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:38.979 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac0f6f700 1 -- 172.21.15.47:0/3711363930 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac00bad0 con 0x7faabc0669b0 2020-11-20T12:04:38.979 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac0f6f700 1 -- 172.21.15.47:0/3711363930 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaac00bc30 con 0x7faabc0669b0 2020-11-20T12:04:38.979 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac0f6f700 1 -- 172.21.15.47:0/3711363930 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac011400 con 0x7faabc0669b0 2020-11-20T12:04:38.982 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac2f73700 1 -- 172.21.15.47:0/3711363930 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7faabc0669b0 msgr2=0x7faabc138960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:38.982 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac2f73700 1 --2- 172.21.15.47:0/3711363930 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7faabc0669b0 0x7faabc138960 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7faabc068050 tx=0x7faaac0091d0).stop 2020-11-20T12:04:38.982 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac2f73700 1 -- 172.21.15.47:0/3711363930 shutdown_connections 2020-11-20T12:04:38.982 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac2f73700 1 --2- 172.21.15.47:0/3711363930 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7faabc0669b0 0x7faabc138960 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.983 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac2f73700 1 --2- 172.21.15.47:0/3711363930 >> v2:172.21.15.47:3300/0 conn(0x7faabc0a6f30 0x7faabc138180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.983 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac2f73700 1 --2- 172.21.15.47:0/3711363930 >> v2:172.21.15.189:3300/0 conn(0x7faabc0a7310 0x7faabc064450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.983 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac2f73700 1 -- 172.21.15.47:0/3711363930 >> 172.21.15.47:0/3711363930 conn(0x7faabc0ad8e0 msgr2=0x7faabc0adcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:38.983 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac2f73700 1 -- 172.21.15.47:0/3711363930 shutdown_connections 2020-11-20T12:04:38.983 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.969+0000 7faac2f73700 1 -- 172.21.15.47:0/3711363930 wait complete. 2020-11-20T12:04:38.984 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.973+0000 7faac2f73700 1 Processor -- start 2020-11-20T12:04:38.984 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.973+0000 7faac2f73700 1 -- start start 2020-11-20T12:04:38.984 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.973+0000 7faac2f73700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faabc0669b0 0x7faabc13d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:38.984 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.973+0000 7faac2f73700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7faabc0a6f30 0x7faabc136870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:38.984 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.973+0000 7faac2f73700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faabc0a7310 0x7faabc136d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:38.985 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.973+0000 7faac2f73700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faabc13e4e0 con 0x7faabc0669b0 2020-11-20T12:04:38.985 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.973+0000 7faac2f73700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faabc137250 con 0x7faabc0a7310 2020-11-20T12:04:38.985 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.973+0000 7faac2f73700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faabc13dfa0 con 0x7faabc0a6f30 2020-11-20T12:04:38.985 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.973+0000 7faac1770700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7faabc0a6f30 0x7faabc136870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:38.985 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.973+0000 7faac1f71700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faabc0669b0 0x7faabc13d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:38.986 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.973+0000 7faac1770700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7faabc0a6f30 0x7faabc136870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:51120/0 (socket says 172.21.15.47:51120) 2020-11-20T12:04:38.986 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.973+0000 7faac1770700 1 -- 172.21.15.47:0/1903105247 learned_addr learned my addr 172.21.15.47:0/1903105247 (peer_addr_for_me v2:172.21.15.47:0/0) 2020-11-20T12:04:38.986 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.973+0000 7faac2772700 1 --2- 172.21.15.47:0/1903105247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faabc0a7310 0x7faabc136d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:38.986 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.973+0000 7faac1770700 1 -- 172.21.15.47:0/1903105247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faabc0a7310 msgr2=0x7faabc136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:38.986 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.973+0000 7faac1770700 1 --2- 172.21.15.47:0/1903105247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faabc0a7310 0x7faabc136d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.987 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.973+0000 7faac1770700 1 -- 172.21.15.47:0/1903105247 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faabc0669b0 msgr2=0x7faabc13d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:38.987 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.973+0000 7faac1770700 1 --2- 172.21.15.47:0/1903105247 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faabc0669b0 0x7faabc13d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:38.987 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.973+0000 7faac1770700 1 -- 172.21.15.47:0/1903105247 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faabc13e240 con 0x7faabc0a6f30 2020-11-20T12:04:38.987 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.973+0000 7faac1770700 1 --2- 172.21.15.47:0/1903105247 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7faabc0a6f30 0x7faabc136870 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7faab000d960 tx=0x7faab0008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:38.987 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.973+0000 7faab9ffb700 1 -- 172.21.15.47:0/1903105247 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab0009640 con 0x7faabc0a6f30 2020-11-20T12:04:38.988 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.973+0000 7faab9ffb700 1 -- 172.21.15.47:0/1903105247 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faab00097a0 con 0x7faabc0a6f30 2020-11-20T12:04:38.988 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.973+0000 7faac2f73700 1 -- 172.21.15.47:0/1903105247 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faabc135e40 con 0x7faabc0a6f30 2020-11-20T12:04:38.988 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.973+0000 7faab9ffb700 1 -- 172.21.15.47:0/1903105247 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab0009b40 con 0x7faabc0a6f30 2020-11-20T12:04:38.988 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.973+0000 7faac2f73700 1 -- 172.21.15.47:0/1903105247 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faabc138410 con 0x7faabc0a6f30 2020-11-20T12:04:38.988 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.973+0000 7faab9ffb700 1 -- 172.21.15.47:0/1903105247 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7faab0006280 con 0x7faabc0a6f30 2020-11-20T12:04:38.989 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.973+0000 7faab9ffb700 1 -- 172.21.15.47:0/1903105247 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1099+0+0 (secure 0 0 0) 0x7faab0005d70 con 0x7faabc0a6f30 2020-11-20T12:04:38.989 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.977+0000 7faac2f73700 1 -- 172.21.15.47:0/1903105247 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faabc066df0 con 0x7faabc0a6f30 2020-11-20T12:04:38.989 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:38.977+0000 7faab9ffb700 1 -- 172.21.15.47:0/1903105247 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7faab00058b0 con 0x7faabc0a6f30 2020-11-20T12:04:39.067 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.061+0000 7faac2f73700 1 -- 172.21.15.47:0/1903105247 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ba49dfac-f2b9-476e-853d-42f05e21a77a", "id": 3} v 0) v1 -- 0x7faabc0446d0 con 0x7faabc0a6f30 2020-11-20T12:04:39.073 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.069+0000 7faab9ffb700 1 -- 172.21.15.47:0/1903105247 <== mon.2 v2:172.21.15.47:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ba49dfac-f2b9-476e-853d-42f05e21a77a", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7faab002b0b0 con 0x7faabc0a6f30 2020-11-20T12:04:39.074 INFO:teuthology.orchestra.run.smithi047.stdout:3 2020-11-20T12:04:39.074 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.069+0000 7faac2f73700 1 -- 172.21.15.47:0/1903105247 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7faabc0a6f30 msgr2=0x7faabc136870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:39.075 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.069+0000 7faac2f73700 1 --2- 172.21.15.47:0/1903105247 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7faabc0a6f30 0x7faabc136870 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7faab000d960 tx=0x7faab0008040).stop 2020-11-20T12:04:39.075 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.069+0000 7faac2f73700 1 -- 172.21.15.47:0/1903105247 shutdown_connections 2020-11-20T12:04:39.075 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.069+0000 7faac2f73700 1 --2- 172.21.15.47:0/1903105247 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faabc0669b0 0x7faabc13d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.075 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.069+0000 7faac2f73700 1 --2- 172.21.15.47:0/1903105247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faabc0a7310 0x7faabc136d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.076 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.069+0000 7faac2f73700 1 --2- 172.21.15.47:0/1903105247 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7faabc0a6f30 0x7faabc136870 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.076 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.069+0000 7faac2f73700 1 -- 172.21.15.47:0/1903105247 >> 172.21.15.47:0/1903105247 conn(0x7faabc0ad8e0 msgr2=0x7faabc0acfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:39.076 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.069+0000 7faac2f73700 1 -- 172.21.15.47:0/1903105247 shutdown_connections 2020-11-20T12:04:39.076 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.069+0000 7faac2f73700 1 -- 172.21.15.47:0/1903105247 wait complete. 2020-11-20T12:04:39.088 INFO:teuthology.orchestra.run.smithi047:> sudo ceph --cluster ceph osd new 3ca7b438-38b2-4689-8b18-fb5a8ddf6aee 4 2020-11-20T12:04:39.205 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39dffff700 1 Processor -- start 2020-11-20T12:04:39.205 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39dffff700 1 -- start start 2020-11-20T12:04:39.208 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39dffff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f39e00a08f0 0x7f39e0134120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:39.208 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39dffff700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f39e00a0cd0 0x7f39e0134610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:39.209 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39dffff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39e00a9d00 con 0x7f39e00aa160 2020-11-20T12:04:39.209 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39dffff700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39e0068580 con 0x7f39e00a0cd0 2020-11-20T12:04:39.209 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39dffff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e013b5c0 con 0x7f39e00a08f0 2020-11-20T12:04:39.209 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39d7fff700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f39e00a0cd0 0x7f39e0134610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:39.209 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39deffd700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f39e00a08f0 0x7f39e0134120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:39.210 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39d7fff700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f39e00a0cd0 0x7f39e0134610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51668/0 (socket says 172.21.15.47:51668) 2020-11-20T12:04:39.210 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39d7fff700 1 -- 172.21.15.47:0/1902390878 learned_addr learned my addr 172.21.15.47:0/1902390878 (peer_addr_for_me v2:172.21.15.47:0/0) 2020-11-20T12:04:39.210 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39d7fff700 1 -- 172.21.15.47:0/1902390878 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f39e00a08f0 msgr2=0x7f39e0134120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:39.210 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39d7fff700 1 --2- 172.21.15.47:0/1902390878 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f39e00a08f0 0x7f39e0134120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.210 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39d7fff700 1 -- 172.21.15.47:0/1902390878 >> v1:172.21.15.189:6789/0 conn(0x7f39e00aa160 legacy=0x7f39e013a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:39.211 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39d7fff700 1 -- 172.21.15.47:0/1902390878 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e013b320 con 0x7f39e00a0cd0 2020-11-20T12:04:39.211 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39d7fff700 1 --2- 172.21.15.47:0/1902390878 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39e00a0cd0 0x7f39e0134610 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f39d0006fd0 tx=0x7f39d0006eb0).ready entity=mon.0 client_cookie=297e608429ec2004 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:39.211 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39de7fc700 1 -- 172.21.15.47:0/1902390878 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d0009810 con 0x7f39e00a0cd0 2020-11-20T12:04:39.211 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39de7fc700 1 -- 172.21.15.47:0/1902390878 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d0009970 con 0x7f39e00a0cd0 2020-11-20T12:04:39.212 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39de7fc700 1 -- 172.21.15.47:0/1902390878 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d0009cf0 con 0x7f39e00a0cd0 2020-11-20T12:04:39.212 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39dffff700 1 -- 172.21.15.47:0/1902390878 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39e00a0cd0 msgr2=0x7f39e0134610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:39.212 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39dffff700 1 --2- 172.21.15.47:0/1902390878 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39e00a0cd0 0x7f39e0134610 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f39d0006fd0 tx=0x7f39d0006eb0).stop 2020-11-20T12:04:39.212 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39dffff700 1 -- 172.21.15.47:0/1902390878 shutdown_connections 2020-11-20T12:04:39.212 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39dffff700 1 --2- 172.21.15.47:0/1902390878 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f39e00a08f0 0x7f39e0134120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.213 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39dffff700 1 --2- 172.21.15.47:0/1902390878 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39e00a0cd0 0x7f39e0134610 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.213 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39dffff700 1 -- 172.21.15.47:0/1902390878 >> 172.21.15.47:0/1902390878 conn(0x7f39e0065910 msgr2=0x7f39e0060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:39.213 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39dffff700 1 -- 172.21.15.47:0/1902390878 shutdown_connections 2020-11-20T12:04:39.213 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.201+0000 7f39dffff700 1 -- 172.21.15.47:0/1902390878 wait complete. 2020-11-20T12:04:39.215 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39dffff700 1 Processor -- start 2020-11-20T12:04:39.216 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39dffff700 1 -- start start 2020-11-20T12:04:39.216 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39dffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39e00a08f0 0x7f39e0135030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:39.216 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39dffff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f39e00a0cd0 0x7f39e00a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:39.216 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39dffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39e00aa160 0x7f39e0149940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:39.217 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39dffff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e00a4ef0 con 0x7f39e00a08f0 2020-11-20T12:04:39.217 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39dffff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e00a5190 con 0x7f39e00aa160 2020-11-20T12:04:39.217 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39dffff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e00a5430 con 0x7f39e00a0cd0 2020-11-20T12:04:39.217 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39deffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39e00a08f0 0x7f39e0135030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:39.217 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39d7fff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f39e00a0cd0 0x7f39e00a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:39.218 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39deffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39e00a08f0 0x7f39e0135030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51672/0 (socket says 172.21.15.47:51672) 2020-11-20T12:04:39.218 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39d7fff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f39e00a0cd0 0x7f39e00a60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:51132/0 (socket says 172.21.15.47:51132) 2020-11-20T12:04:39.218 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39deffd700 1 -- 172.21.15.47:0/1575410407 learned_addr learned my addr 172.21.15.47:0/1575410407 (peer_addr_for_me v2:172.21.15.47:0/0) 2020-11-20T12:04:39.219 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39df7fe700 1 --2- 172.21.15.47:0/1575410407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39e00aa160 0x7f39e0149940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:39.219 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39d7fff700 1 -- 172.21.15.47:0/1575410407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39e00aa160 msgr2=0x7f39e0149940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:39.219 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39d7fff700 1 --2- 172.21.15.47:0/1575410407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39e00aa160 0x7f39e0149940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.219 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39d7fff700 1 -- 172.21.15.47:0/1575410407 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39e00a08f0 msgr2=0x7f39e0135030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:39.219 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39d7fff700 1 --2- 172.21.15.47:0/1575410407 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39e00a08f0 0x7f39e0135030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.220 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39d7fff700 1 -- 172.21.15.47:0/1575410407 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e00a65a0 con 0x7f39e00a0cd0 2020-11-20T12:04:39.220 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39d7fff700 1 --2- 172.21.15.47:0/1575410407 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f39e00a0cd0 0x7f39e00a60b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f39d0006fd0 tx=0x7f39d0015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:39.220 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39d6ffd700 1 -- 172.21.15.47:0/1575410407 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d0011d50 con 0x7f39e00a0cd0 2020-11-20T12:04:39.220 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39d6ffd700 1 -- 172.21.15.47:0/1575410407 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d001e460 con 0x7f39e00a0cd0 2020-11-20T12:04:39.220 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39d6ffd700 1 -- 172.21.15.47:0/1575410407 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d001e7e0 con 0x7f39e00a0cd0 2020-11-20T12:04:39.221 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39dffff700 1 -- 172.21.15.47:0/1575410407 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39e00a49e0 con 0x7f39e00a0cd0 2020-11-20T12:04:39.221 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39dffff700 1 -- 172.21.15.47:0/1575410407 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39e0149e30 con 0x7f39e00a0cd0 2020-11-20T12:04:39.221 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39d6ffd700 1 -- 172.21.15.47:0/1575410407 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f39d001a040 con 0x7f39e00a0cd0 2020-11-20T12:04:39.221 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.205+0000 7f39d6ffd700 1 -- 172.21.15.47:0/1575410407 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1214+0+0 (secure 0 0 0) 0x7f39d001ee30 con 0x7f39e00a0cd0 2020-11-20T12:04:39.222 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.209+0000 7f39dffff700 1 -- 172.21.15.47:0/1575410407 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39e00a4420 con 0x7f39e00a0cd0 2020-11-20T12:04:39.222 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.213+0000 7f39d6ffd700 1 -- 172.21.15.47:0/1575410407 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7f39d002c060 con 0x7f39e00a0cd0 2020-11-20T12:04:39.299 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.293+0000 7f39dffff700 1 -- 172.21.15.47:0/1575410407 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "osd new", "uuid": "3ca7b438-38b2-4689-8b18-fb5a8ddf6aee", "id": 4} v 0) v1 -- 0x7f39e00446d0 con 0x7f39e00a0cd0 2020-11-20T12:04:39.305 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.301+0000 7f39d6ffd700 1 -- 172.21.15.47:0/1575410407 <== mon.2 v2:172.21.15.47:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3ca7b438-38b2-4689-8b18-fb5a8ddf6aee", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f39d0023070 con 0x7f39e00a0cd0 2020-11-20T12:04:39.306 INFO:teuthology.orchestra.run.smithi047.stdout:4 2020-11-20T12:04:39.307 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.301+0000 7f39dffff700 1 -- 172.21.15.47:0/1575410407 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f39e00a0cd0 msgr2=0x7f39e00a60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:39.307 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.301+0000 7f39dffff700 1 --2- 172.21.15.47:0/1575410407 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f39e00a0cd0 0x7f39e00a60b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f39d0006fd0 tx=0x7f39d0015000).stop 2020-11-20T12:04:39.308 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.301+0000 7f39dffff700 1 -- 172.21.15.47:0/1575410407 shutdown_connections 2020-11-20T12:04:39.308 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.301+0000 7f39dffff700 1 --2- 172.21.15.47:0/1575410407 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f39e00a08f0 0x7f39e0135030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.308 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.301+0000 7f39dffff700 1 --2- 172.21.15.47:0/1575410407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39e00aa160 0x7f39e0149940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.308 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.301+0000 7f39dffff700 1 --2- 172.21.15.47:0/1575410407 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f39e00a0cd0 0x7f39e00a60b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.308 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.301+0000 7f39dffff700 1 -- 172.21.15.47:0/1575410407 >> 172.21.15.47:0/1575410407 conn(0x7f39e0065910 msgr2=0x7f39e0138580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:39.309 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.301+0000 7f39dffff700 1 -- 172.21.15.47:0/1575410407 shutdown_connections 2020-11-20T12:04:39.309 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.301+0000 7f39dffff700 1 -- 172.21.15.47:0/1575410407 wait complete. 2020-11-20T12:04:39.319 INFO:teuthology.orchestra.run.smithi047:> sudo ceph --cluster ceph osd new 6a45fca7-83d3-44fd-a1d4-b01e4201a101 5 2020-11-20T12:04:39.440 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59bb592700 1 Processor -- start 2020-11-20T12:04:39.441 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59bb592700 1 -- start start 2020-11-20T12:04:39.443 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59bb592700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f59b40a08f0 0x7f59b4134120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:39.443 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59bb592700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59b40a0cd0 0x7f59b4134610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:39.444 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59bb592700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b40a9d00 con 0x7f59b40aa160 2020-11-20T12:04:39.444 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59bb592700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59b4068580 con 0x7f59b40a08f0 2020-11-20T12:04:39.444 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59bb592700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59b413b5c0 con 0x7f59b40a0cd0 2020-11-20T12:04:39.444 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59b9d8f700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59b40a0cd0 0x7f59b4134610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:39.444 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59b9d8f700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59b40a0cd0 0x7f59b4134610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:51136/0 (socket says 172.21.15.47:51136) 2020-11-20T12:04:39.445 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59b9d8f700 1 -- 172.21.15.47:0/3874888003 learned_addr learned my addr 172.21.15.47:0/3874888003 (peer_addr_for_me v2:172.21.15.47:0/0) 2020-11-20T12:04:39.445 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59ba590700 1 --2- 172.21.15.47:0/3874888003 >> v2:172.21.15.189:3300/0 conn(0x7f59b40a08f0 0x7f59b4134120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:39.445 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59b9d8f700 1 -- 172.21.15.47:0/3874888003 >> v2:172.21.15.189:3300/0 conn(0x7f59b40a08f0 msgr2=0x7f59b4134120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:39.445 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59b9d8f700 1 --2- 172.21.15.47:0/3874888003 >> v2:172.21.15.189:3300/0 conn(0x7f59b40a08f0 0x7f59b4134120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.445 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59b9d8f700 1 -- 172.21.15.47:0/3874888003 >> v1:172.21.15.189:6789/0 conn(0x7f59b40aa160 legacy=0x7f59b413a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:39.446 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59b9d8f700 1 -- 172.21.15.47:0/3874888003 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b413b320 con 0x7f59b40a0cd0 2020-11-20T12:04:39.446 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59b9d8f700 1 --2- 172.21.15.47:0/3874888003 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59b40a0cd0 0x7f59b4134610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f59a4006fd0 tx=0x7f59a4006eb0).ready entity=mon.2 client_cookie=c786c0a7a18cc412 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:39.446 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59b958e700 1 -- 172.21.15.47:0/3874888003 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a40098d0 con 0x7f59b40a0cd0 2020-11-20T12:04:39.446 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59b958e700 1 -- 172.21.15.47:0/3874888003 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59a4009a30 con 0x7f59b40a0cd0 2020-11-20T12:04:39.447 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59b958e700 1 -- 172.21.15.47:0/3874888003 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a4009db0 con 0x7f59b40a0cd0 2020-11-20T12:04:39.447 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59bb592700 1 -- 172.21.15.47:0/3874888003 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59b40a0cd0 msgr2=0x7f59b4134610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:39.447 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59bb592700 1 --2- 172.21.15.47:0/3874888003 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59b40a0cd0 0x7f59b4134610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f59a4006fd0 tx=0x7f59a4006eb0).stop 2020-11-20T12:04:39.447 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59bb592700 1 -- 172.21.15.47:0/3874888003 shutdown_connections 2020-11-20T12:04:39.447 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59bb592700 1 --2- 172.21.15.47:0/3874888003 >> v2:172.21.15.189:3300/0 conn(0x7f59b40a08f0 0x7f59b4134120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.448 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59bb592700 1 --2- 172.21.15.47:0/3874888003 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59b40a0cd0 0x7f59b4134610 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.448 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59bb592700 1 -- 172.21.15.47:0/3874888003 >> 172.21.15.47:0/3874888003 conn(0x7f59b4065910 msgr2=0x7f59b4060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:39.448 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59bb592700 1 -- 172.21.15.47:0/3874888003 shutdown_connections 2020-11-20T12:04:39.448 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.437+0000 7f59bb592700 1 -- 172.21.15.47:0/3874888003 wait complete. 2020-11-20T12:04:39.450 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.441+0000 7f59bb592700 1 Processor -- start 2020-11-20T12:04:39.451 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.441+0000 7f59bb592700 1 -- start start 2020-11-20T12:04:39.451 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.441+0000 7f59bb592700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59b40a08f0 0x7f59b40a7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:39.451 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.441+0000 7f59bb592700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59b40a0cd0 0x7f59b413e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:39.451 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.441+0000 7f59bb592700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59b40aa160 0x7f59b413c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:39.452 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.441+0000 7f59bb592700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b413d5b0 con 0x7f59b40a0cd0 2020-11-20T12:04:39.452 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.441+0000 7f59bb592700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b40ac780 con 0x7f59b40aa160 2020-11-20T12:04:39.452 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.441+0000 7f59bb592700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59b40aca20 con 0x7f59b40a08f0 2020-11-20T12:04:39.453 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.441+0000 7f59b9d8f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59b40a0cd0 0x7f59b413e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:39.453 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.441+0000 7f59ba590700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59b40a08f0 0x7f59b40a7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:39.453 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.441+0000 7f59b9d8f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59b40a0cd0 0x7f59b413e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51684/0 (socket says 172.21.15.47:51684) 2020-11-20T12:04:39.454 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.441+0000 7f59b9d8f700 1 -- 172.21.15.47:0/3671483892 learned_addr learned my addr 172.21.15.47:0/3671483892 (peer_addr_for_me v2:172.21.15.47:0/0) 2020-11-20T12:04:39.454 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.441+0000 7f59bad91700 1 --2- 172.21.15.47:0/3671483892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59b40aa160 0x7f59b413c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:39.454 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.441+0000 7f59b9d8f700 1 -- 172.21.15.47:0/3671483892 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59b40a08f0 msgr2=0x7f59b40a7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:39.454 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.441+0000 7f59b9d8f700 1 --2- 172.21.15.47:0/3671483892 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59b40a08f0 0x7f59b40a7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.455 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.441+0000 7f59b9d8f700 1 -- 172.21.15.47:0/3671483892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59b40aa160 msgr2=0x7f59b413c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:39.455 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.441+0000 7f59b9d8f700 1 --2- 172.21.15.47:0/3671483892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59b40aa160 0x7f59b413c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.455 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.441+0000 7f59b9d8f700 1 -- 172.21.15.47:0/3671483892 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b40accc0 con 0x7f59b40a0cd0 2020-11-20T12:04:39.456 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.441+0000 7f59b9d8f700 1 --2- 172.21.15.47:0/3671483892 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59b40a0cd0 0x7f59b413e020 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f59b40a9e90 tx=0x7f59a4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:39.456 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.441+0000 7f59a27fc700 1 -- 172.21.15.47:0/3671483892 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a4011dd0 con 0x7f59b40a0cd0 2020-11-20T12:04:39.456 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.441+0000 7f59a27fc700 1 -- 172.21.15.47:0/3671483892 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59a401e460 con 0x7f59b40a0cd0 2020-11-20T12:04:39.457 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.441+0000 7f59bb592700 1 -- 172.21.15.47:0/3671483892 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59b40a89a0 con 0x7f59b40a0cd0 2020-11-20T12:04:39.457 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.441+0000 7f59a27fc700 1 -- 172.21.15.47:0/3671483892 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a401e7e0 con 0x7f59b40a0cd0 2020-11-20T12:04:39.458 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.441+0000 7f59bb592700 1 -- 172.21.15.47:0/3671483892 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59b413e730 con 0x7f59b40a0cd0 2020-11-20T12:04:39.458 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.441+0000 7f59a27fc700 1 -- 172.21.15.47:0/3671483892 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f59a4025030 con 0x7f59b40a0cd0 2020-11-20T12:04:39.458 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.441+0000 7f59a27fc700 1 -- 172.21.15.47:0/3671483892 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1329+0+0 (secure 0 0 0) 0x7f59a401edb0 con 0x7f59b40a0cd0 2020-11-20T12:04:39.459 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.445+0000 7f59bb592700 1 -- 172.21.15.47:0/3671483892 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59b40a4420 con 0x7f59b40a0cd0 2020-11-20T12:04:39.459 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.449+0000 7f59a27fc700 1 -- 172.21.15.47:0/3671483892 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7f59a402d060 con 0x7f59b40a0cd0 2020-11-20T12:04:39.534 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.529+0000 7f59bb592700 1 -- 172.21.15.47:0/3671483892 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6a45fca7-83d3-44fd-a1d4-b01e4201a101", "id": 5} v 0) v1 -- 0x7f59b413ae00 con 0x7f59b40a0cd0 2020-11-20T12:04:39.539 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.533+0000 7f59a27fc700 1 -- 172.21.15.47:0/3671483892 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6a45fca7-83d3-44fd-a1d4-b01e4201a101", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f59a4023070 con 0x7f59b40a0cd0 2020-11-20T12:04:39.539 INFO:teuthology.orchestra.run.smithi047.stdout:5 2020-11-20T12:04:39.541 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.537+0000 7f59bb592700 1 -- 172.21.15.47:0/3671483892 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59b40a0cd0 msgr2=0x7f59b413e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:39.541 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.537+0000 7f59bb592700 1 --2- 172.21.15.47:0/3671483892 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59b40a0cd0 0x7f59b413e020 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f59b40a9e90 tx=0x7f59a4015000).stop 2020-11-20T12:04:39.541 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.537+0000 7f59bb592700 1 -- 172.21.15.47:0/3671483892 shutdown_connections 2020-11-20T12:04:39.542 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.537+0000 7f59bb592700 1 --2- 172.21.15.47:0/3671483892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59b40aa160 0x7f59b413c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.542 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.537+0000 7f59bb592700 1 --2- 172.21.15.47:0/3671483892 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59b40a08f0 0x7f59b40a7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.542 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.537+0000 7f59bb592700 1 --2- 172.21.15.47:0/3671483892 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59b40a0cd0 0x7f59b413e020 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.542 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.537+0000 7f59bb592700 1 -- 172.21.15.47:0/3671483892 >> 172.21.15.47:0/3671483892 conn(0x7f59b4065910 msgr2=0x7f59b40adc80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:39.543 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.537+0000 7f59bb592700 1 -- 172.21.15.47:0/3671483892 shutdown_connections 2020-11-20T12:04:39.543 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.537+0000 7f59bb592700 1 -- 172.21.15.47:0/3671483892 wait complete. 2020-11-20T12:04:39.559 INFO:teuthology.orchestra.run.smithi047:> sudo ceph --cluster ceph osd new c4b1b9ed-61f8-4677-9755-dff59a30e447 6 2020-11-20T12:04:39.678 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.673+0000 7f3233fff700 1 Processor -- start 2020-11-20T12:04:39.679 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.673+0000 7f3233fff700 1 -- start start 2020-11-20T12:04:39.681 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.673+0000 7f3233fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3234137b00 0x7f32340a8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:39.681 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.673+0000 7f3233fff700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f3234067d90 0x7f3234066a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:39.682 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.673+0000 7f3233fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32340a9410 con 0x7f3234060ba0 2020-11-20T12:04:39.682 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.673+0000 7f3233fff700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32340684c0 con 0x7f3234067d90 2020-11-20T12:04:39.682 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.673+0000 7f3233fff700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32340678f0 con 0x7f3234137b00 2020-11-20T12:04:39.682 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.673+0000 7f32337fe700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f3234067d90 0x7f3234066a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:39.682 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.673+0000 7f32337fe700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f3234067d90 0x7f3234066a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51692/0 (socket says 172.21.15.47:51692) 2020-11-20T12:04:39.683 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.673+0000 7f32337fe700 1 -- 172.21.15.47:0/1177608223 learned_addr learned my addr 172.21.15.47:0/1177608223 (peer_addr_for_me v2:172.21.15.47:0/0) 2020-11-20T12:04:39.683 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.673+0000 7f322bfff700 1 --2- 172.21.15.47:0/1177608223 >> v2:172.21.15.189:3300/0 conn(0x7f3234137b00 0x7f32340a8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:39.683 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.673+0000 7f32337fe700 1 -- 172.21.15.47:0/1177608223 >> v2:172.21.15.189:3300/0 conn(0x7f3234137b00 msgr2=0x7f32340a8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:39.683 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.673+0000 7f32337fe700 1 --2- 172.21.15.47:0/1177608223 >> v2:172.21.15.189:3300/0 conn(0x7f3234137b00 0x7f32340a8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.683 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f32337fe700 1 -- 172.21.15.47:0/1177608223 >> v1:172.21.15.189:6789/0 conn(0x7f3234060ba0 legacy=0x7f32341373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:39.684 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f32337fe700 1 -- 172.21.15.47:0/1177608223 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3234060f80 con 0x7f3234067d90 2020-11-20T12:04:39.684 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f32337fe700 1 --2- 172.21.15.47:0/1177608223 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3234067d90 0x7f3234066a40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f3234068950 tx=0x7f322c009290).ready entity=mon.0 client_cookie=abd4df2a19bdbd96 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:39.684 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f32327fc700 1 -- 172.21.15.47:0/1177608223 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c00bbc0 con 0x7f3234067d90 2020-11-20T12:04:39.684 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f32327fc700 1 -- 172.21.15.47:0/1177608223 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322c00bd20 con 0x7f3234067d90 2020-11-20T12:04:39.684 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f32327fc700 1 -- 172.21.15.47:0/1177608223 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c011610 con 0x7f3234067d90 2020-11-20T12:04:39.685 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f3233fff700 1 -- 172.21.15.47:0/1177608223 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3234067d90 msgr2=0x7f3234066a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:39.685 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f3233fff700 1 --2- 172.21.15.47:0/1177608223 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3234067d90 0x7f3234066a40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f3234068950 tx=0x7f322c009290).stop 2020-11-20T12:04:39.685 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f3233fff700 1 -- 172.21.15.47:0/1177608223 shutdown_connections 2020-11-20T12:04:39.685 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f3233fff700 1 --2- 172.21.15.47:0/1177608223 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3234067d90 0x7f3234066a40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.685 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f3233fff700 1 --2- 172.21.15.47:0/1177608223 >> v2:172.21.15.189:3300/0 conn(0x7f3234137b00 0x7f32340a8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.686 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f3233fff700 1 -- 172.21.15.47:0/1177608223 >> 172.21.15.47:0/1177608223 conn(0x7f32340ad8e0 msgr2=0x7f32340adcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:39.686 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f3233fff700 1 -- 172.21.15.47:0/1177608223 shutdown_connections 2020-11-20T12:04:39.686 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f3233fff700 1 -- 172.21.15.47:0/1177608223 wait complete. 2020-11-20T12:04:39.690 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f3233fff700 1 Processor -- start 2020-11-20T12:04:39.690 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f3233fff700 1 -- start start 2020-11-20T12:04:39.691 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f3233fff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3234060ba0 0x7f323413cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:39.691 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f3233fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3234067d90 0x7f3234130a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:39.691 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f3233fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3234137b00 0x7f3234149860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:39.692 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f3233fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323414a050 con 0x7f3234137b00 2020-11-20T12:04:39.692 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f3233fff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323414a2f0 con 0x7f3234067d90 2020-11-20T12:04:39.692 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f3233fff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f323414a590 con 0x7f3234060ba0 2020-11-20T12:04:39.693 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f32337fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3234137b00 0x7f3234149860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:39.693 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f3232ffd700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3234060ba0 0x7f323413cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:39.693 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f32337fe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3234137b00 0x7f3234149860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51698/0 (socket says 172.21.15.47:51698) 2020-11-20T12:04:39.694 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f32337fe700 1 -- 172.21.15.47:0/3157226772 learned_addr learned my addr 172.21.15.47:0/3157226772 (peer_addr_for_me v2:172.21.15.47:0/0) 2020-11-20T12:04:39.694 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f322bfff700 1 --2- 172.21.15.47:0/3157226772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3234067d90 0x7f3234130a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:39.694 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f32337fe700 1 -- 172.21.15.47:0/3157226772 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3234060ba0 msgr2=0x7f323413cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:39.695 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f32337fe700 1 --2- 172.21.15.47:0/3157226772 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3234060ba0 0x7f323413cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.695 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f32337fe700 1 -- 172.21.15.47:0/3157226772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3234067d90 msgr2=0x7f3234130a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:39.696 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f32337fe700 1 --2- 172.21.15.47:0/3157226772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3234067d90 0x7f3234130a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.696 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f32337fe700 1 -- 172.21.15.47:0/3157226772 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3234149d50 con 0x7f3234137b00 2020-11-20T12:04:39.696 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f322bfff700 1 --2- 172.21.15.47:0/3157226772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3234067d90 0x7f3234130a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:39.696 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f32337fe700 1 --2- 172.21.15.47:0/3157226772 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3234137b00 0x7f3234149860 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f3234065880 tx=0x7f322c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:39.697 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f322affd700 1 -- 172.21.15.47:0/3157226772 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c00bbc0 con 0x7f3234137b00 2020-11-20T12:04:39.697 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f322affd700 1 -- 172.21.15.47:0/3157226772 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322c00be90 con 0x7f3234137b00 2020-11-20T12:04:39.697 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f322affd700 1 -- 172.21.15.47:0/3157226772 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c010da0 con 0x7f3234137b00 2020-11-20T12:04:39.697 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.677+0000 7f3233fff700 1 -- 172.21.15.47:0/3157226772 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f323413a900 con 0x7f3234137b00 2020-11-20T12:04:39.697 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.681+0000 7f3233fff700 1 -- 172.21.15.47:0/3157226772 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f323413ad20 con 0x7f3234137b00 2020-11-20T12:04:39.698 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.681+0000 7f322affd700 1 -- 172.21.15.47:0/3157226772 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f322c017070 con 0x7f3234137b00 2020-11-20T12:04:39.698 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.681+0000 7f322affd700 1 -- 172.21.15.47:0/3157226772 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1444+0+0 (secure 0 0 0) 0x7f322c026a80 con 0x7f3234137b00 2020-11-20T12:04:39.698 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.681+0000 7f3233fff700 1 -- 172.21.15.47:0/3157226772 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f323414a750 con 0x7f3234137b00 2020-11-20T12:04:39.698 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.685+0000 7f322affd700 1 -- 172.21.15.47:0/3157226772 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7f322c015020 con 0x7f3234137b00 2020-11-20T12:04:39.771 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.765+0000 7f3233fff700 1 -- 172.21.15.47:0/3157226772 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c4b1b9ed-61f8-4677-9755-dff59a30e447", "id": 6} v 0) v1 -- 0x7f3234136de0 con 0x7f3234137b00 2020-11-20T12:04:39.774 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.769+0000 7f322affd700 1 -- 172.21.15.47:0/3157226772 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c4b1b9ed-61f8-4677-9755-dff59a30e447", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f322c01ca70 con 0x7f3234137b00 2020-11-20T12:04:39.775 INFO:teuthology.orchestra.run.smithi047.stdout:6 2020-11-20T12:04:39.777 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.769+0000 7f3233fff700 1 -- 172.21.15.47:0/3157226772 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3234137b00 msgr2=0x7f3234149860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:39.777 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.769+0000 7f3233fff700 1 --2- 172.21.15.47:0/3157226772 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3234137b00 0x7f3234149860 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f3234065880 tx=0x7f322c016000).stop 2020-11-20T12:04:39.778 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.769+0000 7f3233fff700 1 -- 172.21.15.47:0/3157226772 shutdown_connections 2020-11-20T12:04:39.778 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.769+0000 7f3233fff700 1 --2- 172.21.15.47:0/3157226772 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3234137b00 0x7f3234149860 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.778 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.769+0000 7f3233fff700 1 --2- 172.21.15.47:0/3157226772 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3234060ba0 0x7f323413cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.779 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.773+0000 7f3233fff700 1 --2- 172.21.15.47:0/3157226772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3234067d90 0x7f3234130a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.779 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.773+0000 7f3233fff700 1 -- 172.21.15.47:0/3157226772 >> 172.21.15.47:0/3157226772 conn(0x7f32340ad8e0 msgr2=0x7f32340adcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:39.780 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.773+0000 7f3233fff700 1 -- 172.21.15.47:0/3157226772 shutdown_connections 2020-11-20T12:04:39.780 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.773+0000 7f3233fff700 1 -- 172.21.15.47:0/3157226772 wait complete. 2020-11-20T12:04:39.789 INFO:teuthology.orchestra.run.smithi047:> sudo ceph --cluster ceph osd new 4608e4ed-cc40-4c5c-ba8b-3779bbc62f65 7 2020-11-20T12:04:39.902 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.897+0000 7f1d7b345700 1 Processor -- start 2020-11-20T12:04:39.902 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.897+0000 7f1d7b345700 1 -- start start 2020-11-20T12:04:39.905 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.897+0000 7f1d7b345700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1d74060ba0 0x7f1d740a8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:39.906 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.897+0000 7f1d7b345700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1d74137b00 0x7f1d74066a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:39.906 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.897+0000 7f1d7b345700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d740a9410 con 0x7f1d74067d90 2020-11-20T12:04:39.906 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.897+0000 7f1d7b345700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d740684c0 con 0x7f1d74137b00 2020-11-20T12:04:39.906 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.897+0000 7f1d7b345700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d740678f0 con 0x7f1d74060ba0 2020-11-20T12:04:39.907 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.897+0000 7f1d79b42700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f1d74067d90 0x7f1d741373f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:43232/0 (socket says 172.21.15.47:43232) 2020-11-20T12:04:39.907 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.897+0000 7f1d79b42700 1 -- 172.21.15.47:0/3812474960 learned_addr learned my addr 172.21.15.47:0/3812474960 (peer_addr_for_me v1:172.21.15.47:0/0) 2020-11-20T12:04:39.907 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.897+0000 7f1d7a343700 1 --2- 172.21.15.47:0/3812474960 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1d74060ba0 0x7f1d740a8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:39.907 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.897+0000 7f1d7ab44700 1 --2- 172.21.15.47:0/3812474960 >> v2:172.21.15.189:3300/0 conn(0x7f1d74137b00 0x7f1d74066a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:39.907 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.897+0000 7f1d79341700 1 -- 172.21.15.47:0/3812474960 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3497755575 0 0) 0x7f1d740a9410 con 0x7f1d74067d90 2020-11-20T12:04:39.908 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.897+0000 7f1d79341700 1 -- 172.21.15.47:0/3812474960 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d5c003240 con 0x7f1d74067d90 2020-11-20T12:04:39.908 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.897+0000 7f1d7a343700 1 -- 172.21.15.47:0/3812474960 >> v2:172.21.15.189:3300/0 conn(0x7f1d74137b00 msgr2=0x7f1d74066a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:39.908 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.897+0000 7f1d7a343700 1 --2- 172.21.15.47:0/3812474960 >> v2:172.21.15.189:3300/0 conn(0x7f1d74137b00 0x7f1d74066a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.909 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.897+0000 7f1d7a343700 1 -- 172.21.15.47:0/3812474960 >> v1:172.21.15.47:6789/0 conn(0x7f1d74067d90 legacy=0x7f1d741373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:39.909 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.897+0000 7f1d7a343700 1 -- 172.21.15.47:0/3812474960 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d74060f80 con 0x7f1d74060ba0 2020-11-20T12:04:39.909 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.901+0000 7f1d7a343700 1 --2- 172.21.15.47:0/3812474960 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1d74060ba0 0x7f1d740a8d70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f1d6400b9a0 tx=0x7f1d64006df0).ready entity=mon.2 client_cookie=e199d219ecfd816a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:39.909 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.901+0000 7f1d79341700 1 -- 172.21.15.47:0/3812474960 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d64009680 con 0x7f1d74060ba0 2020-11-20T12:04:39.910 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.901+0000 7f1d79341700 1 -- 172.21.15.47:0/3812474960 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d640097e0 con 0x7f1d74060ba0 2020-11-20T12:04:39.910 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.901+0000 7f1d79341700 1 -- 172.21.15.47:0/3812474960 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d64009b60 con 0x7f1d74060ba0 2020-11-20T12:04:39.910 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.901+0000 7f1d7b345700 1 -- 172.21.15.47:0/3812474960 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1d74060ba0 msgr2=0x7f1d740a8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:39.911 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.901+0000 7f1d7b345700 1 --2- 172.21.15.47:0/3812474960 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1d74060ba0 0x7f1d740a8d70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f1d6400b9a0 tx=0x7f1d64006df0).stop 2020-11-20T12:04:39.911 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.901+0000 7f1d7b345700 1 -- 172.21.15.47:0/3812474960 shutdown_connections 2020-11-20T12:04:39.911 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.901+0000 7f1d7b345700 1 --2- 172.21.15.47:0/3812474960 >> v2:172.21.15.189:3300/0 conn(0x7f1d74137b00 0x7f1d74066a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.911 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.901+0000 7f1d7b345700 1 --2- 172.21.15.47:0/3812474960 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1d74060ba0 0x7f1d740a8d70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.911 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.901+0000 7f1d7b345700 1 -- 172.21.15.47:0/3812474960 >> 172.21.15.47:0/3812474960 conn(0x7f1d740ad8e0 msgr2=0x7f1d740adcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:39.912 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.901+0000 7f1d7b345700 1 -- 172.21.15.47:0/3812474960 shutdown_connections 2020-11-20T12:04:39.912 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.901+0000 7f1d7b345700 1 -- 172.21.15.47:0/3812474960 wait complete. 2020-11-20T12:04:39.914 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.901+0000 7f1d7b345700 1 Processor -- start 2020-11-20T12:04:39.914 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.901+0000 7f1d7b345700 1 -- start start 2020-11-20T12:04:39.914 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.901+0000 7f1d7b345700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1d74060ba0 0x7f1d740ac860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:39.915 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.901+0000 7f1d7b345700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d74067d90 0x7f1d740a7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:39.915 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.901+0000 7f1d7b345700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d74137b00 0x7f1d74149c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:39.915 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.901+0000 7f1d7b345700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d740a1320 con 0x7f1d74137b00 2020-11-20T12:04:39.915 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.901+0000 7f1d7b345700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d740a7c90 con 0x7f1d74067d90 2020-11-20T12:04:39.916 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.901+0000 7f1d7b345700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d740a1080 con 0x7f1d74060ba0 2020-11-20T12:04:39.916 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.901+0000 7f1d7a343700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1d74060ba0 0x7f1d740ac860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:39.916 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.901+0000 7f1d7ab44700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d74137b00 0x7f1d74149c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:39.916 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.901+0000 7f1d7a343700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1d74060ba0 0x7f1d740ac860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.47:51164/0 (socket says 172.21.15.47:51164) 2020-11-20T12:04:39.917 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.901+0000 7f1d7ab44700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d74137b00 0x7f1d74149c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.47:51710/0 (socket says 172.21.15.47:51710) 2020-11-20T12:04:39.917 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.901+0000 7f1d7a343700 1 -- 172.21.15.47:0/2672501420 learned_addr learned my addr 172.21.15.47:0/2672501420 (peer_addr_for_me v2:172.21.15.47:0/0) 2020-11-20T12:04:39.917 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.901+0000 7f1d79b42700 1 --2- 172.21.15.47:0/2672501420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d74067d90 0x7f1d740a7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:39.917 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.905+0000 7f1d7a343700 1 -- 172.21.15.47:0/2672501420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d74067d90 msgr2=0x7f1d740a7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:39.917 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.905+0000 7f1d7a343700 1 --2- 172.21.15.47:0/2672501420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d74067d90 0x7f1d740a7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.918 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.905+0000 7f1d7a343700 1 -- 172.21.15.47:0/2672501420 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d74137b00 msgr2=0x7f1d74149c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:39.918 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.905+0000 7f1d7a343700 1 --2- 172.21.15.47:0/2672501420 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d74137b00 0x7f1d74149c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:39.918 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.905+0000 7f1d7a343700 1 -- 172.21.15.47:0/2672501420 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d740a6d80 con 0x7f1d74060ba0 2020-11-20T12:04:39.918 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.905+0000 7f1d7ab44700 1 --2- 172.21.15.47:0/2672501420 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d74137b00 0x7f1d74149c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:04:39.918 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.905+0000 7f1d7a343700 1 --2- 172.21.15.47:0/2672501420 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1d74060ba0 0x7f1d740ac860 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f1d6400c010 tx=0x7f1d64015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:39.919 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.905+0000 7f1d627fc700 1 -- 172.21.15.47:0/2672501420 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d64011c40 con 0x7f1d74060ba0 2020-11-20T12:04:39.919 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.905+0000 7f1d627fc700 1 -- 172.21.15.47:0/2672501420 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d64011da0 con 0x7f1d74060ba0 2020-11-20T12:04:39.919 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.905+0000 7f1d7b345700 1 -- 172.21.15.47:0/2672501420 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d74149670 con 0x7f1d74060ba0 2020-11-20T12:04:39.920 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.905+0000 7f1d627fc700 1 -- 172.21.15.47:0/2672501420 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6401e670 con 0x7f1d74060ba0 2020-11-20T12:04:39.920 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.905+0000 7f1d7b345700 1 -- 172.21.15.47:0/2672501420 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d740a78b0 con 0x7f1d74060ba0 2020-11-20T12:04:39.920 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.905+0000 7f1d627fc700 1 -- 172.21.15.47:0/2672501420 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f1d64025030 con 0x7f1d74060ba0 2020-11-20T12:04:39.920 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.905+0000 7f1d627fc700 1 -- 172.21.15.47:0/2672501420 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1559+0+0 (secure 0 0 0) 0x7f1d64028b20 con 0x7f1d74060ba0 2020-11-20T12:04:39.920 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.905+0000 7f1d7b345700 1 -- 172.21.15.47:0/2672501420 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d740446d0 con 0x7f1d74060ba0 2020-11-20T12:04:39.921 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.909+0000 7f1d627fc700 1 -- 172.21.15.47:0/2672501420 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7f1d6402d060 con 0x7f1d74060ba0 2020-11-20T12:04:39.997 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.993+0000 7f1d7b345700 1 -- 172.21.15.47:0/2672501420 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "osd new", "uuid": "4608e4ed-cc40-4c5c-ba8b-3779bbc62f65", "id": 7} v 0) v1 -- 0x7f1d740670a0 con 0x7f1d74060ba0 2020-11-20T12:04:40.003 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:39.997+0000 7f1d627fc700 1 -- 172.21.15.47:0/2672501420 <== mon.2 v2:172.21.15.47:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4608e4ed-cc40-4c5c-ba8b-3779bbc62f65", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1d64023070 con 0x7f1d74060ba0 2020-11-20T12:04:40.004 INFO:teuthology.orchestra.run.smithi047.stdout:7 2020-11-20T12:04:40.005 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:40.001+0000 7f1d7b345700 1 -- 172.21.15.47:0/2672501420 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1d74060ba0 msgr2=0x7f1d740ac860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:40.005 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:40.001+0000 7f1d7b345700 1 --2- 172.21.15.47:0/2672501420 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1d74060ba0 0x7f1d740ac860 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f1d6400c010 tx=0x7f1d64015000).stop 2020-11-20T12:04:40.005 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:40.001+0000 7f1d7b345700 1 -- 172.21.15.47:0/2672501420 shutdown_connections 2020-11-20T12:04:40.006 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:40.001+0000 7f1d7b345700 1 --2- 172.21.15.47:0/2672501420 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1d74137b00 0x7f1d74149c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:40.006 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:40.001+0000 7f1d7b345700 1 --2- 172.21.15.47:0/2672501420 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1d74060ba0 0x7f1d740ac860 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:40.006 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:40.001+0000 7f1d7b345700 1 --2- 172.21.15.47:0/2672501420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d74067d90 0x7f1d740a7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:40.006 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:40.001+0000 7f1d7b345700 1 -- 172.21.15.47:0/2672501420 >> 172.21.15.47:0/2672501420 conn(0x7f1d740ad8e0 msgr2=0x7f1d741367b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:40.006 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:40.001+0000 7f1d7b345700 1 -- 172.21.15.47:0/2672501420 shutdown_connections 2020-11-20T12:04:40.007 INFO:teuthology.orchestra.run.smithi047.stderr:2020-11-20T12:04:40.001+0000 7f1d7b345700 1 -- 172.21.15.47:0/2672501420 wait complete. 2020-11-20T12:04:40.017 INFO:tasks.ceph.osd.0:Restarting daemon 2020-11-20T12:04:40.017 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2020-11-20T12:04:40.019 INFO:tasks.ceph.osd.0:Started 2020-11-20T12:04:40.020 INFO:tasks.ceph.osd.1:Restarting daemon 2020-11-20T12:04:40.020 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2020-11-20T12:04:40.022 INFO:tasks.ceph.osd.1:Started 2020-11-20T12:04:40.022 INFO:tasks.ceph.osd.2:Restarting daemon 2020-11-20T12:04:40.023 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2020-11-20T12:04:40.024 INFO:tasks.ceph.osd.2:Started 2020-11-20T12:04:40.025 INFO:tasks.ceph.osd.3:Restarting daemon 2020-11-20T12:04:40.025 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2020-11-20T12:04:40.028 INFO:tasks.ceph.osd.3:Started 2020-11-20T12:04:40.028 INFO:tasks.ceph.osd.4:Restarting daemon 2020-11-20T12:04:40.029 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2020-11-20T12:04:40.032 INFO:tasks.ceph.osd.4:Started 2020-11-20T12:04:40.033 INFO:tasks.ceph.osd.5:Restarting daemon 2020-11-20T12:04:40.034 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2020-11-20T12:04:40.037 INFO:tasks.ceph.osd.5:Started 2020-11-20T12:04:40.039 INFO:tasks.ceph.osd.6:Restarting daemon 2020-11-20T12:04:40.039 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2020-11-20T12:04:40.042 INFO:tasks.ceph.osd.6:Started 2020-11-20T12:04:40.042 INFO:tasks.ceph.osd.7:Restarting daemon 2020-11-20T12:04:40.042 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2020-11-20T12:04:40.044 INFO:tasks.ceph.osd.7:Started 2020-11-20T12:04:40.045 INFO:tasks.ceph:Waiting for OSDs to come up 2020-11-20T12:04:40.099 INFO:tasks.ceph.osd.5.smithi047.stderr:2020-11-20T12:04:40.093+0000 7fe3106fdd80 -1 Falling back to public interface 2020-11-20T12:04:40.106 INFO:tasks.ceph.osd.7.smithi047.stderr:2020-11-20T12:04:40.101+0000 7f8964779d80 -1 Falling back to public interface 2020-11-20T12:04:40.113 INFO:tasks.ceph.osd.6.smithi047.stderr:2020-11-20T12:04:40.109+0000 7f75dd651d80 -1 Falling back to public interface 2020-11-20T12:04:40.115 INFO:tasks.ceph.osd.5.smithi047.stderr:2020-11-20T12:04:40.109+0000 7fe3106fdd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-20T12:04:40.115 INFO:tasks.ceph.osd.5.smithi047.stderr:2020-11-20T12:04:40.109+0000 7fe3106fdd80 -1 journal do_read_entry(8192): bad header magic 2020-11-20T12:04:40.115 INFO:tasks.ceph.osd.5.smithi047.stderr:2020-11-20T12:04:40.109+0000 7fe3106fdd80 -1 journal do_read_entry(8192): bad header magic 2020-11-20T12:04:40.123 INFO:tasks.ceph.osd.4.smithi047.stderr:2020-11-20T12:04:40.117+0000 7f9422177d80 -1 Falling back to public interface 2020-11-20T12:04:40.123 INFO:tasks.ceph.osd.7.smithi047.stderr:2020-11-20T12:04:40.117+0000 7f8964779d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-20T12:04:40.123 INFO:tasks.ceph.osd.7.smithi047.stderr:2020-11-20T12:04:40.117+0000 7f8964779d80 -1 journal do_read_entry(8192): bad header magic 2020-11-20T12:04:40.123 INFO:tasks.ceph.osd.7.smithi047.stderr:2020-11-20T12:04:40.117+0000 7f8964779d80 -1 journal do_read_entry(8192): bad header magic 2020-11-20T12:04:40.124 INFO:tasks.ceph.osd.5.smithi047.stderr:2020-11-20T12:04:40.117+0000 7fe3106fdd80 -1 osd.5 0 log_to_monitors {default=true} 2020-11-20T12:04:40.130 INFO:tasks.ceph.osd.7.smithi047.stderr:2020-11-20T12:04:40.125+0000 7f8964779d80 -1 osd.7 0 log_to_monitors {default=true} 2020-11-20T12:04:40.139 INFO:tasks.ceph.osd.6.smithi047.stderr:2020-11-20T12:04:40.133+0000 7f75dd651d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-20T12:04:40.139 INFO:tasks.ceph.osd.6.smithi047.stderr:2020-11-20T12:04:40.133+0000 7f75dd651d80 -1 journal do_read_entry(8192): bad header magic 2020-11-20T12:04:40.140 INFO:tasks.ceph.osd.6.smithi047.stderr:2020-11-20T12:04:40.133+0000 7f75dd651d80 -1 journal do_read_entry(8192): bad header magic 2020-11-20T12:04:40.141 INFO:tasks.ceph.osd.4.smithi047.stderr:2020-11-20T12:04:40.137+0000 7f9422177d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-20T12:04:40.141 INFO:tasks.ceph.osd.4.smithi047.stderr:2020-11-20T12:04:40.137+0000 7f9422177d80 -1 journal do_read_entry(8192): bad header magic 2020-11-20T12:04:40.142 INFO:tasks.ceph.osd.4.smithi047.stderr:2020-11-20T12:04:40.137+0000 7f9422177d80 -1 journal do_read_entry(8192): bad header magic 2020-11-20T12:04:40.144 INFO:tasks.ceph.osd.6.smithi047.stderr:2020-11-20T12:04:40.141+0000 7f75dd651d80 -1 osd.6 0 log_to_monitors {default=true} 2020-11-20T12:04:40.147 INFO:tasks.ceph.osd.4.smithi047.stderr:2020-11-20T12:04:40.141+0000 7f9422177d80 -1 osd.4 0 log_to_monitors {default=true} 2020-11-20T12:04:40.402 INFO:tasks.ceph.osd.3.smithi189.stderr:2020-11-20T12:04:40.401+0000 7f07e3486d80 -1 Falling back to public interface 2020-11-20T12:04:40.403 INFO:tasks.ceph.osd.1.smithi189.stderr:2020-11-20T12:04:40.401+0000 7f3621b45d80 -1 Falling back to public interface 2020-11-20T12:04:40.404 INFO:tasks.ceph.osd.0.smithi189.stderr:2020-11-20T12:04:40.405+0000 7f3179261d80 -1 Falling back to public interface 2020-11-20T12:04:40.407 INFO:tasks.ceph.osd.2.smithi189.stderr:2020-11-20T12:04:40.405+0000 7f82b25bbd80 -1 Falling back to public interface 2020-11-20T12:04:40.420 INFO:tasks.ceph.osd.3.smithi189.stderr:2020-11-20T12:04:40.417+0000 7f07e3486d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-20T12:04:40.420 INFO:tasks.ceph.osd.3.smithi189.stderr:2020-11-20T12:04:40.417+0000 7f07e3486d80 -1 journal do_read_entry(8192): bad header magic 2020-11-20T12:04:40.420 INFO:tasks.ceph.osd.3.smithi189.stderr:2020-11-20T12:04:40.417+0000 7f07e3486d80 -1 journal do_read_entry(8192): bad header magic 2020-11-20T12:04:40.421 INFO:tasks.ceph.osd.0.smithi189.stderr:2020-11-20T12:04:40.421+0000 7f3179261d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-20T12:04:40.421 INFO:tasks.ceph.osd.0.smithi189.stderr:2020-11-20T12:04:40.421+0000 7f3179261d80 -1 journal do_read_entry(8192): bad header magic 2020-11-20T12:04:40.422 INFO:tasks.ceph.osd.0.smithi189.stderr:2020-11-20T12:04:40.421+0000 7f3179261d80 -1 journal do_read_entry(8192): bad header magic 2020-11-20T12:04:40.422 INFO:tasks.ceph.osd.2.smithi189.stderr:2020-11-20T12:04:40.421+0000 7f82b25bbd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-20T12:04:40.422 INFO:tasks.ceph.osd.2.smithi189.stderr:2020-11-20T12:04:40.421+0000 7f82b25bbd80 -1 journal do_read_entry(8192): bad header magic 2020-11-20T12:04:40.423 INFO:tasks.ceph.osd.2.smithi189.stderr:2020-11-20T12:04:40.421+0000 7f82b25bbd80 -1 journal do_read_entry(8192): bad header magic 2020-11-20T12:04:40.426 INFO:tasks.ceph.osd.3.smithi189.stderr:2020-11-20T12:04:40.425+0000 7f07e3486d80 -1 osd.3 0 log_to_monitors {default=true} 2020-11-20T12:04:40.427 INFO:tasks.ceph.osd.0.smithi189.stderr:2020-11-20T12:04:40.425+0000 7f3179261d80 -1 osd.0 0 log_to_monitors {default=true} 2020-11-20T12:04:40.428 INFO:tasks.ceph.osd.2.smithi189.stderr:2020-11-20T12:04:40.425+0000 7f82b25bbd80 -1 osd.2 0 log_to_monitors {default=true} 2020-11-20T12:04:40.428 INFO:tasks.ceph.osd.1.smithi189.stderr:2020-11-20T12:04:40.425+0000 7f3621b45d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2020-11-20T12:04:40.428 INFO:tasks.ceph.osd.1.smithi189.stderr:2020-11-20T12:04:40.429+0000 7f3621b45d80 -1 journal do_read_entry(8192): bad header magic 2020-11-20T12:04:40.429 INFO:tasks.ceph.osd.1.smithi189.stderr:2020-11-20T12:04:40.429+0000 7f3621b45d80 -1 journal do_read_entry(8192): bad header magic 2020-11-20T12:04:40.435 INFO:tasks.ceph.osd.1.smithi189.stderr:2020-11-20T12:04:40.433+0000 7f3621b45d80 -1 osd.1 0 log_to_monitors {default=true} 2020-11-20T12:04:40.851 INFO:teuthology.orchestra.run.smithi189:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2020-11-20T12:04:40.959 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.957+0000 7fc68507e700 1 Processor -- start 2020-11-20T12:04:40.959 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.957+0000 7fc68507e700 1 -- start start 2020-11-20T12:04:40.963 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.957+0000 7fc68507e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc680062600 0x7fc68013a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:40.964 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.957+0000 7fc68507e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc680066fd0 0x7fc680060a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:40.964 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.957+0000 7fc68507e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6800a9bb0 con 0x7fc680062220 2020-11-20T12:04:40.969 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.957+0000 7fc68507e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc680067460 con 0x7fc680062600 2020-11-20T12:04:40.969 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.957+0000 7fc68507e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc68013ab40 con 0x7fc680066fd0 2020-11-20T12:04:40.969 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.957+0000 7fc677fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc680062600 0x7fc68013a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:40.970 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.957+0000 7fc67ffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc680062220 0x7fc680137750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49410/0 (socket says 172.21.15.189:49410) 2020-11-20T12:04:40.970 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.957+0000 7fc67ffff700 1 -- 172.21.15.189:0/3560588130 learned_addr learned my addr 172.21.15.189:0/3560588130 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:04:40.970 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.957+0000 7fc68487d700 1 --2- 172.21.15.189:0/3560588130 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc680066fd0 0x7fc680060a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:40.970 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.961+0000 7fc677fff700 1 -- 172.21.15.189:0/3560588130 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc680066fd0 msgr2=0x7fc680060a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:40.971 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.961+0000 7fc677fff700 1 --2- 172.21.15.189:0/3560588130 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc680066fd0 0x7fc680060a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:40.971 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.961+0000 7fc677fff700 1 -- 172.21.15.189:0/3560588130 >> v1:172.21.15.189:6789/0 conn(0x7fc680062220 legacy=0x7fc680137750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:40.971 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.961+0000 7fc677fff700 1 -- 172.21.15.189:0/3560588130 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc68013ade0 con 0x7fc680062600 2020-11-20T12:04:40.971 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.961+0000 7fc677fff700 1 --2- 172.21.15.189:0/3560588130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc680062600 0x7fc68013a4a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fc67000b9a0 tx=0x7fc670006df0).ready entity=mon.1 client_cookie=3f5b5f609238f954 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:40.972 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.961+0000 7fc67f7fe700 1 -- 172.21.15.189:0/3560588130 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc670009840 con 0x7fc680062600 2020-11-20T12:04:40.972 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.961+0000 7fc67f7fe700 1 -- 172.21.15.189:0/3560588130 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6700099a0 con 0x7fc680062600 2020-11-20T12:04:40.972 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.961+0000 7fc67f7fe700 1 -- 172.21.15.189:0/3560588130 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc670009d20 con 0x7fc680062600 2020-11-20T12:04:40.972 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.961+0000 7fc68507e700 1 -- 172.21.15.189:0/3560588130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc680062600 msgr2=0x7fc68013a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:40.972 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.961+0000 7fc68507e700 1 --2- 172.21.15.189:0/3560588130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc680062600 0x7fc68013a4a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fc67000b9a0 tx=0x7fc670006df0).stop 2020-11-20T12:04:40.973 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.961+0000 7fc68507e700 1 -- 172.21.15.189:0/3560588130 shutdown_connections 2020-11-20T12:04:40.973 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.961+0000 7fc68507e700 1 --2- 172.21.15.189:0/3560588130 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc680066fd0 0x7fc680060a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:40.973 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.961+0000 7fc68507e700 1 --2- 172.21.15.189:0/3560588130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc680062600 0x7fc68013a4a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:40.973 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.961+0000 7fc68507e700 1 -- 172.21.15.189:0/3560588130 >> 172.21.15.189:0/3560588130 conn(0x7fc680065fe0 msgr2=0x7fc680061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:40.973 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.961+0000 7fc68507e700 1 -- 172.21.15.189:0/3560588130 shutdown_connections 2020-11-20T12:04:40.974 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.961+0000 7fc68507e700 1 -- 172.21.15.189:0/3560588130 wait complete. 2020-11-20T12:04:40.974 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.961+0000 7fc68507e700 1 Processor -- start 2020-11-20T12:04:40.974 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.961+0000 7fc68507e700 1 -- start start 2020-11-20T12:04:40.976 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.965+0000 7fc68507e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc680062220 0x7fc68013a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:40.977 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.965+0000 7fc68507e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc680062600 0x7fc6800a2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:40.977 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.965+0000 7fc68507e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc680066fd0 0x7fc6800a4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:40.977 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.965+0000 7fc68507e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6800a2e80 con 0x7fc680062600 2020-11-20T12:04:40.977 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.965+0000 7fc68507e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6800a4660 con 0x7fc680066fd0 2020-11-20T12:04:40.977 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.965+0000 7fc68507e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6800a48b0 con 0x7fc680062220 2020-11-20T12:04:40.978 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.965+0000 7fc68487d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc680066fd0 0x7fc6800a4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:40.978 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.965+0000 7fc68487d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc680066fd0 0x7fc6800a4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47794/0 (socket says 172.21.15.189:47794) 2020-11-20T12:04:40.978 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.965+0000 7fc68487d700 1 -- 172.21.15.189:0/1371874669 learned_addr learned my addr 172.21.15.189:0/1371874669 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:40.978 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.965+0000 7fc67ffff700 1 --2- 172.21.15.189:0/1371874669 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc680062220 0x7fc68013a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:40.979 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.965+0000 7fc677fff700 1 --2- 172.21.15.189:0/1371874669 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc680062600 0x7fc6800a2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:40.979 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.965+0000 7fc68487d700 1 -- 172.21.15.189:0/1371874669 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc680062220 msgr2=0x7fc68013a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:40.979 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.965+0000 7fc68487d700 1 --2- 172.21.15.189:0/1371874669 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc680062220 0x7fc68013a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:40.979 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.965+0000 7fc68487d700 1 -- 172.21.15.189:0/1371874669 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc680062600 msgr2=0x7fc6800a2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:40.979 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.965+0000 7fc68487d700 1 --2- 172.21.15.189:0/1371874669 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc680062600 0x7fc6800a2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:40.980 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.965+0000 7fc68487d700 1 -- 172.21.15.189:0/1371874669 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6800a2250 con 0x7fc680066fd0 2020-11-20T12:04:40.980 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.965+0000 7fc68487d700 1 --2- 172.21.15.189:0/1371874669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc680066fd0 0x7fc6800a4170 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fc67800ed20 tx=0x7fc6780091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:40.980 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.965+0000 7fc67cff9700 1 -- 172.21.15.189:0/1371874669 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67800eee0 con 0x7fc680066fd0 2020-11-20T12:04:40.980 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.965+0000 7fc67cff9700 1 -- 172.21.15.189:0/1371874669 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc67800cb00 con 0x7fc680066fd0 2020-11-20T12:04:40.980 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.965+0000 7fc68507e700 1 -- 172.21.15.189:0/1371874669 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6800a24f0 con 0x7fc680066fd0 2020-11-20T12:04:40.980 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.965+0000 7fc67cff9700 1 -- 172.21.15.189:0/1371874669 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67800cea0 con 0x7fc680066fd0 2020-11-20T12:04:40.981 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.965+0000 7fc68507e700 1 -- 172.21.15.189:0/1371874669 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc68013e760 con 0x7fc680066fd0 2020-11-20T12:04:40.981 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.965+0000 7fc67cff9700 1 -- 172.21.15.189:0/1371874669 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fc678022070 con 0x7fc680066fd0 2020-11-20T12:04:40.981 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.965+0000 7fc67cff9700 1 -- 172.21.15.189:0/1371874669 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1674+0+0 (secure 0 0 0) 0x7fc67801bb00 con 0x7fc680066fd0 2020-11-20T12:04:40.981 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.965+0000 7fc68507e700 1 -- 172.21.15.189:0/1371874669 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6800a2680 con 0x7fc680066fd0 2020-11-20T12:04:40.982 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:40.969+0000 7fc67cff9700 1 -- 172.21.15.189:0/1371874669 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7fc67803a880 con 0x7fc680066fd0 2020-11-20T12:04:41.041 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:41.041+0000 7fc68507e700 1 -- 172.21.15.189:0/1371874669 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6800663c0 con 0x7fc680066fd0 2020-11-20T12:04:41.042 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:41.041+0000 7fc67cff9700 1 -- 172.21.15.189:0/1371874669 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7001 (secure 0 0 0) 0x7fc678029030 con 0x7fc680066fd0 2020-11-20T12:04:41.042 INFO:teuthology.misc.health.smithi189.stdout: 2020-11-20T12:04:41.042 INFO:teuthology.misc.health.smithi189.stdout:{"epoch":11,"fsid":"cda18d0e-64b3-41bb-964a-084069c3e648","created":"2020-11-20T12:04:37.538706+0000","modified":"2020-11-20T12:04:41.001507+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"c81214e1-4f84-4aaa-8ceb-99c398b2a89a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d2f3ec7d-d403-484f-b5ed-b2e6b079d63d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e6cf4790-000c-43eb-8844-175f04c60503","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ba49dfac-f2b9-476e-853d-42f05e21a77a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ca7b438-38b2-4689-8b18-fb5a8ddf6aee","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6a45fca7-83d3-44fd-a1d4-b01e4201a101","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c4b1b9ed-61f8-4677-9755-dff59a30e447","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4608e4ed-cc40-4c5c-ba8b-3779bbc62f65","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:04:41.043 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:41.041+0000 7fc68507e700 1 -- 172.21.15.189:0/1371874669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc680066fd0 msgr2=0x7fc6800a4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:41.044 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:41.041+0000 7fc68507e700 1 --2- 172.21.15.189:0/1371874669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc680066fd0 0x7fc6800a4170 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fc67800ed20 tx=0x7fc6780091d0).stop 2020-11-20T12:04:41.044 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:41.041+0000 7fc68507e700 1 -- 172.21.15.189:0/1371874669 shutdown_connections 2020-11-20T12:04:41.044 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:41.041+0000 7fc68507e700 1 --2- 172.21.15.189:0/1371874669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc680066fd0 0x7fc6800a4170 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:41.044 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:41.041+0000 7fc68507e700 1 --2- 172.21.15.189:0/1371874669 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc680062220 0x7fc68013a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:41.044 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:41.041+0000 7fc68507e700 1 --2- 172.21.15.189:0/1371874669 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc680062600 0x7fc6800a2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:41.045 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:41.041+0000 7fc68507e700 1 -- 172.21.15.189:0/1371874669 >> 172.21.15.189:0/1371874669 conn(0x7fc680065fe0 msgr2=0x7fc6800a7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:41.045 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:41.041+0000 7fc68507e700 1 -- 172.21.15.189:0/1371874669 shutdown_connections 2020-11-20T12:04:41.045 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:41.041+0000 7fc68507e700 1 -- 172.21.15.189:0/1371874669 wait complete. 2020-11-20T12:04:41.053 DEBUG:teuthology.misc:0 of 8 OSDs are up 2020-11-20T12:04:42.011 INFO:tasks.ceph.osd.3.smithi189.stderr:2020-11-20T12:04:42.009+0000 7f07db952700 -1 osd.3 0 waiting for initial osdmap 2020-11-20T12:04:42.012 INFO:tasks.ceph.osd.5.smithi047.stderr:2020-11-20T12:04:42.009+0000 7fe3083c8700 -1 osd.5 0 waiting for initial osdmap 2020-11-20T12:04:42.020 INFO:tasks.ceph.osd.2.smithi189.stderr:2020-11-20T12:04:42.017+0000 7f82aa286700 -1 osd.2 0 waiting for initial osdmap 2020-11-20T12:04:42.020 INFO:tasks.ceph.osd.1.smithi189.stderr:2020-11-20T12:04:42.017+0000 7f361a011700 -1 osd.1 0 waiting for initial osdmap 2020-11-20T12:04:42.021 INFO:tasks.ceph.osd.0.smithi189.stderr:2020-11-20T12:04:42.017+0000 7f317172d700 -1 osd.0 0 waiting for initial osdmap 2020-11-20T12:04:42.021 INFO:tasks.ceph.osd.7.smithi047.stderr:2020-11-20T12:04:42.013+0000 7f895c444700 -1 osd.7 0 waiting for initial osdmap 2020-11-20T12:04:42.022 INFO:tasks.ceph.osd.4.smithi047.stderr:2020-11-20T12:04:42.013+0000 7f9419e42700 -1 osd.4 0 waiting for initial osdmap 2020-11-20T12:04:42.022 INFO:tasks.ceph.osd.6.smithi047.stderr:2020-11-20T12:04:42.013+0000 7f75d5b1d700 -1 osd.6 0 waiting for initial osdmap 2020-11-20T12:04:47.862 INFO:teuthology.orchestra.run.smithi189:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2020-11-20T12:04:47.996 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f1759e700 1 Processor -- start 2020-11-20T12:04:47.999 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f1759e700 1 -- start start 2020-11-20T12:04:47.999 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f1759e700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f9f180aeb80 0x7f9f180aadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:47.999 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f1759e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9f18064830 0x7f9f180a9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:47.999 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f1759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f180a9b80 con 0x7f9f18064e80 2020-11-20T12:04:48.000 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f1759e700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f180ab2c0 con 0x7f9f180aeb80 2020-11-20T12:04:48.000 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f1759e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f180af090 con 0x7f9f18064830 2020-11-20T12:04:48.000 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f1659c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9f18064e80 0x7f9f180ab670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49680/0 (socket says 172.21.15.189:49680) 2020-11-20T12:04:48.000 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f1659c700 1 -- 172.21.15.189:0/1481031827 learned_addr learned my addr 172.21.15.189:0/1481031827 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:04:48.000 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f16d9d700 1 --2- 172.21.15.189:0/1481031827 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9f18064830 0x7f9f180a9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:48.001 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f15d9b700 1 --2- 172.21.15.189:0/1481031827 >> v2:172.21.15.47:3300/0 conn(0x7f9f180aeb80 0x7f9f180aadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:48.001 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f1559a700 1 -- 172.21.15.189:0/1481031827 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1357105987 0 0) 0x7f9f180a9b80 con 0x7f9f18064e80 2020-11-20T12:04:48.001 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f1559a700 1 -- 172.21.15.189:0/1481031827 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9efc003240 con 0x7f9f18064e80 2020-11-20T12:04:48.005 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f1559a700 1 -- 172.21.15.189:0/1481031827 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 816363441 0 0) 0x7f9efc003240 con 0x7f9f18064e80 2020-11-20T12:04:48.005 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f1559a700 1 -- 172.21.15.189:0/1481031827 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9f18064830 msgr2=0x7f9f180a9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:48.005 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f1559a700 1 --2- 172.21.15.189:0/1481031827 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9f18064830 0x7f9f180a9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:48.005 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f1559a700 1 -- 172.21.15.189:0/1481031827 >> v2:172.21.15.47:3300/0 conn(0x7f9f180aeb80 msgr2=0x7f9f180aadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:48.005 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f1559a700 1 --2- 172.21.15.189:0/1481031827 >> v2:172.21.15.47:3300/0 conn(0x7f9f180aeb80 0x7f9f180aadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:48.006 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f1559a700 1 -- 172.21.15.189:0/1481031827 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f180af330 con 0x7f9f18064e80 2020-11-20T12:04:48.006 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f1559a700 1 -- 172.21.15.189:0/1481031827 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f9f080014e0 con 0x7f9f18064e80 2020-11-20T12:04:48.006 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f16d9d700 1 --2- 172.21.15.189:0/1481031827 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9f18064830 0x7f9f180a9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:04:48.006 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f15d9b700 1 --2- 172.21.15.189:0/1481031827 >> v2:172.21.15.47:3300/0 conn(0x7f9f180aeb80 0x7f9f180aadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:04:48.007 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f1559a700 1 -- 172.21.15.189:0/1481031827 >> v1:172.21.15.189:6789/0 conn(0x7f9f18064e80 legacy=0x7f9f180ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:48.007 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f1559a700 1 --2- 172.21.15.189:0/1481031827 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9f18064830 0x7f9efc003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:48.007 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f1559a700 1 --2- 172.21.15.189:0/1481031827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9efc003c20 0x7f9efc0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:48.007 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f1559a700 1 --2- 172.21.15.189:0/1481031827 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9efc0049d0 0x7f9efc004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:48.007 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f1559a700 1 -- 172.21.15.189:0/1481031827 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f180ab2c0 con 0x7f9efc0049d0 2020-11-20T12:04:48.008 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f1559a700 1 -- 172.21.15.189:0/1481031827 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f180af090 con 0x7f9efc003c20 2020-11-20T12:04:48.008 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f1559a700 1 -- 172.21.15.189:0/1481031827 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc004170 con 0x7f9f18064830 2020-11-20T12:04:48.008 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f15d9b700 1 --2- 172.21.15.189:0/1481031827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9efc003c20 0x7f9efc0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:48.008 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f1659c700 1 --2- 172.21.15.189:0/1481031827 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9f18064830 0x7f9efc003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:48.008 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.993+0000 7f9f16d9d700 1 --2- 172.21.15.189:0/1481031827 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9efc0049d0 0x7f9efc004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:48.009 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.997+0000 7f9f15d9b700 1 -- 172.21.15.189:0/1481031827 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9f18064830 msgr2=0x7f9efc003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:48.009 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.997+0000 7f9f15d9b700 1 --2- 172.21.15.189:0/1481031827 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9f18064830 0x7f9efc003730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:48.009 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.997+0000 7f9f15d9b700 1 -- 172.21.15.189:0/1481031827 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9efc0049d0 msgr2=0x7f9efc004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:48.009 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.997+0000 7f9f15d9b700 1 --2- 172.21.15.189:0/1481031827 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9efc0049d0 0x7f9efc004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:48.009 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.997+0000 7f9f15d9b700 1 -- 172.21.15.189:0/1481031827 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9efc003240 con 0x7f9efc003c20 2020-11-20T12:04:48.010 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.997+0000 7f9f16d9d700 1 --2- 172.21.15.189:0/1481031827 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9efc0049d0 0x7f9efc004e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:48.010 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.997+0000 7f9f1659c700 1 --2- 172.21.15.189:0/1481031827 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9f18064830 0x7f9efc003730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:48.010 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.997+0000 7f9f15d9b700 1 --2- 172.21.15.189:0/1481031827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9efc003c20 0x7f9efc0044e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f9f0000be80 tx=0x7f9f00006df0).ready entity=mon.1 client_cookie=b2ce3edab9cb00e8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:48.010 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.997+0000 7f9f1559a700 1 -- 172.21.15.189:0/1481031827 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0000b590 con 0x7f9efc003c20 2020-11-20T12:04:48.010 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.997+0000 7f9f1559a700 1 -- 172.21.15.189:0/1481031827 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0000edc0 con 0x7f9efc003c20 2020-11-20T12:04:48.011 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.997+0000 7f9f1559a700 1 -- 172.21.15.189:0/1481031827 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f00012600 con 0x7f9efc003c20 2020-11-20T12:04:48.011 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.997+0000 7f9f1759e700 1 -- 172.21.15.189:0/1481031827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9efc003c20 msgr2=0x7f9efc0044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:48.011 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.997+0000 7f9f1759e700 1 --2- 172.21.15.189:0/1481031827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9efc003c20 0x7f9efc0044e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f9f0000be80 tx=0x7f9f00006df0).stop 2020-11-20T12:04:48.011 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.997+0000 7f9f16d9d700 1 -- 172.21.15.189:0/1481031827 reap_dead start 2020-11-20T12:04:48.011 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.997+0000 7f9f1759e700 1 -- 172.21.15.189:0/1481031827 shutdown_connections 2020-11-20T12:04:48.012 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.997+0000 7f9f1759e700 1 -- 172.21.15.189:0/1481031827 >> 172.21.15.189:0/1481031827 conn(0x7f9f18062080 msgr2=0x7f9f18062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:48.012 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.997+0000 7f9f1759e700 1 -- 172.21.15.189:0/1481031827 shutdown_connections 2020-11-20T12:04:48.012 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:47.997+0000 7f9f1759e700 1 -- 172.21.15.189:0/1481031827 wait complete. 2020-11-20T12:04:48.012 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.001+0000 7f9f1759e700 1 Processor -- start 2020-11-20T12:04:48.012 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.001+0000 7f9f1759e700 1 -- start start 2020-11-20T12:04:48.013 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.001+0000 7f9f1759e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f180a4680 0x7f9f18141fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:48.013 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.001+0000 7f9f1759e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f180a7640 0x7f9f181424c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:48.013 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.001+0000 7f9f1759e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9f1813fe30 0x7f9f18140290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:48.013 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.001+0000 7f9f1759e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f180a27c0 con 0x7f9f180a4680 2020-11-20T12:04:48.013 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.001+0000 7f9f1759e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f18141470 con 0x7f9f180a7640 2020-11-20T12:04:48.014 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.001+0000 7f9f1759e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f18141710 con 0x7f9f1813fe30 2020-11-20T12:04:48.014 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.001+0000 7f9f15d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f180a7640 0x7f9f181424c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:48.014 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.001+0000 7f9f15d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f180a7640 0x7f9f181424c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48070/0 (socket says 172.21.15.189:48070) 2020-11-20T12:04:48.014 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.001+0000 7f9f15d9b700 1 -- 172.21.15.189:0/1156357087 learned_addr learned my addr 172.21.15.189:0/1156357087 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:48.016 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.001+0000 7f9f1659c700 1 --2- 172.21.15.189:0/1156357087 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f180a4680 0x7f9f18141fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:48.017 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.001+0000 7f9f16d9d700 1 --2- 172.21.15.189:0/1156357087 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9f1813fe30 0x7f9f18140290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:48.017 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.001+0000 7f9f15d9b700 1 -- 172.21.15.189:0/1156357087 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9f1813fe30 msgr2=0x7f9f18140290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:48.017 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.001+0000 7f9f15d9b700 1 --2- 172.21.15.189:0/1156357087 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9f1813fe30 0x7f9f18140290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:48.017 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.001+0000 7f9f15d9b700 1 -- 172.21.15.189:0/1156357087 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f180a4680 msgr2=0x7f9f18141fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:48.018 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.001+0000 7f9f15d9b700 1 --2- 172.21.15.189:0/1156357087 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f180a4680 0x7f9f18141fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:48.018 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.001+0000 7f9f15d9b700 1 -- 172.21.15.189:0/1156357087 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f181419b0 con 0x7f9f180a7640 2020-11-20T12:04:48.018 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.001+0000 7f9f16d9d700 1 --2- 172.21.15.189:0/1156357087 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9f1813fe30 0x7f9f18140290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:04:48.018 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.001+0000 7f9f15d9b700 1 --2- 172.21.15.189:0/1156357087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f180a7640 0x7f9f181424c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f9f00013000 tx=0x7f9f00019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:48.018 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.001+0000 7f9f0e7fc700 1 -- 172.21.15.189:0/1156357087 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0001d620 con 0x7f9f180a7640 2020-11-20T12:04:48.019 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.001+0000 7f9f0e7fc700 1 -- 172.21.15.189:0/1156357087 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0001d7a0 con 0x7f9f180a7640 2020-11-20T12:04:48.019 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.001+0000 7f9f0e7fc700 1 -- 172.21.15.189:0/1156357087 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0001d950 con 0x7f9f180a7640 2020-11-20T12:04:48.019 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.001+0000 7f9f1759e700 1 -- 172.21.15.189:0/1156357087 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f180a4a60 con 0x7f9f180a7640 2020-11-20T12:04:48.019 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.001+0000 7f9f1759e700 1 -- 172.21.15.189:0/1156357087 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f180a52e0 con 0x7f9f180a7640 2020-11-20T12:04:48.019 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.005+0000 7f9f1759e700 1 -- 172.21.15.189:0/1156357087 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f180a5d80 con 0x7f9f180a7640 2020-11-20T12:04:48.020 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.005+0000 7f9f0e7fc700 1 -- 172.21.15.189:0/1156357087 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f9f0002d070 con 0x7f9f180a7640 2020-11-20T12:04:48.020 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.005+0000 7f9f0e7fc700 1 --2- 172.21.15.189:0/1156357087 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9ee80557c0 0x7f9ee8057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:48.020 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.005+0000 7f9f0e7fc700 1 -- 172.21.15.189:0/1156357087 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4900+0+0 (secure 0 0 0) 0x7f9f00070250 con 0x7f9f180a7640 2020-11-20T12:04:48.020 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.005+0000 7f9f1659c700 1 --2- 172.21.15.189:0/1156357087 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9ee80557c0 0x7f9ee8057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:48.020 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.005+0000 7f9f1659c700 1 --2- 172.21.15.189:0/1156357087 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9ee80557c0 0x7f9ee8057c80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f9f08008a10 tx=0x7f9f0800a040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:48.021 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.009+0000 7f9f0e7fc700 1 -- 172.21.15.189:0/1156357087 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9f00046580 con 0x7f9f180a7640 2020-11-20T12:04:48.166 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.165+0000 7f9f1759e700 1 -- 172.21.15.189:0/1156357087 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f180446d0 con 0x7f9f180a7640 2020-11-20T12:04:48.167 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.165+0000 7f9f0e7fc700 1 -- 172.21.15.189:0/1156357087 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+12152 (secure 0 0 0) 0x7f9f00049ba0 con 0x7f9f180a7640 2020-11-20T12:04:48.168 INFO:teuthology.misc.health.smithi189.stdout: 2020-11-20T12:04:48.168 INFO:teuthology.misc.health.smithi189.stdout:{"epoch":15,"fsid":"cda18d0e-64b3-41bb-964a-084069c3e648","created":"2020-11-20T12:04:37.538706+0000","modified":"2020-11-20T12:04:45.054090+0000","last_up_change":"2020-11-20T12:04:43.006631+0000","last_in_change":"2020-11-20T12:04:43.006631+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:41.395395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"c81214e1-4f84-4aaa-8ceb-99c398b2a89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6811","nonce":11760},{"type":"v1","addr":"172.21.15.189:6814","nonce":11760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":11760},{"type":"v1","addr":"172.21.15.189:6818","nonce":11760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":11760},{"type":"v1","addr":"172.21.15.189:6828","nonce":11760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6821","nonce":11760},{"type":"v1","addr":"172.21.15.189:6824","nonce":11760}]},"public_addr":"172.21.15.189:6814/11760","cluster_addr":"172.21.15.189:6818/11760","heartbeat_back_addr":"172.21.15.189:6828/11760","heartbeat_front_addr":"172.21.15.189:6824/11760","state":["exists","up"]},{"osd":1,"uuid":"d2f3ec7d-d403-484f-b5ed-b2e6b079d63d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6805","nonce":11761},{"type":"v1","addr":"172.21.15.189:6807","nonce":11761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":11761},{"type":"v1","addr":"172.21.15.189:6812","nonce":11761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6817","nonce":11761},{"type":"v1","addr":"172.21.15.189:6820","nonce":11761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6813","nonce":11761},{"type":"v1","addr":"172.21.15.189:6815","nonce":11761}]},"public_addr":"172.21.15.189:6807/11761","cluster_addr":"172.21.15.189:6812/11761","heartbeat_back_addr":"172.21.15.189:6820/11761","heartbeat_front_addr":"172.21.15.189:6815/11761","state":["exists","up"]},{"osd":2,"uuid":"e6cf4790-000c-43eb-8844-175f04c60503","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6819","nonce":11751},{"type":"v1","addr":"172.21.15.189:6822","nonce":11751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":11751},{"type":"v1","addr":"172.21.15.189:6825","nonce":11751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":11751},{"type":"v1","addr":"172.21.15.189:6831","nonce":11751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6827","nonce":11751},{"type":"v1","addr":"172.21.15.189:6829","nonce":11751}]},"public_addr":"172.21.15.189:6822/11751","cluster_addr":"172.21.15.189:6825/11751","heartbeat_back_addr":"172.21.15.189:6831/11751","heartbeat_front_addr":"172.21.15.189:6829/11751","state":["exists","up"]},{"osd":3,"uuid":"ba49dfac-f2b9-476e-853d-42f05e21a77a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":11762},{"type":"v1","addr":"172.21.15.189:6801","nonce":11762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":11762},{"type":"v1","addr":"172.21.15.189:6803","nonce":11762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":11762},{"type":"v1","addr":"172.21.15.189:6809","nonce":11762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":11762},{"type":"v1","addr":"172.21.15.189:6806","nonce":11762}]},"public_addr":"172.21.15.189:6801/11762","cluster_addr":"172.21.15.189:6803/11762","heartbeat_back_addr":"172.21.15.189:6809/11762","heartbeat_front_addr":"172.21.15.189:6806/11762","state":["exists","up"]},{"osd":4,"uuid":"3ca7b438-38b2-4689-8b18-fb5a8ddf6aee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":12114},{"type":"v1","addr":"172.21.15.47:6825","nonce":12114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":12114},{"type":"v1","addr":"172.21.15.47:6827","nonce":12114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":12114},{"type":"v1","addr":"172.21.15.47:6831","nonce":12114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":12114},{"type":"v1","addr":"172.21.15.47:6829","nonce":12114}]},"public_addr":"172.21.15.47:6825/12114","cluster_addr":"172.21.15.47:6827/12114","heartbeat_back_addr":"172.21.15.47:6831/12114","heartbeat_front_addr":"172.21.15.47:6829/12114","state":["exists","up"]},{"osd":5,"uuid":"6a45fca7-83d3-44fd-a1d4-b01e4201a101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":12111},{"type":"v1","addr":"172.21.15.47:6801","nonce":12111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":12111},{"type":"v1","addr":"172.21.15.47:6803","nonce":12111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":12111},{"type":"v1","addr":"172.21.15.47:6807","nonce":12111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":12111},{"type":"v1","addr":"172.21.15.47:6805","nonce":12111}]},"public_addr":"172.21.15.47:6801/12111","cluster_addr":"172.21.15.47:6803/12111","heartbeat_back_addr":"172.21.15.47:6807/12111","heartbeat_front_addr":"172.21.15.47:6805/12111","state":["exists","up"]},{"osd":6,"uuid":"c4b1b9ed-61f8-4677-9755-dff59a30e447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":12113},{"type":"v1","addr":"172.21.15.47:6817","nonce":12113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":12113},{"type":"v1","addr":"172.21.15.47:6819","nonce":12113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":12113},{"type":"v1","addr":"172.21.15.47:6823","nonce":12113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":12113},{"type":"v1","addr":"172.21.15.47:6821","nonce":12113}]},"public_addr":"172.21.15.47:6817/12113","cluster_addr":"172.21.15.47:6819/12113","heartbeat_back_addr":"172.21.15.47:6823/12113","heartbeat_front_addr":"172.21.15.47:6821/12113","state":["exists","up"]},{"osd":7,"uuid":"4608e4ed-cc40-4c5c-ba8b-3779bbc62f65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":12112},{"type":"v1","addr":"172.21.15.47:6809","nonce":12112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":12112},{"type":"v1","addr":"172.21.15.47:6811","nonce":12112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":12112},{"type":"v1","addr":"172.21.15.47:6815","nonce":12112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":12112},{"type":"v1","addr":"172.21.15.47:6813","nonce":12112}]},"public_addr":"172.21.15.47:6809/12112","cluster_addr":"172.21.15.47:6811/12112","heartbeat_back_addr":"172.21.15.47:6815/12112","heartbeat_front_addr":"172.21.15.47:6813/12112","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.435265+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.472776+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.388370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.417366+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.102392+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.116529+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.167430+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.148139+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:04:48.169 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.169+0000 7f9f1759e700 1 -- 172.21.15.189:0/1156357087 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9ee80557c0 msgr2=0x7f9ee8057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:48.170 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.169+0000 7f9f1759e700 1 --2- 172.21.15.189:0/1156357087 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9ee80557c0 0x7f9ee8057c80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f9f08008a10 tx=0x7f9f0800a040).stop 2020-11-20T12:04:48.170 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.169+0000 7f9f1759e700 1 -- 172.21.15.189:0/1156357087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f180a7640 msgr2=0x7f9f181424c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:48.170 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.169+0000 7f9f1759e700 1 --2- 172.21.15.189:0/1156357087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f180a7640 0x7f9f181424c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f9f00013000 tx=0x7f9f00019000).stop 2020-11-20T12:04:48.170 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.169+0000 7f9f1759e700 1 -- 172.21.15.189:0/1156357087 shutdown_connections 2020-11-20T12:04:48.171 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.169+0000 7f9f1759e700 1 --2- 172.21.15.189:0/1156357087 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9ee80557c0 0x7f9ee8057c80 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:48.171 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.169+0000 7f9f1759e700 1 --2- 172.21.15.189:0/1156357087 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f180a4680 0x7f9f18141fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:48.172 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.169+0000 7f9f1759e700 1 --2- 172.21.15.189:0/1156357087 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9f1813fe30 0x7f9f18140290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:48.172 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.169+0000 7f9f1759e700 1 --2- 172.21.15.189:0/1156357087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f180a7640 0x7f9f181424c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:48.172 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.169+0000 7f9f1759e700 1 -- 172.21.15.189:0/1156357087 >> 172.21.15.189:0/1156357087 conn(0x7f9f18062080 msgr2=0x7f9f1805ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:48.172 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.169+0000 7f9f1759e700 1 -- 172.21.15.189:0/1156357087 shutdown_connections 2020-11-20T12:04:48.173 INFO:teuthology.misc.health.smithi189.stderr:2020-11-20T12:04:48.169+0000 7f9f1759e700 1 -- 172.21.15.189:0/1156357087 wait complete. 2020-11-20T12:04:48.181 DEBUG:teuthology.misc:8 of 8 OSDs are up 2020-11-20T12:04:48.182 INFO:tasks.ceph:Creating RBD pool 2020-11-20T12:04:48.182 INFO:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd pool create rbd 8 2020-11-20T12:04:48.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.313+0000 7fe086251700 1 Processor -- start 2020-11-20T12:04:48.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.313+0000 7fe086251700 1 -- start start 2020-11-20T12:04:48.317 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe086251700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe080062930 0x7fe0800b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:48.317 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe086251700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe080066fb0 con 0x7fe0800b17c0 2020-11-20T12:04:48.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe086251700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe080067250 con 0x7fe080066a30 2020-11-20T12:04:48.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe086251700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0800666b0 con 0x7fe080062930 2020-11-20T12:04:48.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe084a4e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe080066a30 0x7fe0801373e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49698/0 (socket says 172.21.15.189:49698) 2020-11-20T12:04:48.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe084a4e700 1 -- 172.21.15.189:0/4094265717 learned_addr learned my addr 172.21.15.189:0/4094265717 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:04:48.320 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe08524f700 1 --2- 172.21.15.189:0/4094265717 >> v2:172.21.15.47:3300/0 conn(0x7fe080062930 0x7fe0800b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:48.320 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe073fff700 1 -- 172.21.15.189:0/4094265717 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1227103564 0 0) 0x7fe080067250 con 0x7fe080066a30 2020-11-20T12:04:48.320 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe073fff700 1 -- 172.21.15.189:0/4094265717 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe064003240 con 0x7fe080066a30 2020-11-20T12:04:48.320 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe073fff700 1 -- 172.21.15.189:0/4094265717 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1248696287 0 0) 0x7fe064003240 con 0x7fe080066a30 2020-11-20T12:04:48.320 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe073fff700 1 -- 172.21.15.189:0/4094265717 >> v2:172.21.15.47:3300/0 conn(0x7fe080062930 msgr2=0x7fe0800b12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:48.321 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe073fff700 1 --2- 172.21.15.189:0/4094265717 >> v2:172.21.15.47:3300/0 conn(0x7fe080062930 0x7fe0800b12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:48.321 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe073fff700 1 -- 172.21.15.189:0/4094265717 >> v1:172.21.15.47:6789/0 conn(0x7fe0800b17c0 legacy=0x7fe080137af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:48.321 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe073fff700 1 -- 172.21.15.189:0/4094265717 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0800b0e30 con 0x7fe080066a30 2020-11-20T12:04:48.321 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe073fff700 1 -- 172.21.15.189:0/4094265717 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7fe07c0014e0 con 0x7fe080066a30 2020-11-20T12:04:48.321 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe073fff700 1 -- 172.21.15.189:0/4094265717 >> v1:172.21.15.189:6789/0 conn(0x7fe080066a30 legacy=0x7fe0801373e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:48.322 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe073fff700 1 --2- 172.21.15.189:0/4094265717 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe064003730 0x7fe064003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:48.322 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe073fff700 1 --2- 172.21.15.189:0/4094265717 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0640044e0 0x7fe064004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:48.322 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe073fff700 1 --2- 172.21.15.189:0/4094265717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe064004e50 0x7fe06400b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:48.322 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe073fff700 1 -- 172.21.15.189:0/4094265717 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe064004260 con 0x7fe0640044e0 2020-11-20T12:04:48.322 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe073fff700 1 -- 172.21.15.189:0/4094265717 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06400bc20 con 0x7fe064004e50 2020-11-20T12:04:48.323 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe073fff700 1 -- 172.21.15.189:0/4094265717 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe06400bd60 con 0x7fe064003730 2020-11-20T12:04:48.323 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe085a50700 1 --2- 172.21.15.189:0/4094265717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe064004e50 0x7fe06400b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:48.323 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe08524f700 1 --2- 172.21.15.189:0/4094265717 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe064003730 0x7fe064003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:48.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe084a4e700 1 --2- 172.21.15.189:0/4094265717 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0640044e0 0x7fe064004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:48.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe085a50700 1 -- 172.21.15.189:0/4094265717 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe064003730 msgr2=0x7fe064003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:48.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe085a50700 1 --2- 172.21.15.189:0/4094265717 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe064003730 0x7fe064003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:48.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe085a50700 1 -- 172.21.15.189:0/4094265717 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0640044e0 msgr2=0x7fe064004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:48.328 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe085a50700 1 --2- 172.21.15.189:0/4094265717 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0640044e0 0x7fe064004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:48.328 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe085a50700 1 -- 172.21.15.189:0/4094265717 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe064003240 con 0x7fe064004e50 2020-11-20T12:04:48.328 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe084a4e700 1 --2- 172.21.15.189:0/4094265717 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0640044e0 0x7fe064004960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:48.328 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe085a50700 1 -- 172.21.15.189:0/4094265717 reap_dead start 2020-11-20T12:04:48.328 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.317+0000 7fe085a50700 1 --2- 172.21.15.189:0/4094265717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe064004e50 0x7fe06400b730 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fe078001e70 tx=0x7fe078007f10).ready entity=mon.1 client_cookie=59cc5cfd08a9bad7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:48.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.321+0000 7fe073fff700 1 -- 172.21.15.189:0/4094265717 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe078001540 con 0x7fe064004e50 2020-11-20T12:04:48.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.321+0000 7fe073fff700 1 -- 172.21.15.189:0/4094265717 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe07800ca70 con 0x7fe064004e50 2020-11-20T12:04:48.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.321+0000 7fe073fff700 1 -- 172.21.15.189:0/4094265717 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07800cdf0 con 0x7fe064004e50 2020-11-20T12:04:48.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.321+0000 7fe086251700 1 -- 172.21.15.189:0/4094265717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe064004e50 msgr2=0x7fe06400b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:48.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.321+0000 7fe086251700 1 --2- 172.21.15.189:0/4094265717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe064004e50 0x7fe06400b730 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fe078001e70 tx=0x7fe078007f10).stop 2020-11-20T12:04:48.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.321+0000 7fe086251700 1 -- 172.21.15.189:0/4094265717 shutdown_connections 2020-11-20T12:04:48.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.321+0000 7fe086251700 1 --2- 172.21.15.189:0/4094265717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe064004e50 0x7fe06400b730 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:48.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.321+0000 7fe086251700 1 -- 172.21.15.189:0/4094265717 >> 172.21.15.189:0/4094265717 conn(0x7fe080065080 msgr2=0x7fe080062080 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:48.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.321+0000 7fe086251700 1 -- 172.21.15.189:0/4094265717 shutdown_connections 2020-11-20T12:04:48.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.321+0000 7fe086251700 1 -- 172.21.15.189:0/4094265717 wait complete. 2020-11-20T12:04:48.331 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.321+0000 7fe086251700 1 Processor -- start 2020-11-20T12:04:48.331 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.321+0000 7fe086251700 1 -- start start 2020-11-20T12:04:48.331 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.325+0000 7fe086251700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe064004e50 0x7fe0800aab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:48.331 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.325+0000 7fe086251700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe08013e0f0 0x7fe08013c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:48.331 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.325+0000 7fe086251700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe08013c590 0x7fe08014d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:48.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.325+0000 7fe086251700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08013e570 con 0x7fe08013c590 2020-11-20T12:04:48.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.325+0000 7fe086251700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08013ca10 con 0x7fe08013e0f0 2020-11-20T12:04:48.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.325+0000 7fe086251700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0800ab1a0 con 0x7fe064004e50 2020-11-20T12:04:48.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.325+0000 7fe084a4e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe08013e0f0 0x7fe08013c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:48.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.325+0000 7fe08524f700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe064004e50 0x7fe0800aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:48.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.325+0000 7fe084a4e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe08013e0f0 0x7fe08013c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48092/0 (socket says 172.21.15.189:48092) 2020-11-20T12:04:48.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.325+0000 7fe084a4e700 1 -- 172.21.15.189:0/4249972719 learned_addr learned my addr 172.21.15.189:0/4249972719 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:48.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.325+0000 7fe085a50700 1 --2- 172.21.15.189:0/4249972719 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe08013c590 0x7fe08014d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:48.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.325+0000 7fe084a4e700 1 -- 172.21.15.189:0/4249972719 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe064004e50 msgr2=0x7fe0800aab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:48.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.325+0000 7fe084a4e700 1 --2- 172.21.15.189:0/4249972719 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe064004e50 0x7fe0800aab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:48.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.325+0000 7fe084a4e700 1 -- 172.21.15.189:0/4249972719 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe08013c590 msgr2=0x7fe08014d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:48.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.325+0000 7fe084a4e700 1 --2- 172.21.15.189:0/4249972719 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe08013c590 0x7fe08014d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:48.336 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.325+0000 7fe084a4e700 1 -- 172.21.15.189:0/4249972719 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe08013b960 con 0x7fe08013e0f0 2020-11-20T12:04:48.336 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.325+0000 7fe084a4e700 1 --2- 172.21.15.189:0/4249972719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe08013e0f0 0x7fe08013c0a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fe07c0011c0 tx=0x7fe07c0014b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:48.337 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.325+0000 7fe0717fa700 1 -- 172.21.15.189:0/4249972719 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c005240 con 0x7fe08013e0f0 2020-11-20T12:04:48.337 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.325+0000 7fe0717fa700 1 -- 172.21.15.189:0/4249972719 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe07c010040 con 0x7fe08013e0f0 2020-11-20T12:04:48.337 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.325+0000 7fe086251700 1 -- 172.21.15.189:0/4249972719 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe08013bc00 con 0x7fe08013e0f0 2020-11-20T12:04:48.337 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.325+0000 7fe0717fa700 1 -- 172.21.15.189:0/4249972719 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c00a520 con 0x7fe08013e0f0 2020-11-20T12:04:48.338 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.325+0000 7fe086251700 1 -- 172.21.15.189:0/4249972719 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0800a9a30 con 0x7fe08013e0f0 2020-11-20T12:04:48.338 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.325+0000 7fe086251700 1 -- 172.21.15.189:0/4249972719 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0800aaff0 con 0x7fe08013e0f0 2020-11-20T12:04:48.338 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.329+0000 7fe0717fa700 1 -- 172.21.15.189:0/4249972719 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fe07c00a890 con 0x7fe08013e0f0 2020-11-20T12:04:48.338 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.329+0000 7fe0717fa700 1 --2- 172.21.15.189:0/4249972719 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe0580557c0 0x7fe058057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:48.338 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.329+0000 7fe0717fa700 1 -- 172.21.15.189:0/4249972719 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4900+0+0 (secure 0 0 0) 0x7fe07c06dcb0 con 0x7fe08013e0f0 2020-11-20T12:04:48.339 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.329+0000 7fe08524f700 1 --2- 172.21.15.189:0/4249972719 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe0580557c0 0x7fe058057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:48.339 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.329+0000 7fe08524f700 1 --2- 172.21.15.189:0/4249972719 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe0580557c0 0x7fe058057c80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fe074006860 tx=0x7fe074006df0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:48.339 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.333+0000 7fe0717fa700 1 -- 172.21.15.189:0/4249972719 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe07c043fe0 con 0x7fe08013e0f0 2020-11-20T12:04:48.492 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:48.489+0000 7fe086251700 1 -- 172.21.15.189:0/4249972719 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7fe0800446d0 con 0x7fe08013e0f0 2020-11-20T12:04:49.485 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.485+0000 7fe0717fa700 1 -- 172.21.15.189:0/4249972719 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v16) v1 ==== 109+0+0 (secure 0 0 0) 0x7fe07c047600 con 0x7fe08013e0f0 2020-11-20T12:04:49.489 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.489+0000 7fe086251700 1 -- 172.21.15.189:0/4249972719 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe0580557c0 msgr2=0x7fe058057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:49.490 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.489+0000 7fe086251700 1 --2- 172.21.15.189:0/4249972719 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe0580557c0 0x7fe058057c80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fe074006860 tx=0x7fe074006df0).stop 2020-11-20T12:04:49.490 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.489+0000 7fe086251700 1 -- 172.21.15.189:0/4249972719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe08013e0f0 msgr2=0x7fe08013c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:49.490 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.489+0000 7fe086251700 1 --2- 172.21.15.189:0/4249972719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe08013e0f0 0x7fe08013c0a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fe07c0011c0 tx=0x7fe07c0014b0).stop 2020-11-20T12:04:49.490 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.489+0000 7fe086251700 1 -- 172.21.15.189:0/4249972719 shutdown_connections 2020-11-20T12:04:49.491 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.489+0000 7fe086251700 1 --2- 172.21.15.189:0/4249972719 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe0580557c0 0x7fe058057c80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:49.491 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.489+0000 7fe086251700 1 --2- 172.21.15.189:0/4249972719 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe08013c590 0x7fe08014d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:49.491 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.489+0000 7fe086251700 1 --2- 172.21.15.189:0/4249972719 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe064004e50 0x7fe0800aab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:49.491 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.489+0000 7fe086251700 1 --2- 172.21.15.189:0/4249972719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe08013e0f0 0x7fe08013c0a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:49.491 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.489+0000 7fe086251700 1 -- 172.21.15.189:0/4249972719 >> 172.21.15.189:0/4249972719 conn(0x7fe080065080 msgr2=0x7fe0800608c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:49.492 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.489+0000 7fe086251700 1 -- 172.21.15.189:0/4249972719 shutdown_connections 2020-11-20T12:04:49.492 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.489+0000 7fe086251700 1 -- 172.21.15.189:0/4249972719 wait complete. 2020-11-20T12:04:49.492 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'rbd' created 2020-11-20T12:04:49.506 INFO:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2020-11-20T12:04:49.640 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5646419700 1 Processor -- start 2020-11-20T12:04:49.641 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5646419700 1 -- start start 2020-11-20T12:04:49.642 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5646419700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5640062790 0x7f564013ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:49.642 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5646419700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f564013bf60 0x7f56400ab030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:49.642 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5646419700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5640138a30 con 0x7f5640062b70 2020-11-20T12:04:49.643 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5646419700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5640135ff0 con 0x7f564013bf60 2020-11-20T12:04:49.643 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5646419700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5640139410 con 0x7f5640062790 2020-11-20T12:04:49.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5644c16700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5640062b70 0x7f56400aa920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49720/0 (socket says 172.21.15.189:49720) 2020-11-20T12:04:49.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5645c18700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f564013bf60 0x7f56400ab030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:49.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5644c16700 1 -- 172.21.15.189:0/1425921284 learned_addr learned my addr 172.21.15.189:0/1425921284 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:04:49.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5645417700 1 --2- 172.21.15.189:0/1425921284 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5640062790 0x7f564013ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:49.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5645c18700 1 -- 172.21.15.189:0/1425921284 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5640062790 msgr2=0x7f564013ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:49.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5645c18700 1 --2- 172.21.15.189:0/1425921284 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5640062790 0x7f564013ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:49.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5645c18700 1 -- 172.21.15.189:0/1425921284 >> v1:172.21.15.189:6789/0 conn(0x7f5640062b70 legacy=0x7f56400aa920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:49.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5645c18700 1 -- 172.21.15.189:0/1425921284 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5640138cd0 con 0x7f564013bf60 2020-11-20T12:04:49.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5645c18700 1 --2- 172.21.15.189:0/1425921284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f564013bf60 0x7f56400ab030 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f5638009cc0 tx=0x7f56380091d0).ready entity=mon.1 client_cookie=1555cac0918d74ae server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:49.648 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5637fff700 1 -- 172.21.15.189:0/1425921284 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563800ba90 con 0x7f564013bf60 2020-11-20T12:04:49.648 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5637fff700 1 -- 172.21.15.189:0/1425921284 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563800bbf0 con 0x7f564013bf60 2020-11-20T12:04:49.648 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5637fff700 1 -- 172.21.15.189:0/1425921284 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5638011400 con 0x7f564013bf60 2020-11-20T12:04:49.649 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5646419700 1 -- 172.21.15.189:0/1425921284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f564013bf60 msgr2=0x7f56400ab030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:49.649 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5646419700 1 --2- 172.21.15.189:0/1425921284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f564013bf60 0x7f56400ab030 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f5638009cc0 tx=0x7f56380091d0).stop 2020-11-20T12:04:49.649 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5646419700 1 -- 172.21.15.189:0/1425921284 shutdown_connections 2020-11-20T12:04:49.649 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5646419700 1 --2- 172.21.15.189:0/1425921284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f564013bf60 0x7f56400ab030 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:49.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5646419700 1 --2- 172.21.15.189:0/1425921284 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5640062790 0x7f564013ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:49.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5646419700 1 -- 172.21.15.189:0/1425921284 >> 172.21.15.189:0/1425921284 conn(0x7f5640065ef0 msgr2=0x7f5640062080 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:49.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5646419700 1 -- 172.21.15.189:0/1425921284 shutdown_connections 2020-11-20T12:04:49.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.641+0000 7f5646419700 1 -- 172.21.15.189:0/1425921284 wait complete. 2020-11-20T12:04:49.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.645+0000 7f5646419700 1 Processor -- start 2020-11-20T12:04:49.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.645+0000 7f5646419700 1 -- start start 2020-11-20T12:04:49.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.645+0000 7f5646419700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5640062790 0x7f56400a2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:49.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.645+0000 7f5646419700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5640062b70 0x7f56400a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:49.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.645+0000 7f5646419700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f564013bf60 0x7f564014abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:49.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.645+0000 7f5646419700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56400a30e0 con 0x7f5640062b70 2020-11-20T12:04:49.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.645+0000 7f5646419700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56400a34f0 con 0x7f564013bf60 2020-11-20T12:04:49.652 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.645+0000 7f5646419700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56400a3790 con 0x7f5640062790 2020-11-20T12:04:49.652 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.645+0000 7f5645c18700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f564013bf60 0x7f564014abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:49.652 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.645+0000 7f5645c18700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f564013bf60 0x7f564014abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48106/0 (socket says 172.21.15.189:48106) 2020-11-20T12:04:49.652 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.645+0000 7f5645c18700 1 -- 172.21.15.189:0/1525496873 learned_addr learned my addr 172.21.15.189:0/1525496873 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:49.652 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.645+0000 7f5645417700 1 --2- 172.21.15.189:0/1525496873 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5640062790 0x7f56400a2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:49.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.645+0000 7f5644c16700 1 --2- 172.21.15.189:0/1525496873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5640062b70 0x7f56400a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:49.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.645+0000 7f5645c18700 1 -- 172.21.15.189:0/1525496873 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5640062790 msgr2=0x7f56400a2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:49.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.645+0000 7f5645c18700 1 --2- 172.21.15.189:0/1525496873 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5640062790 0x7f56400a2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:49.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.645+0000 7f5645c18700 1 -- 172.21.15.189:0/1525496873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5640062b70 msgr2=0x7f56400a5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:49.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.645+0000 7f5645c18700 1 --2- 172.21.15.189:0/1525496873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5640062b70 0x7f56400a5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:49.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.645+0000 7f5645c18700 1 -- 172.21.15.189:0/1525496873 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56400a3a30 con 0x7f564013bf60 2020-11-20T12:04:49.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.645+0000 7f5645c18700 1 --2- 172.21.15.189:0/1525496873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f564013bf60 0x7f564014abf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f5638009210 tx=0x7f5638016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:49.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.645+0000 7f56357fa700 1 -- 172.21.15.189:0/1525496873 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5638010700 con 0x7f564013bf60 2020-11-20T12:04:49.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.645+0000 7f56357fa700 1 -- 172.21.15.189:0/1525496873 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5638010860 con 0x7f564013bf60 2020-11-20T12:04:49.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.645+0000 7f5646419700 1 -- 172.21.15.189:0/1525496873 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56400a3ed0 con 0x7f564013bf60 2020-11-20T12:04:49.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.645+0000 7f56357fa700 1 -- 172.21.15.189:0/1525496873 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5638010be0 con 0x7f564013bf60 2020-11-20T12:04:49.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.645+0000 7f5646419700 1 -- 172.21.15.189:0/1525496873 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56400ae950 con 0x7f564013bf60 2020-11-20T12:04:49.656 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.649+0000 7f5646419700 1 -- 172.21.15.189:0/1525496873 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56400a2750 con 0x7f564013bf60 2020-11-20T12:04:49.656 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.649+0000 7f56357fa700 1 -- 172.21.15.189:0/1525496873 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f5638017070 con 0x7f564013bf60 2020-11-20T12:04:49.656 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.649+0000 7f56357fa700 1 --2- 172.21.15.189:0/1525496873 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f56180557c0 0x7f5618057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:49.656 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.649+0000 7f5645417700 1 --2- 172.21.15.189:0/1525496873 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f56180557c0 0x7f5618057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:49.657 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.649+0000 7f56357fa700 1 -- 172.21.15.189:0/1525496873 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f5638070840 con 0x7f564013bf60 2020-11-20T12:04:49.657 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.649+0000 7f5645417700 1 --2- 172.21.15.189:0/1525496873 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f56180557c0 0x7f5618057c80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f5630006fd0 tx=0x7f5630006df0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:49.657 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.653+0000 7f56357fa700 1 -- 172.21.15.189:0/1525496873 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5638046a60 con 0x7f564013bf60 2020-11-20T12:04:49.814 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:49.813+0000 7f5646419700 1 -- 172.21.15.189:0/1525496873 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f56400446d0 con 0x7f564013bf60 2020-11-20T12:04:50.495 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.493+0000 7f56357fa700 1 -- 172.21.15.189:0/1525496873 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v17) v1 ==== 173+0+0 (secure 0 0 0) 0x7f5638015070 con 0x7f564013bf60 2020-11-20T12:04:50.500 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.497+0000 7f5646419700 1 -- 172.21.15.189:0/1525496873 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f56180557c0 msgr2=0x7f5618057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:50.500 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.497+0000 7f5646419700 1 --2- 172.21.15.189:0/1525496873 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f56180557c0 0x7f5618057c80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f5630006fd0 tx=0x7f5630006df0).stop 2020-11-20T12:04:50.500 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.497+0000 7f5646419700 1 -- 172.21.15.189:0/1525496873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f564013bf60 msgr2=0x7f564014abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:50.501 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.497+0000 7f5646419700 1 --2- 172.21.15.189:0/1525496873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f564013bf60 0x7f564014abf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f5638009210 tx=0x7f5638016000).stop 2020-11-20T12:04:50.501 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.497+0000 7f5646419700 1 -- 172.21.15.189:0/1525496873 shutdown_connections 2020-11-20T12:04:50.501 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.497+0000 7f5646419700 1 --2- 172.21.15.189:0/1525496873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f564013bf60 0x7f564014abf0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:50.501 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.497+0000 7f5646419700 1 --2- 172.21.15.189:0/1525496873 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5640062790 0x7f56400a2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:50.502 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.497+0000 7f5646419700 1 --2- 172.21.15.189:0/1525496873 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f56180557c0 0x7f5618057c80 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:50.502 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.497+0000 7f5646419700 1 --2- 172.21.15.189:0/1525496873 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5640062b70 0x7f56400a5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:50.502 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.497+0000 7f5646419700 1 -- 172.21.15.189:0/1525496873 >> 172.21.15.189:0/1525496873 conn(0x7f5640065ef0 msgr2=0x7f56400aa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:50.502 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.497+0000 7f5646419700 1 -- 172.21.15.189:0/1525496873 shutdown_connections 2020-11-20T12:04:50.503 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.497+0000 7f5646419700 1 -- 172.21.15.189:0/1525496873 wait complete. 2020-11-20T12:04:50.503 INFO:teuthology.orchestra.run.smithi189.stderr:enabled application 'rbd' on pool 'rbd' 2020-11-20T12:04:50.512 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2020-11-20T12:04:50.512 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:04:50.512 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:04:50.515 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:04:50.515 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:04:50.516 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:04:50.518 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:04:50.518 INFO:tasks.ceph:Setting up CephFS filesystem... 2020-11-20T12:04:50.519 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:04:50.597 INFO:tasks.ceph.mds.b.smithi047.stdout:starting mds.b at 2020-11-20T12:04:50.599 INFO:tasks.ceph.mds.a.smithi189.stdout:starting mds.a at 2020-11-20T12:04:50.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.649+0000 7f9547fff700 1 Processor -- start 2020-11-20T12:04:50.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.649+0000 7f9547fff700 1 -- start start 2020-11-20T12:04:50.652 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.649+0000 7f9547fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f95480aeb80 0x7f95480aadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:50.652 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.649+0000 7f9547fff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9548064830 0x7f95480a9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:50.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.649+0000 7f9547fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95480a9b80 con 0x7f9548064e80 2020-11-20T12:04:50.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.649+0000 7f9547fff700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95480ab2c0 con 0x7f95480aeb80 2020-11-20T12:04:50.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.649+0000 7f9547fff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95480af090 con 0x7f9548064830 2020-11-20T12:04:50.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.649+0000 7f95467fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f95480aeb80 0x7f95480aadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:50.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.649+0000 7f95467fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f95480aeb80 0x7f95480aadd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48124/0 (socket says 172.21.15.189:48124) 2020-11-20T12:04:50.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.649+0000 7f95467fc700 1 -- 172.21.15.189:0/1156975979 learned_addr learned my addr 172.21.15.189:0/1156975979 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:50.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.649+0000 7f95477fe700 1 --2- 172.21.15.189:0/1156975979 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9548064830 0x7f95480a9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:50.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.649+0000 7f95467fc700 1 -- 172.21.15.189:0/1156975979 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9548064830 msgr2=0x7f95480a9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:50.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.649+0000 7f95467fc700 1 --2- 172.21.15.189:0/1156975979 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9548064830 0x7f95480a9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:50.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.649+0000 7f95467fc700 1 -- 172.21.15.189:0/1156975979 >> v1:172.21.15.47:6789/0 conn(0x7f9548064e80 legacy=0x7f95480ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:50.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.649+0000 7f95467fc700 1 -- 172.21.15.189:0/1156975979 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95480af330 con 0x7f95480aeb80 2020-11-20T12:04:50.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.653+0000 7f95467fc700 1 --2- 172.21.15.189:0/1156975979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95480aeb80 0x7f95480aadd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f9530006fd0 tx=0x7f9530006eb0).ready entity=mon.1 client_cookie=a586bced4ba29a7c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:50.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.653+0000 7f9545ffb700 1 -- 172.21.15.189:0/1156975979 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9530009920 con 0x7f95480aeb80 2020-11-20T12:04:50.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.653+0000 7f9545ffb700 1 -- 172.21.15.189:0/1156975979 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9530009a80 con 0x7f95480aeb80 2020-11-20T12:04:50.656 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.653+0000 7f9545ffb700 1 -- 172.21.15.189:0/1156975979 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9530009e00 con 0x7f95480aeb80 2020-11-20T12:04:50.658 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.653+0000 7f9547fff700 1 -- 172.21.15.189:0/1156975979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95480aeb80 msgr2=0x7f95480aadd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:50.658 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.653+0000 7f9547fff700 1 --2- 172.21.15.189:0/1156975979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95480aeb80 0x7f95480aadd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f9530006fd0 tx=0x7f9530006eb0).stop 2020-11-20T12:04:50.659 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.653+0000 7f9547fff700 1 -- 172.21.15.189:0/1156975979 shutdown_connections 2020-11-20T12:04:50.659 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.653+0000 7f9547fff700 1 --2- 172.21.15.189:0/1156975979 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9548064830 0x7f95480a9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:50.659 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.653+0000 7f9547fff700 1 --2- 172.21.15.189:0/1156975979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95480aeb80 0x7f95480aadd0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:50.659 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.653+0000 7f9547fff700 1 -- 172.21.15.189:0/1156975979 >> 172.21.15.189:0/1156975979 conn(0x7f9548062080 msgr2=0x7f9548062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:50.660 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.653+0000 7f9547fff700 1 -- 172.21.15.189:0/1156975979 shutdown_connections 2020-11-20T12:04:50.660 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.653+0000 7f9547fff700 1 -- 172.21.15.189:0/1156975979 wait complete. 2020-11-20T12:04:50.660 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.653+0000 7f9547fff700 1 Processor -- start 2020-11-20T12:04:50.660 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.653+0000 7f9547fff700 1 -- start start 2020-11-20T12:04:50.660 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.657+0000 7f9547fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9548064830 0x7f9548151780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:50.661 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.657+0000 7f9547fff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9548064e80 0x7f9548141080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:50.661 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.657+0000 7f9547fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95480aeb80 0x7f9548142170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:50.661 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.657+0000 7f9547fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9548141570 con 0x7f95480aeb80 2020-11-20T12:04:50.661 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.657+0000 7f9547fff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95480a8530 con 0x7f9548064830 2020-11-20T12:04:50.661 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.657+0000 7f9547fff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9548142660 con 0x7f9548064e80 2020-11-20T12:04:50.662 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.657+0000 7f9546ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9548064830 0x7f9548151780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:50.662 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.657+0000 7f9546ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9548064830 0x7f9548151780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48128/0 (socket says 172.21.15.189:48128) 2020-11-20T12:04:50.662 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.657+0000 7f9546ffd700 1 -- 172.21.15.189:0/203950264 learned_addr learned my addr 172.21.15.189:0/203950264 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:50.662 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.657+0000 7f95467fc700 1 --2- 172.21.15.189:0/203950264 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9548064e80 0x7f9548141080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:50.663 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.657+0000 7f95477fe700 1 --2- 172.21.15.189:0/203950264 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95480aeb80 0x7f9548142170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:50.663 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.657+0000 7f9546ffd700 1 -- 172.21.15.189:0/203950264 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9548064e80 msgr2=0x7f9548141080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:50.663 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.657+0000 7f9546ffd700 1 --2- 172.21.15.189:0/203950264 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9548064e80 0x7f9548141080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:50.663 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.657+0000 7f9546ffd700 1 -- 172.21.15.189:0/203950264 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95480aeb80 msgr2=0x7f9548142170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:50.663 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.657+0000 7f9546ffd700 1 --2- 172.21.15.189:0/203950264 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95480aeb80 0x7f9548142170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:50.664 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.657+0000 7f9546ffd700 1 -- 172.21.15.189:0/203950264 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9548142d80 con 0x7f9548064830 2020-11-20T12:04:50.664 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.657+0000 7f9546ffd700 1 --2- 172.21.15.189:0/203950264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9548064830 0x7f9548151780 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f954000cae0 tx=0x7f9540002430).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:50.664 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.657+0000 7f953effd700 1 -- 172.21.15.189:0/203950264 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954000a730 con 0x7f9548064830 2020-11-20T12:04:50.664 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.657+0000 7f953effd700 1 -- 172.21.15.189:0/203950264 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f954000a890 con 0x7f9548064830 2020-11-20T12:04:50.665 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.657+0000 7f9547fff700 1 -- 172.21.15.189:0/203950264 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9548141790 con 0x7f9548064830 2020-11-20T12:04:50.665 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.657+0000 7f953effd700 1 -- 172.21.15.189:0/203950264 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954000ac30 con 0x7f9548064830 2020-11-20T12:04:50.665 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.657+0000 7f9547fff700 1 -- 172.21.15.189:0/203950264 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9548152260 con 0x7f9548064830 2020-11-20T12:04:50.666 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.657+0000 7f9547fff700 1 -- 172.21.15.189:0/203950264 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95481524b0 con 0x7f9548064830 2020-11-20T12:04:50.666 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.661+0000 7f953effd700 1 -- 172.21.15.189:0/203950264 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f9540019030 con 0x7f9548064830 2020-11-20T12:04:50.666 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.661+0000 7f953effd700 1 --2- 172.21.15.189:0/203950264 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f95240557c0 0x7f9524057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:50.667 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.661+0000 7f953effd700 1 -- 172.21.15.189:0/203950264 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f954006db20 con 0x7f9548064830 2020-11-20T12:04:50.667 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.661+0000 7f95467fc700 1 --2- 172.21.15.189:0/203950264 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f95240557c0 0x7f9524057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:50.667 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.661+0000 7f95467fc700 1 --2- 172.21.15.189:0/203950264 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f95240557c0 0x7f9524057c80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f9530009fd0 tx=0x7f9530015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:50.667 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.665+0000 7f953effd700 1 -- 172.21.15.189:0/203950264 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9540043da0 con 0x7f9548064830 2020-11-20T12:04:50.820 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.821+0000 7f9547fff700 1 -- 172.21.15.189:0/203950264 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9548141ed0 con 0x7f9548064830 2020-11-20T12:04:50.822 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.821+0000 7f953effd700 1 -- 172.21.15.189:0/203950264 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+13685 (secure 0 0 0) 0x7f9540024590 con 0x7f9548064830 2020-11-20T12:04:50.822 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:50.823 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":17,"fsid":"cda18d0e-64b3-41bb-964a-084069c3e648","created":"2020-11-20T12:04:37.538706+0000","modified":"2020-11-20T12:04:50.481309+0000","last_up_change":"2020-11-20T12:04:43.006631+0000","last_in_change":"2020-11-20T12:04:43.006631+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:41.395395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:48.550454+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c81214e1-4f84-4aaa-8ceb-99c398b2a89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6811","nonce":11760},{"type":"v1","addr":"172.21.15.189:6814","nonce":11760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":11760},{"type":"v1","addr":"172.21.15.189:6818","nonce":11760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":11760},{"type":"v1","addr":"172.21.15.189:6828","nonce":11760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6821","nonce":11760},{"type":"v1","addr":"172.21.15.189:6824","nonce":11760}]},"public_addr":"172.21.15.189:6814/11760","cluster_addr":"172.21.15.189:6818/11760","heartbeat_back_addr":"172.21.15.189:6828/11760","heartbeat_front_addr":"172.21.15.189:6824/11760","state":["exists","up"]},{"osd":1,"uuid":"d2f3ec7d-d403-484f-b5ed-b2e6b079d63d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6805","nonce":11761},{"type":"v1","addr":"172.21.15.189:6807","nonce":11761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":11761},{"type":"v1","addr":"172.21.15.189:6812","nonce":11761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6817","nonce":11761},{"type":"v1","addr":"172.21.15.189:6820","nonce":11761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6813","nonce":11761},{"type":"v1","addr":"172.21.15.189:6815","nonce":11761}]},"public_addr":"172.21.15.189:6807/11761","cluster_addr":"172.21.15.189:6812/11761","heartbeat_back_addr":"172.21.15.189:6820/11761","heartbeat_front_addr":"172.21.15.189:6815/11761","state":["exists","up"]},{"osd":2,"uuid":"e6cf4790-000c-43eb-8844-175f04c60503","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6819","nonce":11751},{"type":"v1","addr":"172.21.15.189:6822","nonce":11751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":11751},{"type":"v1","addr":"172.21.15.189:6825","nonce":11751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":11751},{"type":"v1","addr":"172.21.15.189:6831","nonce":11751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6827","nonce":11751},{"type":"v1","addr":"172.21.15.189:6829","nonce":11751}]},"public_addr":"172.21.15.189:6822/11751","cluster_addr":"172.21.15.189:6825/11751","heartbeat_back_addr":"172.21.15.189:6831/11751","heartbeat_front_addr":"172.21.15.189:6829/11751","state":["exists","up"]},{"osd":3,"uuid":"ba49dfac-f2b9-476e-853d-42f05e21a77a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":11762},{"type":"v1","addr":"172.21.15.189:6801","nonce":11762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":11762},{"type":"v1","addr":"172.21.15.189:6803","nonce":11762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":11762},{"type":"v1","addr":"172.21.15.189:6809","nonce":11762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":11762},{"type":"v1","addr":"172.21.15.189:6806","nonce":11762}]},"public_addr":"172.21.15.189:6801/11762","cluster_addr":"172.21.15.189:6803/11762","heartbeat_back_addr":"172.21.15.189:6809/11762","heartbeat_front_addr":"172.21.15.189:6806/11762","state":["exists","up"]},{"osd":4,"uuid":"3ca7b438-38b2-4689-8b18-fb5a8ddf6aee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":12114},{"type":"v1","addr":"172.21.15.47:6825","nonce":12114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":12114},{"type":"v1","addr":"172.21.15.47:6827","nonce":12114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":12114},{"type":"v1","addr":"172.21.15.47:6831","nonce":12114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":12114},{"type":"v1","addr":"172.21.15.47:6829","nonce":12114}]},"public_addr":"172.21.15.47:6825/12114","cluster_addr":"172.21.15.47:6827/12114","heartbeat_back_addr":"172.21.15.47:6831/12114","heartbeat_front_addr":"172.21.15.47:6829/12114","state":["exists","up"]},{"osd":5,"uuid":"6a45fca7-83d3-44fd-a1d4-b01e4201a101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":12111},{"type":"v1","addr":"172.21.15.47:6801","nonce":12111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":12111},{"type":"v1","addr":"172.21.15.47:6803","nonce":12111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":12111},{"type":"v1","addr":"172.21.15.47:6807","nonce":12111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":12111},{"type":"v1","addr":"172.21.15.47:6805","nonce":12111}]},"public_addr":"172.21.15.47:6801/12111","cluster_addr":"172.21.15.47:6803/12111","heartbeat_back_addr":"172.21.15.47:6807/12111","heartbeat_front_addr":"172.21.15.47:6805/12111","state":["exists","up"]},{"osd":6,"uuid":"c4b1b9ed-61f8-4677-9755-dff59a30e447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":12113},{"type":"v1","addr":"172.21.15.47:6817","nonce":12113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":12113},{"type":"v1","addr":"172.21.15.47:6819","nonce":12113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":12113},{"type":"v1","addr":"172.21.15.47:6823","nonce":12113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":12113},{"type":"v1","addr":"172.21.15.47:6821","nonce":12113}]},"public_addr":"172.21.15.47:6817/12113","cluster_addr":"172.21.15.47:6819/12113","heartbeat_back_addr":"172.21.15.47:6823/12113","heartbeat_front_addr":"172.21.15.47:6821/12113","state":["exists","up"]},{"osd":7,"uuid":"4608e4ed-cc40-4c5c-ba8b-3779bbc62f65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":12112},{"type":"v1","addr":"172.21.15.47:6809","nonce":12112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":12112},{"type":"v1","addr":"172.21.15.47:6811","nonce":12112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":12112},{"type":"v1","addr":"172.21.15.47:6815","nonce":12112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":12112},{"type":"v1","addr":"172.21.15.47:6813","nonce":12112}]},"public_addr":"172.21.15.47:6809/12112","cluster_addr":"172.21.15.47:6811/12112","heartbeat_back_addr":"172.21.15.47:6815/12112","heartbeat_front_addr":"172.21.15.47:6813/12112","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.435265+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.472776+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.388370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.417366+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.102392+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.116529+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.167430+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.148139+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:04:50.824 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.821+0000 7f9547fff700 1 -- 172.21.15.189:0/203950264 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f95240557c0 msgr2=0x7f9524057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:50.824 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.821+0000 7f9547fff700 1 --2- 172.21.15.189:0/203950264 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f95240557c0 0x7f9524057c80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f9530009fd0 tx=0x7f9530015000).stop 2020-11-20T12:04:50.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.821+0000 7f9547fff700 1 -- 172.21.15.189:0/203950264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9548064830 msgr2=0x7f9548151780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:50.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.821+0000 7f9547fff700 1 --2- 172.21.15.189:0/203950264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9548064830 0x7f9548151780 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f954000cae0 tx=0x7f9540002430).stop 2020-11-20T12:04:50.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.821+0000 7f9547fff700 1 -- 172.21.15.189:0/203950264 shutdown_connections 2020-11-20T12:04:50.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.821+0000 7f9547fff700 1 --2- 172.21.15.189:0/203950264 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f95240557c0 0x7f9524057c80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:50.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.821+0000 7f9547fff700 1 --2- 172.21.15.189:0/203950264 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f95480aeb80 0x7f9548142170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:50.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.821+0000 7f9547fff700 1 --2- 172.21.15.189:0/203950264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9548064830 0x7f9548151780 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:50.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.821+0000 7f9547fff700 1 --2- 172.21.15.189:0/203950264 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9548064e80 0x7f9548141080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:50.827 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.825+0000 7f9547fff700 1 -- 172.21.15.189:0/203950264 >> 172.21.15.189:0/203950264 conn(0x7f9548062080 msgr2=0x7f954805ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:50.827 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.825+0000 7f9547fff700 1 -- 172.21.15.189:0/203950264 shutdown_connections 2020-11-20T12:04:50.827 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.825+0000 7f9547fff700 1 -- 172.21.15.189:0/203950264 wait complete. 2020-11-20T12:04:50.837 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:41.395395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:48.550454+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-20T12:04:50.838 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:04:50.967 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.965+0000 7f9b53a77700 1 Processor -- start 2020-11-20T12:04:50.968 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.965+0000 7f9b53a77700 1 -- start start 2020-11-20T12:04:50.971 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b53a77700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9b4c0ae0d0 0x7f9b4c13aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:50.971 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b53a77700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f9b4c13af60 0x7f9b4c060170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:50.971 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b53a77700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b4c066960 con 0x7f9b4c060660 2020-11-20T12:04:50.971 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b53a77700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b4c13a680 con 0x7f9b4c13af60 2020-11-20T12:04:50.972 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b53a77700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b4c067110 con 0x7f9b4c0ae0d0 2020-11-20T12:04:50.972 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b52014700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9b4c060660 0x7f9b4c137170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49762/0 (socket says 172.21.15.189:49762) 2020-11-20T12:04:50.972 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b51813700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9b4c0ae0d0 0x7f9b4c13aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:50.972 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b52014700 1 -- 172.21.15.189:0/3088816333 learned_addr learned my addr 172.21.15.189:0/3088816333 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:04:50.973 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b51012700 1 --2- 172.21.15.189:0/3088816333 >> v2:172.21.15.47:3300/0 conn(0x7f9b4c13af60 0x7f9b4c060170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:50.973 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b50811700 1 -- 172.21.15.189:0/3088816333 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418311451 0 0) 0x7f9b4c066960 con 0x7f9b4c060660 2020-11-20T12:04:50.973 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b50811700 1 -- 172.21.15.189:0/3088816333 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b34003240 con 0x7f9b4c060660 2020-11-20T12:04:50.974 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b51813700 1 -- 172.21.15.189:0/3088816333 >> v2:172.21.15.47:3300/0 conn(0x7f9b4c13af60 msgr2=0x7f9b4c060170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:50.974 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b51813700 1 --2- 172.21.15.189:0/3088816333 >> v2:172.21.15.47:3300/0 conn(0x7f9b4c13af60 0x7f9b4c060170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:50.974 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b51813700 1 -- 172.21.15.189:0/3088816333 >> v1:172.21.15.189:6789/0 conn(0x7f9b4c060660 legacy=0x7f9b4c137170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:50.974 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b51813700 1 -- 172.21.15.189:0/3088816333 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b4c0673b0 con 0x7f9b4c0ae0d0 2020-11-20T12:04:50.974 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b51813700 1 --2- 172.21.15.189:0/3088816333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b4c0ae0d0 0x7f9b4c13aa70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f9b3c00b9a0 tx=0x7f9b3c006df0).ready entity=mon.1 client_cookie=b1455dbcbd0792a9 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:50.975 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b50811700 1 -- 172.21.15.189:0/3088816333 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c009640 con 0x7f9b4c0ae0d0 2020-11-20T12:04:50.975 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b50811700 1 -- 172.21.15.189:0/3088816333 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b3c0097a0 con 0x7f9b4c0ae0d0 2020-11-20T12:04:50.975 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b50811700 1 -- 172.21.15.189:0/3088816333 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c009b20 con 0x7f9b4c0ae0d0 2020-11-20T12:04:50.975 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b53a77700 1 -- 172.21.15.189:0/3088816333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b4c0ae0d0 msgr2=0x7f9b4c13aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:50.976 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b53a77700 1 --2- 172.21.15.189:0/3088816333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b4c0ae0d0 0x7f9b4c13aa70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f9b3c00b9a0 tx=0x7f9b3c006df0).stop 2020-11-20T12:04:50.976 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b53a77700 1 -- 172.21.15.189:0/3088816333 shutdown_connections 2020-11-20T12:04:50.976 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b53a77700 1 --2- 172.21.15.189:0/3088816333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b4c0ae0d0 0x7f9b4c13aa70 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:50.977 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b53a77700 1 --2- 172.21.15.189:0/3088816333 >> v2:172.21.15.47:3300/0 conn(0x7f9b4c13af60 0x7f9b4c060170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:50.977 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b53a77700 1 -- 172.21.15.189:0/3088816333 >> 172.21.15.189:0/3088816333 conn(0x7f9b4c0b1850 msgr2=0x7f9b4c0ad820 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:50.977 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b53a77700 1 -- 172.21.15.189:0/3088816333 shutdown_connections 2020-11-20T12:04:50.977 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.969+0000 7f9b53a77700 1 -- 172.21.15.189:0/3088816333 wait complete. 2020-11-20T12:04:50.980 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.973+0000 7f9b53a77700 1 Processor -- start 2020-11-20T12:04:50.980 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.973+0000 7f9b53a77700 1 -- start start 2020-11-20T12:04:50.980 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.973+0000 7f9b53a77700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b4c060660 0x7f9b4c130460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:50.981 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.973+0000 7f9b53a77700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9b4c0ae0d0 0x7f9b4c0af020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:50.981 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.973+0000 7f9b53a77700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b4c13af60 0x7f9b4c0af510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:50.981 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.973+0000 7f9b53a77700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4c13ed20 con 0x7f9b4c13af60 2020-11-20T12:04:50.981 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.973+0000 7f9b53a77700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4c13fd00 con 0x7f9b4c060660 2020-11-20T12:04:50.981 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.973+0000 7f9b53a77700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4c13ef80 con 0x7f9b4c0ae0d0 2020-11-20T12:04:50.982 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.973+0000 7f9b51813700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b4c060660 0x7f9b4c130460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:50.982 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.973+0000 7f9b51813700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b4c060660 0x7f9b4c130460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48142/0 (socket says 172.21.15.189:48142) 2020-11-20T12:04:50.982 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.973+0000 7f9b51813700 1 -- 172.21.15.189:0/4169753597 learned_addr learned my addr 172.21.15.189:0/4169753597 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:50.982 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.973+0000 7f9b51012700 1 --2- 172.21.15.189:0/4169753597 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9b4c0ae0d0 0x7f9b4c0af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:50.983 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.973+0000 7f9b52014700 1 --2- 172.21.15.189:0/4169753597 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b4c13af60 0x7f9b4c0af510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:50.983 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.973+0000 7f9b51813700 1 -- 172.21.15.189:0/4169753597 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9b4c0ae0d0 msgr2=0x7f9b4c0af020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:50.983 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.973+0000 7f9b51813700 1 --2- 172.21.15.189:0/4169753597 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9b4c0ae0d0 0x7f9b4c0af020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:50.983 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.973+0000 7f9b51813700 1 -- 172.21.15.189:0/4169753597 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b4c13af60 msgr2=0x7f9b4c0af510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:50.984 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.973+0000 7f9b51813700 1 --2- 172.21.15.189:0/4169753597 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b4c13af60 0x7f9b4c0af510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:50.984 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.973+0000 7f9b51813700 1 -- 172.21.15.189:0/4169753597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b4c13f220 con 0x7f9b4c060660 2020-11-20T12:04:50.984 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.973+0000 7f9b51012700 1 --2- 172.21.15.189:0/4169753597 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9b4c0ae0d0 0x7f9b4c0af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:50.984 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.973+0000 7f9b51813700 1 --2- 172.21.15.189:0/4169753597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b4c060660 0x7f9b4c130460 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f9b3c006e30 tx=0x7f9b3c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:50.985 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.973+0000 7f9b39ffb700 1 -- 172.21.15.189:0/4169753597 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c011e20 con 0x7f9b4c060660 2020-11-20T12:04:50.985 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.973+0000 7f9b39ffb700 1 -- 172.21.15.189:0/4169753597 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b3c01d450 con 0x7f9b4c060660 2020-11-20T12:04:50.985 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.973+0000 7f9b39ffb700 1 -- 172.21.15.189:0/4169753597 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c01d810 con 0x7f9b4c060660 2020-11-20T12:04:50.985 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.973+0000 7f9b53a77700 1 -- 172.21.15.189:0/4169753597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b4c12e4b0 con 0x7f9b4c060660 2020-11-20T12:04:50.986 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.973+0000 7f9b53a77700 1 -- 172.21.15.189:0/4169753597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b4c12e9b0 con 0x7f9b4c060660 2020-11-20T12:04:50.986 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.973+0000 7f9b53a77700 1 -- 172.21.15.189:0/4169753597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b4c044f60 con 0x7f9b4c060660 2020-11-20T12:04:50.986 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.977+0000 7f9b39ffb700 1 -- 172.21.15.189:0/4169753597 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f9b3c00f070 con 0x7f9b4c060660 2020-11-20T12:04:50.986 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.977+0000 7f9b39ffb700 1 --2- 172.21.15.189:0/4169753597 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9b240557c0 0x7f9b24057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:50.987 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.977+0000 7f9b39ffb700 1 -- 172.21.15.189:0/4169753597 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f9b3c070140 con 0x7f9b4c060660 2020-11-20T12:04:50.987 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.977+0000 7f9b51012700 1 --2- 172.21.15.189:0/4169753597 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9b240557c0 0x7f9b24057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:50.987 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.977+0000 7f9b51012700 1 --2- 172.21.15.189:0/4169753597 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9b240557c0 0x7f9b24057c80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f9b4c062d80 tx=0x7f9b48008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:50.987 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:50.985+0000 7f9b39ffb700 1 -- 172.21.15.189:0/4169753597 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9b3c046310 con 0x7f9b4c060660 2020-11-20T12:04:51.132 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.133+0000 7f9b53a77700 1 -- 172.21.15.189:0/4169753597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f9b4c064a30 con 0x7f9b4c060660 2020-11-20T12:04:51.133 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.133+0000 7f9b39ffb700 1 -- 172.21.15.189:0/4169753597 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v17) v1 ==== 110+0+10 (secure 0 0 0) 0x7f9b3c049930 con 0x7f9b4c060660 2020-11-20T12:04:51.134 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2020-11-20T12:04:51.135 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.133+0000 7f9b53a77700 1 -- 172.21.15.189:0/4169753597 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9b240557c0 msgr2=0x7f9b24057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.135 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.133+0000 7f9b53a77700 1 --2- 172.21.15.189:0/4169753597 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9b240557c0 0x7f9b24057c80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f9b4c062d80 tx=0x7f9b48008040).stop 2020-11-20T12:04:51.136 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.133+0000 7f9b53a77700 1 -- 172.21.15.189:0/4169753597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b4c060660 msgr2=0x7f9b4c130460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.136 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.133+0000 7f9b53a77700 1 --2- 172.21.15.189:0/4169753597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b4c060660 0x7f9b4c130460 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f9b3c006e30 tx=0x7f9b3c015000).stop 2020-11-20T12:04:51.136 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.133+0000 7f9b53a77700 1 -- 172.21.15.189:0/4169753597 shutdown_connections 2020-11-20T12:04:51.136 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.133+0000 7f9b53a77700 1 --2- 172.21.15.189:0/4169753597 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9b240557c0 0x7f9b24057c80 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.136 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.133+0000 7f9b53a77700 1 --2- 172.21.15.189:0/4169753597 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b4c13af60 0x7f9b4c0af510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.137 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.133+0000 7f9b53a77700 1 --2- 172.21.15.189:0/4169753597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b4c060660 0x7f9b4c130460 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.137 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.133+0000 7f9b53a77700 1 --2- 172.21.15.189:0/4169753597 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9b4c0ae0d0 0x7f9b4c0af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.137 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.133+0000 7f9b53a77700 1 -- 172.21.15.189:0/4169753597 >> 172.21.15.189:0/4169753597 conn(0x7f9b4c0b1850 msgr2=0x7f9b4c140fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:51.137 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.133+0000 7f9b53a77700 1 -- 172.21.15.189:0/4169753597 shutdown_connections 2020-11-20T12:04:51.137 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.133+0000 7f9b53a77700 1 -- 172.21.15.189:0/4169753597 wait complete. 2020-11-20T12:04:51.147 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:04:51.271 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.269+0000 7fa0ff955700 1 Processor -- start 2020-11-20T12:04:51.271 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.269+0000 7fa0ff955700 1 -- start start 2020-11-20T12:04:51.272 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.269+0000 7fa0ff955700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa0f813b180 0x7fa0f8137170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.272 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.269+0000 7fa0ff955700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f8066960 con 0x7fa0f80ae0d0 2020-11-20T12:04:51.273 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.269+0000 7fa0ff955700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f8067110 con 0x7fa0f8137660 2020-11-20T12:04:51.273 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.269+0000 7fa0ff955700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f813a680 con 0x7fa0f813b180 2020-11-20T12:04:51.273 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0fdef2700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa0f8137660 0x7fa0f81451e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49776/0 (socket says 172.21.15.189:49776) 2020-11-20T12:04:51.273 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0fdef2700 1 -- 172.21.15.189:0/742444540 learned_addr learned my addr 172.21.15.189:0/742444540 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:04:51.273 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0fcef0700 1 --2- 172.21.15.189:0/742444540 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa0f813b180 0x7fa0f8137170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.274 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0e3fff700 1 -- 172.21.15.189:0/742444540 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715616312 0 0) 0x7fa0f8067110 con 0x7fa0f8137660 2020-11-20T12:04:51.274 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0e3fff700 1 -- 172.21.15.189:0/742444540 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0dc003240 con 0x7fa0f8137660 2020-11-20T12:04:51.277 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0e3fff700 1 -- 172.21.15.189:0/742444540 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913650345 0 0) 0x7fa0f8066960 con 0x7fa0f80ae0d0 2020-11-20T12:04:51.278 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0e3fff700 1 -- 172.21.15.189:0/742444540 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0f8067110 con 0x7fa0f80ae0d0 2020-11-20T12:04:51.278 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0e3fff700 1 -- 172.21.15.189:0/742444540 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3002411288 0 0) 0x7fa0dc003240 con 0x7fa0f8137660 2020-11-20T12:04:51.278 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0e3fff700 1 -- 172.21.15.189:0/742444540 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa0f813b180 msgr2=0x7fa0f8137170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:51.279 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0e3fff700 1 --2- 172.21.15.189:0/742444540 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa0f813b180 0x7fa0f8137170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.279 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0e3fff700 1 -- 172.21.15.189:0/742444540 >> v1:172.21.15.47:6789/0 conn(0x7fa0f80ae0d0 legacy=0x7fa0f813aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.279 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0e3fff700 1 -- 172.21.15.189:0/742444540 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f80673b0 con 0x7fa0f8137660 2020-11-20T12:04:51.279 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0e3fff700 1 -- 172.21.15.189:0/742444540 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7fa0f00017e0 con 0x7fa0f8137660 2020-11-20T12:04:51.279 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0e3fff700 1 -- 172.21.15.189:0/742444540 >> v1:172.21.15.189:6789/0 conn(0x7fa0f8137660 legacy=0x7fa0f81451e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.280 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0e3fff700 1 --2- 172.21.15.189:0/742444540 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa0f813b180 0x7fa0dc0038c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.280 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0e3fff700 1 --2- 172.21.15.189:0/742444540 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0dc003e00 0x7fa0dc0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.280 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0e3fff700 1 --2- 172.21.15.189:0/742444540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0dc0049d0 0x7fa0dc004e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.280 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0e3fff700 1 -- 172.21.15.189:0/742444540 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0dc0053a0 con 0x7fa0dc003e00 2020-11-20T12:04:51.281 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0e3fff700 1 -- 172.21.15.189:0/742444540 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0dc00b760 con 0x7fa0dc0049d0 2020-11-20T12:04:51.281 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0e3fff700 1 -- 172.21.15.189:0/742444540 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0dc00b920 con 0x7fa0f813b180 2020-11-20T12:04:51.281 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0fd6f1700 1 --2- 172.21.15.189:0/742444540 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa0f813b180 0x7fa0dc0038c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.281 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0fdef2700 1 --2- 172.21.15.189:0/742444540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0dc0049d0 0x7fa0dc004e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.282 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0fcef0700 1 --2- 172.21.15.189:0/742444540 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0dc003e00 0x7fa0dc0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.282 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0fd6f1700 1 -- 172.21.15.189:0/742444540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0dc0049d0 msgr2=0x7fa0dc004e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:51.282 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0fd6f1700 1 --2- 172.21.15.189:0/742444540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0dc0049d0 0x7fa0dc004e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.282 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0fd6f1700 1 -- 172.21.15.189:0/742444540 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0dc003e00 msgr2=0x7fa0dc0044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:51.283 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0fd6f1700 1 --2- 172.21.15.189:0/742444540 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0dc003e00 0x7fa0dc0044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.283 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0fd6f1700 1 -- 172.21.15.189:0/742444540 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0dc003240 con 0x7fa0f813b180 2020-11-20T12:04:51.283 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0fd6f1700 1 --2- 172.21.15.189:0/742444540 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa0f813b180 0x7fa0dc0038c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fa0f4001a80 tx=0x7fa0f4007b90).ready entity=mon.2 client_cookie=571a371217c38125 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:51.283 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0e3fff700 1 -- 172.21.15.189:0/742444540 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f4007020 con 0x7fa0f813b180 2020-11-20T12:04:51.284 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0e3fff700 1 -- 172.21.15.189:0/742444540 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f4007180 con 0x7fa0f813b180 2020-11-20T12:04:51.284 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0e3fff700 1 -- 172.21.15.189:0/742444540 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f400c830 con 0x7fa0f813b180 2020-11-20T12:04:51.284 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0ff955700 1 -- 172.21.15.189:0/742444540 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa0f813b180 msgr2=0x7fa0dc0038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.284 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0ff955700 1 --2- 172.21.15.189:0/742444540 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa0f813b180 0x7fa0dc0038c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fa0f4001a80 tx=0x7fa0f4007b90).stop 2020-11-20T12:04:51.284 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0fdef2700 1 -- 172.21.15.189:0/742444540 reap_dead start 2020-11-20T12:04:51.285 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0ff955700 1 -- 172.21.15.189:0/742444540 shutdown_connections 2020-11-20T12:04:51.285 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0ff955700 1 -- 172.21.15.189:0/742444540 >> 172.21.15.189:0/742444540 conn(0x7fa0f80b1850 msgr2=0x7fa0f80ad820 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:51.285 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0ff955700 1 -- 172.21.15.189:0/742444540 shutdown_connections 2020-11-20T12:04:51.285 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.273+0000 7fa0ff955700 1 -- 172.21.15.189:0/742444540 wait complete. 2020-11-20T12:04:51.286 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.277+0000 7fa0ff955700 1 Processor -- start 2020-11-20T12:04:51.286 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.277+0000 7fa0ff955700 1 -- start start 2020-11-20T12:04:51.288 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.277+0000 7fa0ff955700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0f81395c0 0x7fa0f813ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.288 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.277+0000 7fa0ff955700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa0f81399a0 0x7fa0f813bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.289 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.277+0000 7fa0ff955700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0f80a98a0 0x7fa0f813dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.289 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.277+0000 7fa0ff955700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f814a8b0 con 0x7fa0f81395c0 2020-11-20T12:04:51.289 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.277+0000 7fa0ff955700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f8149ed0 con 0x7fa0f80a98a0 2020-11-20T12:04:51.289 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.277+0000 7fa0ff955700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f814a170 con 0x7fa0f81399a0 2020-11-20T12:04:51.290 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.277+0000 7fa0fdef2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0f80a98a0 0x7fa0f813dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.290 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.277+0000 7fa0fd6f1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0f81395c0 0x7fa0f813ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.290 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.277+0000 7fa0fdef2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0f80a98a0 0x7fa0f813dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48166/0 (socket says 172.21.15.189:48166) 2020-11-20T12:04:51.290 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.277+0000 7fa0fdef2700 1 -- 172.21.15.189:0/124973535 learned_addr learned my addr 172.21.15.189:0/124973535 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:51.291 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.277+0000 7fa0fcef0700 1 --2- 172.21.15.189:0/124973535 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa0f81399a0 0x7fa0f813bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.291 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.277+0000 7fa0fdef2700 1 -- 172.21.15.189:0/124973535 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa0f81399a0 msgr2=0x7fa0f813bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.291 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.277+0000 7fa0fdef2700 1 --2- 172.21.15.189:0/124973535 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa0f81399a0 0x7fa0f813bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.291 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.277+0000 7fa0fdef2700 1 -- 172.21.15.189:0/124973535 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0f81395c0 msgr2=0x7fa0f813ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.292 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.277+0000 7fa0fdef2700 1 --2- 172.21.15.189:0/124973535 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0f81395c0 0x7fa0f813ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.292 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.277+0000 7fa0fdef2700 1 -- 172.21.15.189:0/124973535 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f814a410 con 0x7fa0f80a98a0 2020-11-20T12:04:51.292 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.277+0000 7fa0fdef2700 1 --2- 172.21.15.189:0/124973535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0f80a98a0 0x7fa0f813dfd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fa0f8066340 tx=0x7fa0f8067540).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:51.292 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.281+0000 7fa0e9ffb700 1 -- 172.21.15.189:0/124973535 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f000f900 con 0x7fa0f80a98a0 2020-11-20T12:04:51.292 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.281+0000 7fa0e9ffb700 1 -- 172.21.15.189:0/124973535 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f000fa60 con 0x7fa0f80a98a0 2020-11-20T12:04:51.293 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.281+0000 7fa0ff955700 1 -- 172.21.15.189:0/124973535 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f813a0f0 con 0x7fa0f80a98a0 2020-11-20T12:04:51.293 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.281+0000 7fa0e9ffb700 1 -- 172.21.15.189:0/124973535 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f000fe00 con 0x7fa0f80a98a0 2020-11-20T12:04:51.293 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.281+0000 7fa0ff955700 1 -- 172.21.15.189:0/124973535 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f8149b70 con 0x7fa0f80a98a0 2020-11-20T12:04:51.293 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.281+0000 7fa0ff955700 1 -- 172.21.15.189:0/124973535 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f814a5a0 con 0x7fa0f80a98a0 2020-11-20T12:04:51.294 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.281+0000 7fa0e9ffb700 1 -- 172.21.15.189:0/124973535 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fa0f0015620 con 0x7fa0f80a98a0 2020-11-20T12:04:51.294 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.281+0000 7fa0e9ffb700 1 --2- 172.21.15.189:0/124973535 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa0cc0557c0 0x7fa0cc057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.294 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.281+0000 7fa0fd6f1700 1 --2- 172.21.15.189:0/124973535 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa0cc0557c0 0x7fa0cc057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.294 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.281+0000 7fa0e9ffb700 1 -- 172.21.15.189:0/124973535 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fa0f006df00 con 0x7fa0f80a98a0 2020-11-20T12:04:51.295 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.281+0000 7fa0fd6f1700 1 --2- 172.21.15.189:0/124973535 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa0cc0557c0 0x7fa0cc057c80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fa0f40010e0 tx=0x7fa0f4016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:51.295 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.289+0000 7fa0e9ffb700 1 -- 172.21.15.189:0/124973535 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa0f0043f60 con 0x7fa0f80a98a0 2020-11-20T12:04:51.443 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.441+0000 7fa0ff955700 1 -- 172.21.15.189:0/124973535 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fa0f8044f60 con 0x7fa0f80a98a0 2020-11-20T12:04:51.443 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.441+0000 7fa0e9ffb700 1 -- 172.21.15.189:0/124973535 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v17) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa0f0001510 con 0x7fa0f80a98a0 2020-11-20T12:04:51.444 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 8 2020-11-20T12:04:51.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.445+0000 7fa0ff955700 1 -- 172.21.15.189:0/124973535 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa0cc0557c0 msgr2=0x7fa0cc057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.445+0000 7fa0ff955700 1 --2- 172.21.15.189:0/124973535 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa0cc0557c0 0x7fa0cc057c80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fa0f40010e0 tx=0x7fa0f4016000).stop 2020-11-20T12:04:51.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.445+0000 7fa0ff955700 1 -- 172.21.15.189:0/124973535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0f80a98a0 msgr2=0x7fa0f813dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.445+0000 7fa0ff955700 1 --2- 172.21.15.189:0/124973535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0f80a98a0 0x7fa0f813dfd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fa0f8066340 tx=0x7fa0f8067540).stop 2020-11-20T12:04:51.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.445+0000 7fa0ff955700 1 -- 172.21.15.189:0/124973535 shutdown_connections 2020-11-20T12:04:51.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.445+0000 7fa0ff955700 1 --2- 172.21.15.189:0/124973535 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa0cc0557c0 0x7fa0cc057c80 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.445+0000 7fa0ff955700 1 --2- 172.21.15.189:0/124973535 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0f81395c0 0x7fa0f813ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.445+0000 7fa0ff955700 1 --2- 172.21.15.189:0/124973535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0f80a98a0 0x7fa0f813dfd0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.445+0000 7fa0ff955700 1 --2- 172.21.15.189:0/124973535 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa0f81399a0 0x7fa0f813bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.445+0000 7fa0ff955700 1 -- 172.21.15.189:0/124973535 >> 172.21.15.189:0/124973535 conn(0x7fa0f80b1850 msgr2=0x7fa0f80608c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:51.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.445+0000 7fa0ff955700 1 -- 172.21.15.189:0/124973535 shutdown_connections 2020-11-20T12:04:51.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.445+0000 7fa0ff955700 1 -- 172.21.15.189:0/124973535 wait complete. 2020-11-20T12:04:51.459 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2020-11-20T12:04:51.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.569+0000 7f11a4c85700 1 Processor -- start 2020-11-20T12:04:51.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.569+0000 7f11a4c85700 1 -- start start 2020-11-20T12:04:51.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.569+0000 7f11a4c85700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f11a00612b0 0x7f11a00a0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.569+0000 7f11a4c85700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f11a00a85a0 0x7f11a00a8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.569+0000 7f11a4c85700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f11a01339a0 0x7f11a0133e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.569+0000 7f11a4c85700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11a00a0f60 con 0x7f11a00612b0 2020-11-20T12:04:51.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.569+0000 7f11a4c85700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11a0068440 con 0x7f11a01339a0 2020-11-20T12:04:51.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.569+0000 7f11a4c85700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a0067d00 con 0x7f11a00a85a0 2020-11-20T12:04:51.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.569+0000 7f119ed9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f11a01339a0 0x7f11a0133e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.569+0000 7f119ed9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f11a01339a0 0x7f11a0133e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48174/0 (socket says 172.21.15.189:48174) 2020-11-20T12:04:51.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.569+0000 7f119ed9d700 1 -- 172.21.15.189:0/1556141549 learned_addr learned my addr 172.21.15.189:0/1556141549 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:51.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.569+0000 7f119e59c700 1 --2- 172.21.15.189:0/1556141549 >> v2:172.21.15.47:3300/0 conn(0x7f11a00612b0 0x7f11a00a0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.569+0000 7f1195d9b700 1 --2- 172.21.15.189:0/1556141549 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f11a00a85a0 0x7f11a00a8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.569+0000 7f119ed9d700 1 -- 172.21.15.189:0/1556141549 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f11a00a85a0 msgr2=0x7f11a00a8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:51.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.569+0000 7f119ed9d700 1 --2- 172.21.15.189:0/1556141549 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f11a00a85a0 0x7f11a00a8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.569+0000 7f119ed9d700 1 -- 172.21.15.189:0/1556141549 >> v2:172.21.15.47:3300/0 conn(0x7f11a00612b0 msgr2=0x7f11a00a0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:51.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.569+0000 7f119ed9d700 1 --2- 172.21.15.189:0/1556141549 >> v2:172.21.15.47:3300/0 conn(0x7f11a00612b0 0x7f11a00a0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.573+0000 7f119ed9d700 1 -- 172.21.15.189:0/1556141549 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a0067fa0 con 0x7f11a01339a0 2020-11-20T12:04:51.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.573+0000 7f1195d9b700 1 --2- 172.21.15.189:0/1556141549 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f11a00a85a0 0x7f11a00a8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:51.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.573+0000 7f119e59c700 1 --2- 172.21.15.189:0/1556141549 >> v2:172.21.15.47:3300/0 conn(0x7f11a00612b0 0x7f11a00a0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:51.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.573+0000 7f119ed9d700 1 --2- 172.21.15.189:0/1556141549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11a01339a0 0x7f11a0133e30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f11a00a82b0 tx=0x7f1198006eb0).ready entity=mon.1 client_cookie=a88cdeb1ceafede3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:51.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.573+0000 7f119dd9b700 1 -- 172.21.15.189:0/1556141549 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11980098d0 con 0x7f11a01339a0 2020-11-20T12:04:51.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.573+0000 7f119dd9b700 1 -- 172.21.15.189:0/1556141549 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1198009a30 con 0x7f11a01339a0 2020-11-20T12:04:51.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.573+0000 7f119dd9b700 1 -- 172.21.15.189:0/1556141549 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1198009db0 con 0x7f11a01339a0 2020-11-20T12:04:51.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.573+0000 7f11a4c85700 1 -- 172.21.15.189:0/1556141549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11a01339a0 msgr2=0x7f11a0133e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.573+0000 7f11a4c85700 1 --2- 172.21.15.189:0/1556141549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11a01339a0 0x7f11a0133e30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f11a00a82b0 tx=0x7f1198006eb0).stop 2020-11-20T12:04:51.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.573+0000 7f11a4c85700 1 -- 172.21.15.189:0/1556141549 shutdown_connections 2020-11-20T12:04:51.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.573+0000 7f11a4c85700 1 --2- 172.21.15.189:0/1556141549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11a01339a0 0x7f11a0133e30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.573+0000 7f11a4c85700 1 --2- 172.21.15.189:0/1556141549 >> v2:172.21.15.47:3300/0 conn(0x7f11a00612b0 0x7f11a00a0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.573+0000 7f11a4c85700 1 --2- 172.21.15.189:0/1556141549 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f11a00a85a0 0x7f11a00a8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.573+0000 7f11a4c85700 1 -- 172.21.15.189:0/1556141549 >> 172.21.15.189:0/1556141549 conn(0x7f11a0065910 msgr2=0x7f11a0060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:51.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.573+0000 7f11a4c85700 1 -- 172.21.15.189:0/1556141549 shutdown_connections 2020-11-20T12:04:51.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.573+0000 7f11a4c85700 1 -- 172.21.15.189:0/1556141549 wait complete. 2020-11-20T12:04:51.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.573+0000 7f11a4c85700 1 Processor -- start 2020-11-20T12:04:51.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.573+0000 7f11a4c85700 1 -- start start 2020-11-20T12:04:51.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.577+0000 7f11a4c85700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11a00612b0 0x7f11a0145130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.577+0000 7f11a4c85700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11a00a85a0 0x7f11a00a6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.577+0000 7f11a4c85700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f11a01339a0 0x7f11a0137090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.577+0000 7f11a4c85700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a00a4df0 con 0x7f11a00612b0 2020-11-20T12:04:51.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.577+0000 7f11a4c85700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a0146080 con 0x7f11a00a85a0 2020-11-20T12:04:51.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.577+0000 7f11a4c85700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a00a5020 con 0x7f11a01339a0 2020-11-20T12:04:51.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.577+0000 7f1195d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11a00a85a0 0x7f11a00a6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.577+0000 7f1195d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11a00a85a0 0x7f11a00a6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48180/0 (socket says 172.21.15.189:48180) 2020-11-20T12:04:51.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.577+0000 7f1195d9b700 1 -- 172.21.15.189:0/4154840931 learned_addr learned my addr 172.21.15.189:0/4154840931 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:51.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.577+0000 7f119ed9d700 1 --2- 172.21.15.189:0/4154840931 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f11a01339a0 0x7f11a0137090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.577+0000 7f119e59c700 1 --2- 172.21.15.189:0/4154840931 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11a00612b0 0x7f11a0145130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.577+0000 7f1195d9b700 1 -- 172.21.15.189:0/4154840931 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f11a01339a0 msgr2=0x7f11a0137090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.577+0000 7f1195d9b700 1 --2- 172.21.15.189:0/4154840931 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f11a01339a0 0x7f11a0137090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.577+0000 7f1195d9b700 1 -- 172.21.15.189:0/4154840931 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11a00612b0 msgr2=0x7f11a0145130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.577+0000 7f1195d9b700 1 --2- 172.21.15.189:0/4154840931 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11a00612b0 0x7f11a0145130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.577+0000 7f1195d9b700 1 -- 172.21.15.189:0/4154840931 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a00a52c0 con 0x7f11a00a85a0 2020-11-20T12:04:51.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.577+0000 7f119e59c700 1 --2- 172.21.15.189:0/4154840931 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11a00612b0 0x7f11a0145130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:51.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.577+0000 7f1195d9b700 1 --2- 172.21.15.189:0/4154840931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11a00a85a0 0x7f11a00a6d10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f118800ecf0 tx=0x7f118800c2b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:51.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.577+0000 7f1196ffd700 1 -- 172.21.15.189:0/4154840931 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118800ab40 con 0x7f11a00a85a0 2020-11-20T12:04:51.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.577+0000 7f1196ffd700 1 -- 172.21.15.189:0/4154840931 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f118800aca0 con 0x7f11a00a85a0 2020-11-20T12:04:51.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.577+0000 7f1196ffd700 1 -- 172.21.15.189:0/4154840931 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118801a620 con 0x7f11a00a85a0 2020-11-20T12:04:51.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.577+0000 7f11a4c85700 1 -- 172.21.15.189:0/4154840931 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11a0136540 con 0x7f11a00a85a0 2020-11-20T12:04:51.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.577+0000 7f11a4c85700 1 -- 172.21.15.189:0/4154840931 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11a0145620 con 0x7f11a00a85a0 2020-11-20T12:04:51.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.577+0000 7f11a4c85700 1 -- 172.21.15.189:0/4154840931 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11a00a5450 con 0x7f11a00a85a0 2020-11-20T12:04:51.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.581+0000 7f1196ffd700 1 -- 172.21.15.189:0/4154840931 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f118801a990 con 0x7f11a00a85a0 2020-11-20T12:04:51.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.581+0000 7f1196ffd700 1 --2- 172.21.15.189:0/4154840931 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f117c0557c0 0x7f117c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.588 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.581+0000 7f1196ffd700 1 -- 172.21.15.189:0/4154840931 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f118806f9d0 con 0x7f11a00a85a0 2020-11-20T12:04:51.588 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.581+0000 7f119e59c700 1 --2- 172.21.15.189:0/4154840931 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f117c0557c0 0x7f117c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.588 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.581+0000 7f119e59c700 1 --2- 172.21.15.189:0/4154840931 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f117c0557c0 0x7f117c057c80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f1190007c00 tx=0x7f1190008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:51.589 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.585+0000 7f1196ffd700 1 -- 172.21.15.189:0/4154840931 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1188045ba0 con 0x7f11a00a85a0 2020-11-20T12:04:51.684 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:04:51.687 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:04:51.745 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.745+0000 7f11a4c85700 1 -- 172.21.15.189:0/4154840931 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f11a0061690 con 0x7f11a00a85a0 2020-11-20T12:04:51.746 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.745+0000 7f1196ffd700 1 -- 172.21.15.189:0/4154840931 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) v1 ==== 84+0+143 (secure 0 0 0) 0x7f1188015020 con 0x7f11a00a85a0 2020-11-20T12:04:51.747 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:51.747 INFO:teuthology.orchestra.run.smithi189.stdout:[ 2020-11-20T12:04:51.747 INFO:teuthology.orchestra.run.smithi189.stdout: { 2020-11-20T12:04:51.747 INFO:teuthology.orchestra.run.smithi189.stdout: "poolnum": 1, 2020-11-20T12:04:51.748 INFO:teuthology.orchestra.run.smithi189.stdout: "poolname": "device_health_metrics" 2020-11-20T12:04:51.748 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2020-11-20T12:04:51.748 INFO:teuthology.orchestra.run.smithi189.stdout: { 2020-11-20T12:04:51.748 INFO:teuthology.orchestra.run.smithi189.stdout: "poolnum": 2, 2020-11-20T12:04:51.748 INFO:teuthology.orchestra.run.smithi189.stdout: "poolname": "rbd" 2020-11-20T12:04:51.749 INFO:teuthology.orchestra.run.smithi189.stdout: } 2020-11-20T12:04:51.749 INFO:teuthology.orchestra.run.smithi189.stdout:] 2020-11-20T12:04:51.750 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.749+0000 7f11a4c85700 1 -- 172.21.15.189:0/4154840931 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f117c0557c0 msgr2=0x7f117c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.750 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.749+0000 7f11a4c85700 1 --2- 172.21.15.189:0/4154840931 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f117c0557c0 0x7f117c057c80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f1190007c00 tx=0x7f1190008040).stop 2020-11-20T12:04:51.751 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.749+0000 7f11a4c85700 1 -- 172.21.15.189:0/4154840931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11a00a85a0 msgr2=0x7f11a00a6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.751 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.749+0000 7f11a4c85700 1 --2- 172.21.15.189:0/4154840931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11a00a85a0 0x7f11a00a6d10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f118800ecf0 tx=0x7f118800c2b0).stop 2020-11-20T12:04:51.751 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.749+0000 7f11a4c85700 1 -- 172.21.15.189:0/4154840931 shutdown_connections 2020-11-20T12:04:51.751 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.749+0000 7f11a4c85700 1 --2- 172.21.15.189:0/4154840931 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f117c0557c0 0x7f117c057c80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.751 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.749+0000 7f11a4c85700 1 --2- 172.21.15.189:0/4154840931 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f11a00612b0 0x7f11a0145130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.752 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.749+0000 7f11a4c85700 1 --2- 172.21.15.189:0/4154840931 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f11a01339a0 0x7f11a0137090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.752 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.749+0000 7f11a4c85700 1 --2- 172.21.15.189:0/4154840931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11a00a85a0 0x7f11a00a6d10 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.752 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.749+0000 7f11a4c85700 1 -- 172.21.15.189:0/4154840931 >> 172.21.15.189:0/4154840931 conn(0x7f11a0065910 msgr2=0x7f11a00a8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:51.752 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.749+0000 7f11a4c85700 1 -- 172.21.15.189:0/4154840931 shutdown_connections 2020-11-20T12:04:51.753 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.749+0000 7f11a4c85700 1 -- 172.21.15.189:0/4154840931 wait complete. 2020-11-20T12:04:51.767 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2020-11-20T12:04:51.767 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2020-11-20T12:04:51.890 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.889+0000 7f6363fff700 1 Processor -- start 2020-11-20T12:04:51.890 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.889+0000 7f6363fff700 1 -- start start 2020-11-20T12:04:51.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.889+0000 7f6363fff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f63640a8a60 0x7f6364067400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.889+0000 7f6363fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f636413e460 0x7f6364137640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.889+0000 7f6363fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6364066ad0 con 0x7f63640678f0 2020-11-20T12:04:51.893 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.889+0000 7f6363fff700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6364065f70 con 0x7f636413e460 2020-11-20T12:04:51.893 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.889+0000 7f6363fff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6364066d70 con 0x7f63640a8a60 2020-11-20T12:04:51.893 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.889+0000 7f63637fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f636413e460 0x7f6364137640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.893 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.889+0000 7f63637fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f636413e460 0x7f6364137640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48186/0 (socket says 172.21.15.189:48186) 2020-11-20T12:04:51.893 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.889+0000 7f63637fe700 1 -- 172.21.15.189:0/88556777 learned_addr learned my addr 172.21.15.189:0/88556777 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:51.894 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.889+0000 7f6362ffd700 1 --2- 172.21.15.189:0/88556777 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f63640a8a60 0x7f6364067400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.894 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.889+0000 7f63637fe700 1 -- 172.21.15.189:0/88556777 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f63640a8a60 msgr2=0x7f6364067400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:51.894 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.889+0000 7f63637fe700 1 --2- 172.21.15.189:0/88556777 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f63640a8a60 0x7f6364067400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.894 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.889+0000 7f63637fe700 1 -- 172.21.15.189:0/88556777 >> v1:172.21.15.47:6789/0 conn(0x7f63640678f0 legacy=0x7f636413dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.894 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.889+0000 7f63637fe700 1 -- 172.21.15.189:0/88556777 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6364067010 con 0x7f636413e460 2020-11-20T12:04:51.895 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.889+0000 7f6362ffd700 1 --2- 172.21.15.189:0/88556777 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f63640a8a60 0x7f6364067400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:04:51.895 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.889+0000 7f63637fe700 1 --2- 172.21.15.189:0/88556777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f636413e460 0x7f6364137640 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f6354009d50 tx=0x7f6354009290).ready entity=mon.1 client_cookie=7cf0a137c0100a97 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:51.895 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6361ffb700 1 -- 172.21.15.189:0/88556777 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635400bc80 con 0x7f636413e460 2020-11-20T12:04:51.895 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6361ffb700 1 -- 172.21.15.189:0/88556777 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635400bde0 con 0x7f636413e460 2020-11-20T12:04:51.895 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6361ffb700 1 -- 172.21.15.189:0/88556777 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6354011610 con 0x7f636413e460 2020-11-20T12:04:51.899 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6363fff700 1 -- 172.21.15.189:0/88556777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f636413e460 msgr2=0x7f6364137640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.899 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6363fff700 1 --2- 172.21.15.189:0/88556777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f636413e460 0x7f6364137640 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f6354009d50 tx=0x7f6354009290).stop 2020-11-20T12:04:51.899 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6363fff700 1 -- 172.21.15.189:0/88556777 shutdown_connections 2020-11-20T12:04:51.899 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6363fff700 1 --2- 172.21.15.189:0/88556777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f636413e460 0x7f6364137640 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.900 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6363fff700 1 --2- 172.21.15.189:0/88556777 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f63640a8a60 0x7f6364067400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.900 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6363fff700 1 -- 172.21.15.189:0/88556777 >> 172.21.15.189:0/88556777 conn(0x7f6364065420 msgr2=0x7f6364062080 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:51.900 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6363fff700 1 -- 172.21.15.189:0/88556777 shutdown_connections 2020-11-20T12:04:51.900 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6363fff700 1 -- 172.21.15.189:0/88556777 wait complete. 2020-11-20T12:04:51.900 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6363fff700 1 Processor -- start 2020-11-20T12:04:51.901 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6363fff700 1 -- start start 2020-11-20T12:04:51.901 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6363fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63640678f0 0x7f63641368a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.901 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6363fff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f63640a8a60 0x7f63640a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.901 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6363fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f636413e460 0x7f636412f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.901 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6363fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636413ba20 con 0x7f636413e460 2020-11-20T12:04:51.902 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6363fff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63640a4290 con 0x7f63640678f0 2020-11-20T12:04:51.902 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6363fff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63640a4a30 con 0x7f63640a8a60 2020-11-20T12:04:51.902 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6362ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63640678f0 0x7f63641368a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.902 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6362ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63640678f0 0x7f63641368a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48192/0 (socket says 172.21.15.189:48192) 2020-11-20T12:04:51.902 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6362ffd700 1 -- 172.21.15.189:0/1087545999 learned_addr learned my addr 172.21.15.189:0/1087545999 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:51.903 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f63637fe700 1 --2- 172.21.15.189:0/1087545999 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f636413e460 0x7f636412f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.903 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f63627fc700 1 --2- 172.21.15.189:0/1087545999 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f63640a8a60 0x7f63640a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.903 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6362ffd700 1 -- 172.21.15.189:0/1087545999 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f63640a8a60 msgr2=0x7f63640a4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.903 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6362ffd700 1 --2- 172.21.15.189:0/1087545999 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f63640a8a60 0x7f63640a4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.903 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6362ffd700 1 -- 172.21.15.189:0/1087545999 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f636413e460 msgr2=0x7f636412f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:51.904 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6362ffd700 1 --2- 172.21.15.189:0/1087545999 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f636413e460 0x7f636412f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:51.904 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6362ffd700 1 -- 172.21.15.189:0/1087545999 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636413ce30 con 0x7f63640678f0 2020-11-20T12:04:51.904 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6362ffd700 1 --2- 172.21.15.189:0/1087545999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63640678f0 0x7f63641368a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f635c00cb80 tx=0x7f635c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:51.904 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f635affd700 1 -- 172.21.15.189:0/1087545999 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635c0077d0 con 0x7f63640678f0 2020-11-20T12:04:51.904 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f635affd700 1 -- 172.21.15.189:0/1087545999 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635c007930 con 0x7f63640678f0 2020-11-20T12:04:51.905 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6363fff700 1 -- 172.21.15.189:0/1087545999 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63640a3d80 con 0x7f63640678f0 2020-11-20T12:04:51.905 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f635affd700 1 -- 172.21.15.189:0/1087545999 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635c007cb0 con 0x7f63640678f0 2020-11-20T12:04:51.905 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.893+0000 7f6363fff700 1 -- 172.21.15.189:0/1087545999 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63640a6c80 con 0x7f63640678f0 2020-11-20T12:04:51.905 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.897+0000 7f635affd700 1 -- 172.21.15.189:0/1087545999 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f635c011620 con 0x7f63640678f0 2020-11-20T12:04:51.905 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.897+0000 7f6363fff700 1 -- 172.21.15.189:0/1087545999 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63640a4100 con 0x7f63640678f0 2020-11-20T12:04:51.906 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.897+0000 7f635affd700 1 --2- 172.21.15.189:0/1087545999 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f63380557c0 0x7f6338057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:51.906 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.897+0000 7f635affd700 1 -- 172.21.15.189:0/1087545999 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f635c06e5e0 con 0x7f63640678f0 2020-11-20T12:04:51.906 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.897+0000 7f63627fc700 1 --2- 172.21.15.189:0/1087545999 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f63380557c0 0x7f6338057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:51.906 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.897+0000 7f63627fc700 1 --2- 172.21.15.189:0/1087545999 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f63380557c0 0x7f6338057c80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f63640a3a80 tx=0x7f634c009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:51.907 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:51.901+0000 7f635affd700 1 -- 172.21.15.189:0/1087545999 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f635c0447b0 con 0x7f63640678f0 2020-11-20T12:04:52.053 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.053+0000 7f6363fff700 1 -- 172.21.15.189:0/1087545999 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f6364044f60 con 0x7f63640678f0 2020-11-20T12:04:52.643 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.641+0000 7f635affd700 1 -- 172.21.15.189:0/1087545999 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v19) v1 ==== 133+0+0 (secure 0 0 0) 0x7f635c047dd0 con 0x7f63640678f0 2020-11-20T12:04:52.647 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.645+0000 7f6363fff700 1 -- 172.21.15.189:0/1087545999 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f63380557c0 msgr2=0x7f6338057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:52.647 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.645+0000 7f6363fff700 1 --2- 172.21.15.189:0/1087545999 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f63380557c0 0x7f6338057c80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f63640a3a80 tx=0x7f634c009040).stop 2020-11-20T12:04:52.647 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.645+0000 7f6363fff700 1 -- 172.21.15.189:0/1087545999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63640678f0 msgr2=0x7f63641368a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:52.648 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.645+0000 7f6363fff700 1 --2- 172.21.15.189:0/1087545999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63640678f0 0x7f63641368a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f635c00cb80 tx=0x7f635c008040).stop 2020-11-20T12:04:52.648 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.645+0000 7f6363fff700 1 -- 172.21.15.189:0/1087545999 shutdown_connections 2020-11-20T12:04:52.648 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.645+0000 7f6363fff700 1 --2- 172.21.15.189:0/1087545999 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f63380557c0 0x7f6338057c80 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:52.648 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.645+0000 7f6363fff700 1 --2- 172.21.15.189:0/1087545999 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f636413e460 0x7f636412f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:52.649 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.645+0000 7f6363fff700 1 --2- 172.21.15.189:0/1087545999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63640678f0 0x7f63641368a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:52.649 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.645+0000 7f6363fff700 1 --2- 172.21.15.189:0/1087545999 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f63640a8a60 0x7f63640a4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:52.649 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.645+0000 7f6363fff700 1 -- 172.21.15.189:0/1087545999 >> 172.21.15.189:0/1087545999 conn(0x7f6364065420 msgr2=0x7f636405ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:52.649 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.645+0000 7f6363fff700 1 -- 172.21.15.189:0/1087545999 shutdown_connections 2020-11-20T12:04:52.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.645+0000 7f6363fff700 1 -- 172.21.15.189:0/1087545999 wait complete. 2020-11-20T12:04:52.650 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_metadata' created 2020-11-20T12:04:52.662 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2020-11-20T12:04:52.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.793+0000 7fb456085700 1 Processor -- start 2020-11-20T12:04:52.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.793+0000 7fb456085700 1 -- start start 2020-11-20T12:04:52.796 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.793+0000 7fb456085700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb45013f2c0 0x7fb45013b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:52.796 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.793+0000 7fb456085700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4500666c0 con 0x7fb45013bc90 2020-11-20T12:04:52.797 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.793+0000 7fb456085700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb450066960 con 0x7fb4500a9cc0 2020-11-20T12:04:52.797 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.793+0000 7fb456085700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb450066d90 con 0x7fb45013f2c0 2020-11-20T12:04:52.797 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.793+0000 7fb44f7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb4500a9cc0 0x7fb45013ebb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49822/0 (socket says 172.21.15.189:49822) 2020-11-20T12:04:52.797 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.793+0000 7fb44f7fe700 1 -- 172.21.15.189:0/3196426601 learned_addr learned my addr 172.21.15.189:0/3196426601 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:04:52.797 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.793+0000 7fb44effd700 1 --2- 172.21.15.189:0/3196426601 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb45013f2c0 0x7fb45013b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:52.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.793+0000 7fb44e7fc700 1 -- 172.21.15.189:0/3196426601 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839070689 0 0) 0x7fb450066960 con 0x7fb4500a9cc0 2020-11-20T12:04:52.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.793+0000 7fb44e7fc700 1 -- 172.21.15.189:0/3196426601 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb43c003240 con 0x7fb4500a9cc0 2020-11-20T12:04:52.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.793+0000 7fb44e7fc700 1 -- 172.21.15.189:0/3196426601 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988561512 0 0) 0x7fb4500666c0 con 0x7fb45013bc90 2020-11-20T12:04:52.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.793+0000 7fb44e7fc700 1 -- 172.21.15.189:0/3196426601 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb450066960 con 0x7fb45013bc90 2020-11-20T12:04:52.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.793+0000 7fb44e7fc700 1 -- 172.21.15.189:0/3196426601 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1018191802 0 0) 0x7fb43c003240 con 0x7fb4500a9cc0 2020-11-20T12:04:52.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb44effd700 1 -- 172.21.15.189:0/3196426601 >> v1:172.21.15.189:6789/0 conn(0x7fb4500a9cc0 legacy=0x7fb45013ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:52.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb44effd700 1 -- 172.21.15.189:0/3196426601 >> v1:172.21.15.47:6789/0 conn(0x7fb45013bc90 legacy=0x7fb450149650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:52.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb44effd700 1 -- 172.21.15.189:0/3196426601 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb450066f80 con 0x7fb45013f2c0 2020-11-20T12:04:52.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb44effd700 1 --2- 172.21.15.189:0/3196426601 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb45013f2c0 0x7fb45013b7a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fb448009c40 tx=0x7fb4480091d0).ready entity=mon.2 client_cookie=3873a9ed016422b6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:52.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb44e7fc700 1 -- 172.21.15.189:0/3196426601 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4440014e0 con 0x7fb45013f2c0 2020-11-20T12:04:52.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb44e7fc700 1 -- 172.21.15.189:0/3196426601 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb44800b730 con 0x7fb45013f2c0 2020-11-20T12:04:52.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb44e7fc700 1 -- 172.21.15.189:0/3196426601 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44800bab0 con 0x7fb45013f2c0 2020-11-20T12:04:52.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb456085700 1 -- 172.21.15.189:0/3196426601 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb45013f2c0 msgr2=0x7fb45013b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:52.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb456085700 1 --2- 172.21.15.189:0/3196426601 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb45013f2c0 0x7fb45013b7a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fb448009c40 tx=0x7fb4480091d0).stop 2020-11-20T12:04:52.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb456085700 1 -- 172.21.15.189:0/3196426601 shutdown_connections 2020-11-20T12:04:52.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb456085700 1 --2- 172.21.15.189:0/3196426601 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb45013f2c0 0x7fb45013b7a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:52.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb456085700 1 -- 172.21.15.189:0/3196426601 >> 172.21.15.189:0/3196426601 conn(0x7fb4500ad440 msgr2=0x7fb4500a9410 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:52.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb456085700 1 -- 172.21.15.189:0/3196426601 shutdown_connections 2020-11-20T12:04:52.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb456085700 1 -- 172.21.15.189:0/3196426601 wait complete. 2020-11-20T12:04:52.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb456085700 1 Processor -- start 2020-11-20T12:04:52.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb456085700 1 -- start start 2020-11-20T12:04:52.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb456085700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb4500a9cc0 0x7fb4501442f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:52.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb456085700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb45013bc90 0x7fb450151650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:52.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb456085700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb45013f2c0 0x7fb450151b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:52.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb456085700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb450152030 con 0x7fb45013bc90 2020-11-20T12:04:52.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb456085700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb450142be0 con 0x7fb45013f2c0 2020-11-20T12:04:52.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb456085700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb450142e80 con 0x7fb4500a9cc0 2020-11-20T12:04:52.806 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb44ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb45013f2c0 0x7fb450151b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:52.806 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb44ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb45013f2c0 0x7fb450151b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48208/0 (socket says 172.21.15.189:48208) 2020-11-20T12:04:52.806 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb44ffff700 1 -- 172.21.15.189:0/744744706 learned_addr learned my addr 172.21.15.189:0/744744706 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:52.806 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb44f7fe700 1 --2- 172.21.15.189:0/744744706 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb4500a9cc0 0x7fb4501442f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:52.806 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.797+0000 7fb44effd700 1 --2- 172.21.15.189:0/744744706 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb45013bc90 0x7fb450151650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:52.807 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.801+0000 7fb44ffff700 1 -- 172.21.15.189:0/744744706 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb4500a9cc0 msgr2=0x7fb4501442f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:52.807 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.801+0000 7fb44ffff700 1 --2- 172.21.15.189:0/744744706 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb4500a9cc0 0x7fb4501442f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:52.807 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.801+0000 7fb44ffff700 1 -- 172.21.15.189:0/744744706 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb45013bc90 msgr2=0x7fb450151650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:52.807 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.801+0000 7fb44ffff700 1 --2- 172.21.15.189:0/744744706 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb45013bc90 0x7fb450151650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:52.807 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.801+0000 7fb44ffff700 1 -- 172.21.15.189:0/744744706 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4501511b0 con 0x7fb45013f2c0 2020-11-20T12:04:52.808 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.801+0000 7fb44ffff700 1 --2- 172.21.15.189:0/744744706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb45013f2c0 0x7fb450151b40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fb44000c740 tx=0x7fb440007a50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:52.808 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.801+0000 7fb4377fe700 1 -- 172.21.15.189:0/744744706 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44000e490 con 0x7fb45013f2c0 2020-11-20T12:04:52.808 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.801+0000 7fb4377fe700 1 -- 172.21.15.189:0/744744706 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb44000e5f0 con 0x7fb45013f2c0 2020-11-20T12:04:52.808 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.801+0000 7fb4377fe700 1 -- 172.21.15.189:0/744744706 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44000e970 con 0x7fb45013f2c0 2020-11-20T12:04:52.808 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.801+0000 7fb456085700 1 -- 172.21.15.189:0/744744706 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb450144b70 con 0x7fb45013f2c0 2020-11-20T12:04:52.809 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.801+0000 7fb456085700 1 -- 172.21.15.189:0/744744706 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4501428f0 con 0x7fb45013f2c0 2020-11-20T12:04:52.809 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.801+0000 7fb456085700 1 -- 172.21.15.189:0/744744706 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb450151340 con 0x7fb45013f2c0 2020-11-20T12:04:52.809 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.801+0000 7fb4377fe700 1 -- 172.21.15.189:0/744744706 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fb440019030 con 0x7fb45013f2c0 2020-11-20T12:04:52.810 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.801+0000 7fb4377fe700 1 --2- 172.21.15.189:0/744744706 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb42c0557f0 0x7fb42c057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:52.810 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.801+0000 7fb4377fe700 1 -- 172.21.15.189:0/744744706 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5607+0+0 (secure 0 0 0) 0x7fb44006d880 con 0x7fb45013f2c0 2020-11-20T12:04:52.811 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.801+0000 7fb44f7fe700 1 --2- 172.21.15.189:0/744744706 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb42c0557f0 0x7fb42c057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:52.811 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.801+0000 7fb44f7fe700 1 --2- 172.21.15.189:0/744744706 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb42c0557f0 0x7fb42c057cb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fb4440011a0 tx=0x7fb444000b80).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:52.811 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.805+0000 7fb4377fe700 1 -- 172.21.15.189:0/744744706 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb4400439a0 con 0x7fb45013f2c0 2020-11-20T12:04:52.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:52.957+0000 7fb456085700 1 -- 172.21.15.189:0/744744706 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7fb450044f60 con 0x7fb45013f2c0 2020-11-20T12:04:53.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.653+0000 7fb4377fe700 1 -- 172.21.15.189:0/744744706 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v20) v1 ==== 125+0+0 (secure 0 0 0) 0x7fb440002e60 con 0x7fb45013f2c0 2020-11-20T12:04:53.659 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.657+0000 7fb456085700 1 -- 172.21.15.189:0/744744706 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb42c0557f0 msgr2=0x7fb42c057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:53.659 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.657+0000 7fb456085700 1 --2- 172.21.15.189:0/744744706 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb42c0557f0 0x7fb42c057cb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fb4440011a0 tx=0x7fb444000b80).stop 2020-11-20T12:04:53.659 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.657+0000 7fb456085700 1 -- 172.21.15.189:0/744744706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb45013f2c0 msgr2=0x7fb450151b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:53.660 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.657+0000 7fb456085700 1 --2- 172.21.15.189:0/744744706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb45013f2c0 0x7fb450151b40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fb44000c740 tx=0x7fb440007a50).stop 2020-11-20T12:04:53.660 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.657+0000 7fb456085700 1 -- 172.21.15.189:0/744744706 shutdown_connections 2020-11-20T12:04:53.661 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.657+0000 7fb456085700 1 --2- 172.21.15.189:0/744744706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb45013f2c0 0x7fb450151b40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:53.661 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.657+0000 7fb456085700 1 --2- 172.21.15.189:0/744744706 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb4500a9cc0 0x7fb4501442f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:53.662 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.657+0000 7fb456085700 1 --2- 172.21.15.189:0/744744706 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb42c0557f0 0x7fb42c057cb0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:53.662 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.657+0000 7fb456085700 1 --2- 172.21.15.189:0/744744706 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb45013bc90 0x7fb450151650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:53.662 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.657+0000 7fb456085700 1 -- 172.21.15.189:0/744744706 >> 172.21.15.189:0/744744706 conn(0x7fb4500ad440 msgr2=0x7fb4500a9270 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:53.662 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.657+0000 7fb456085700 1 -- 172.21.15.189:0/744744706 shutdown_connections 2020-11-20T12:04:53.663 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.657+0000 7fb456085700 1 -- 172.21.15.189:0/744744706 wait complete. 2020-11-20T12:04:53.663 INFO:teuthology.orchestra.run.smithi189.stderr:pool 'cephfs_data' created 2020-11-20T12:04:53.674 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2020-11-20T12:04:53.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.797+0000 7fcc6e018700 1 Processor -- start 2020-11-20T12:04:53.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.797+0000 7fcc6e018700 1 -- start start 2020-11-20T12:04:53.801 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc6e018700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcc680ae1b0 0x7fcc680a6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:53.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc6e018700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc681358b0 0x7fcc680a7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:53.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc6e018700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc68135d10 con 0x7fcc68060f80 2020-11-20T12:04:53.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc6e018700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc680a7930 con 0x7fcc680ae1b0 2020-11-20T12:04:53.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc6e018700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc680a86d0 con 0x7fcc681358b0 2020-11-20T12:04:53.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc6d016700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcc680ae1b0 0x7fcc680a6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:53.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc6c815700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fcc68060f80 0x7fcc681351a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49838/0 (socket says 172.21.15.189:49838) 2020-11-20T12:04:53.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc6c815700 1 -- 172.21.15.189:0/3504519990 learned_addr learned my addr 172.21.15.189:0/3504519990 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:04:53.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc6d817700 1 --2- 172.21.15.189:0/3504519990 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc681358b0 0x7fcc680a7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:53.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc6d016700 1 -- 172.21.15.189:0/3504519990 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc681358b0 msgr2=0x7fcc680a7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:53.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc67fff700 1 -- 172.21.15.189:0/3504519990 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1578347026 0 0) 0x7fcc68135d10 con 0x7fcc68060f80 2020-11-20T12:04:53.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc6d016700 1 --2- 172.21.15.189:0/3504519990 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc681358b0 0x7fcc680a7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:53.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc6d016700 1 -- 172.21.15.189:0/3504519990 >> v1:172.21.15.189:6789/0 conn(0x7fcc68060f80 legacy=0x7fcc681351a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:53.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc6d016700 1 -- 172.21.15.189:0/3504519990 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc680a8970 con 0x7fcc680ae1b0 2020-11-20T12:04:53.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc6d817700 1 --2- 172.21.15.189:0/3504519990 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc681358b0 0x7fcc680a7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:53.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc6d016700 1 --2- 172.21.15.189:0/3504519990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc680ae1b0 0x7fcc680a6f50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fcc60009c90 tx=0x7fcc600091d0).ready entity=mon.1 client_cookie=d09dd428913373af server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:53.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc67fff700 1 -- 172.21.15.189:0/3504519990 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6000bbc0 con 0x7fcc680ae1b0 2020-11-20T12:04:53.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc67fff700 1 -- 172.21.15.189:0/3504519990 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc6000bd20 con 0x7fcc680ae1b0 2020-11-20T12:04:53.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc67fff700 1 -- 172.21.15.189:0/3504519990 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc60011610 con 0x7fcc680ae1b0 2020-11-20T12:04:53.808 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc6e018700 1 -- 172.21.15.189:0/3504519990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc680ae1b0 msgr2=0x7fcc680a6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:53.808 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc6e018700 1 --2- 172.21.15.189:0/3504519990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc680ae1b0 0x7fcc680a6f50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fcc60009c90 tx=0x7fcc600091d0).stop 2020-11-20T12:04:53.809 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc6e018700 1 -- 172.21.15.189:0/3504519990 shutdown_connections 2020-11-20T12:04:53.809 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc6e018700 1 --2- 172.21.15.189:0/3504519990 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc681358b0 0x7fcc680a7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:53.809 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc6e018700 1 --2- 172.21.15.189:0/3504519990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc680ae1b0 0x7fcc680a6f50 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0x7fcc60009c90 tx=0x7fcc600091d0).stop 2020-11-20T12:04:53.809 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc6e018700 1 -- 172.21.15.189:0/3504519990 >> 172.21.15.189:0/3504519990 conn(0x7fcc68065fe0 msgr2=0x7fcc680ad900 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:53.810 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc6e018700 1 -- 172.21.15.189:0/3504519990 shutdown_connections 2020-11-20T12:04:53.810 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc6e018700 1 -- 172.21.15.189:0/3504519990 wait complete. 2020-11-20T12:04:53.810 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc6e018700 1 Processor -- start 2020-11-20T12:04:53.810 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.801+0000 7fcc6e018700 1 -- start start 2020-11-20T12:04:53.811 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc6e018700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc68060f80 0x7fcc68138840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:53.811 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc6e018700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc681358b0 0x7fcc68130790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:53.811 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc6e018700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc681476e0 0x7fcc68137df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:53.811 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc6e018700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc68137720 con 0x7fcc681358b0 2020-11-20T12:04:53.811 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc6e018700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc68138d30 con 0x7fcc681476e0 2020-11-20T12:04:53.812 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc6e018700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc68138fc0 con 0x7fcc68060f80 2020-11-20T12:04:53.812 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc6d817700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc681476e0 0x7fcc68137df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:53.812 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc6d817700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc681476e0 0x7fcc68137df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48224/0 (socket says 172.21.15.189:48224) 2020-11-20T12:04:53.812 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc6d817700 1 -- 172.21.15.189:0/4018427411 learned_addr learned my addr 172.21.15.189:0/4018427411 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:53.813 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc6d016700 1 --2- 172.21.15.189:0/4018427411 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc68060f80 0x7fcc68138840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:53.813 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc6c815700 1 --2- 172.21.15.189:0/4018427411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc681358b0 0x7fcc68130790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:53.813 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc6d817700 1 -- 172.21.15.189:0/4018427411 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc68060f80 msgr2=0x7fcc68138840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:53.813 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc6d817700 1 --2- 172.21.15.189:0/4018427411 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc68060f80 0x7fcc68138840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:53.814 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc6d817700 1 -- 172.21.15.189:0/4018427411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc681358b0 msgr2=0x7fcc68130790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:53.814 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc6d817700 1 --2- 172.21.15.189:0/4018427411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc681358b0 0x7fcc68130790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:53.814 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc6d817700 1 -- 172.21.15.189:0/4018427411 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc68137950 con 0x7fcc681476e0 2020-11-20T12:04:53.814 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc6c815700 1 --2- 172.21.15.189:0/4018427411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc681358b0 0x7fcc68130790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:53.814 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc6d817700 1 --2- 172.21.15.189:0/4018427411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc681476e0 0x7fcc68137df0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fcc5800d840 tx=0x7fcc58008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:53.815 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc657fa700 1 -- 172.21.15.189:0/4018427411 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc58009630 con 0x7fcc681476e0 2020-11-20T12:04:53.815 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc657fa700 1 -- 172.21.15.189:0/4018427411 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc58009790 con 0x7fcc681476e0 2020-11-20T12:04:53.815 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc6e018700 1 -- 172.21.15.189:0/4018427411 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc68130c80 con 0x7fcc681476e0 2020-11-20T12:04:53.815 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc657fa700 1 -- 172.21.15.189:0/4018427411 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc58009b30 con 0x7fcc681476e0 2020-11-20T12:04:53.815 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc6e018700 1 -- 172.21.15.189:0/4018427411 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc6813ad60 con 0x7fcc681476e0 2020-11-20T12:04:53.816 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc657fa700 1 -- 172.21.15.189:0/4018427411 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fcc58013480 con 0x7fcc681476e0 2020-11-20T12:04:53.816 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc6e018700 1 -- 172.21.15.189:0/4018427411 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc6813b040 con 0x7fcc681476e0 2020-11-20T12:04:53.816 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc657fa700 1 --2- 172.21.15.189:0/4018427411 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcc400557c0 0x7fcc40057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:53.816 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc657fa700 1 -- 172.21.15.189:0/4018427411 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5957+0+0 (secure 0 0 0) 0x7fcc5806e9b0 con 0x7fcc681476e0 2020-11-20T12:04:53.816 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.805+0000 7fcc6d016700 1 --2- 172.21.15.189:0/4018427411 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcc400557c0 0x7fcc40057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:53.817 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.809+0000 7fcc6d016700 1 --2- 172.21.15.189:0/4018427411 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcc400557c0 0x7fcc40057c80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fcc68065500 tx=0x7fcc60016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:53.817 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:53.813+0000 7fcc657fa700 1 -- 172.21.15.189:0/4018427411 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcc58044860 con 0x7fcc681476e0 2020-11-20T12:04:54.009 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.009+0000 7fcc6e018700 1 -- 172.21.15.189:0/4018427411 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7fcc68137ae0 con 0x7fcc681476e0 2020-11-20T12:04:54.024 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.025+0000 7fcc657fa700 1 -- 172.21.15.189:0/4018427411 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 187+0+0 (secure 0 0 0) 0x7fcc58047e80 con 0x7fcc681476e0 2020-11-20T12:04:54.027 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.025+0000 7fcc6e018700 1 -- 172.21.15.189:0/4018427411 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcc400557c0 msgr2=0x7fcc40057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.028 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.025+0000 7fcc6e018700 1 --2- 172.21.15.189:0/4018427411 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcc400557c0 0x7fcc40057c80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fcc68065500 tx=0x7fcc60016000).stop 2020-11-20T12:04:54.028 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.025+0000 7fcc6e018700 1 -- 172.21.15.189:0/4018427411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc681476e0 msgr2=0x7fcc68137df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.028 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.025+0000 7fcc6e018700 1 --2- 172.21.15.189:0/4018427411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc681476e0 0x7fcc68137df0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fcc5800d840 tx=0x7fcc58008040).stop 2020-11-20T12:04:54.029 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.029+0000 7fcc6e018700 1 -- 172.21.15.189:0/4018427411 shutdown_connections 2020-11-20T12:04:54.029 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.029+0000 7fcc6e018700 1 --2- 172.21.15.189:0/4018427411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc681476e0 0x7fcc68137df0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.030 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.029+0000 7fcc6e018700 1 --2- 172.21.15.189:0/4018427411 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc68060f80 0x7fcc68138840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.030 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.029+0000 7fcc6e018700 1 --2- 172.21.15.189:0/4018427411 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcc400557c0 0x7fcc40057c80 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.030 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.029+0000 7fcc6e018700 1 --2- 172.21.15.189:0/4018427411 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc681358b0 0x7fcc68130790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.030 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.029+0000 7fcc6e018700 1 -- 172.21.15.189:0/4018427411 >> 172.21.15.189:0/4018427411 conn(0x7fcc68065fe0 msgr2=0x7fcc680a6f50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:54.030 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.029+0000 7fcc6e018700 1 -- 172.21.15.189:0/4018427411 shutdown_connections 2020-11-20T12:04:54.031 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.029+0000 7fcc6e018700 1 -- 172.21.15.189:0/4018427411 wait complete. 2020-11-20T12:04:54.031 INFO:teuthology.orchestra.run.smithi189.stderr:new fs with metadata pool 3 and data pool 4 2020-11-20T12:04:54.042 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:04:54.154 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.153+0000 7f7068872700 1 Processor -- start 2020-11-20T12:04:54.154 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.153+0000 7f7068872700 1 -- start start 2020-11-20T12:04:54.157 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.153+0000 7f7068872700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f70640ad000 0x7f706413e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.157 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.153+0000 7f7068872700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f706413a6e0 0x7f706413ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.157 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.153+0000 7f7068872700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7064067730 con 0x7f70640ad3e0 2020-11-20T12:04:54.157 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.153+0000 7f7068872700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7064138190 con 0x7f706413a6e0 2020-11-20T12:04:54.158 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.153+0000 7f7068872700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f706413dd40 con 0x7f70640ad000 2020-11-20T12:04:54.158 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.153+0000 7f706259c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f70640ad3e0 0x7f7064066d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49860/0 (socket says 172.21.15.189:49860) 2020-11-20T12:04:54.158 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.153+0000 7f706259c700 1 -- 172.21.15.189:0/2360729815 learned_addr learned my addr 172.21.15.189:0/2360729815 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:04:54.158 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.153+0000 7f706359e700 1 --2- 172.21.15.189:0/2360729815 >> v2:172.21.15.189:3300/0 conn(0x7f706413a6e0 0x7f706413ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.158 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.153+0000 7f7062d9d700 1 --2- 172.21.15.189:0/2360729815 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f70640ad000 0x7f706413e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.159 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.153+0000 7f7061d9b700 1 -- 172.21.15.189:0/2360729815 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150406112 0 0) 0x7f7064067730 con 0x7f70640ad3e0 2020-11-20T12:04:54.159 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.153+0000 7f7061d9b700 1 -- 172.21.15.189:0/2360729815 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7044003240 con 0x7f70640ad3e0 2020-11-20T12:04:54.159 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.153+0000 7f706359e700 1 -- 172.21.15.189:0/2360729815 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f70640ad000 msgr2=0x7f706413e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:54.159 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.153+0000 7f706359e700 1 --2- 172.21.15.189:0/2360729815 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f70640ad000 0x7f706413e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.159 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.153+0000 7f706359e700 1 -- 172.21.15.189:0/2360729815 >> v1:172.21.15.189:6789/0 conn(0x7f70640ad3e0 legacy=0x7f7064066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.160 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.153+0000 7f706359e700 1 -- 172.21.15.189:0/2360729815 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706413dfe0 con 0x7f706413a6e0 2020-11-20T12:04:54.160 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.153+0000 7f7062d9d700 1 --2- 172.21.15.189:0/2360729815 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f70640ad000 0x7f706413e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:54.160 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.153+0000 7f706359e700 1 --2- 172.21.15.189:0/2360729815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f706413a6e0 0x7f706413ad00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f705400b9a0 tx=0x7f7054006df0).ready entity=mon.1 client_cookie=262d9b94600341 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:54.160 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.153+0000 7f7061d9b700 1 -- 172.21.15.189:0/2360729815 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70540097b0 con 0x7f706413a6e0 2020-11-20T12:04:54.160 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.153+0000 7f7061d9b700 1 -- 172.21.15.189:0/2360729815 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7054009910 con 0x7f706413a6e0 2020-11-20T12:04:54.161 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.153+0000 7f7061d9b700 1 -- 172.21.15.189:0/2360729815 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7054009c90 con 0x7f706413a6e0 2020-11-20T12:04:54.161 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.157+0000 7f7068872700 1 -- 172.21.15.189:0/2360729815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f706413a6e0 msgr2=0x7f706413ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.161 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.157+0000 7f7068872700 1 --2- 172.21.15.189:0/2360729815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f706413a6e0 0x7f706413ad00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f705400b9a0 tx=0x7f7054006df0).stop 2020-11-20T12:04:54.161 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.157+0000 7f7068872700 1 -- 172.21.15.189:0/2360729815 shutdown_connections 2020-11-20T12:04:54.161 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.157+0000 7f7068872700 1 --2- 172.21.15.189:0/2360729815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f706413a6e0 0x7f706413ad00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.162 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.157+0000 7f7068872700 1 --2- 172.21.15.189:0/2360729815 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f70640ad000 0x7f706413e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.162 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.157+0000 7f7068872700 1 -- 172.21.15.189:0/2360729815 >> 172.21.15.189:0/2360729815 conn(0x7f7064062080 msgr2=0x7f7064062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:54.162 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.157+0000 7f7068872700 1 -- 172.21.15.189:0/2360729815 shutdown_connections 2020-11-20T12:04:54.162 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.157+0000 7f7068872700 1 -- 172.21.15.189:0/2360729815 wait complete. 2020-11-20T12:04:54.165 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.157+0000 7f7068872700 1 Processor -- start 2020-11-20T12:04:54.165 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.157+0000 7f7068872700 1 -- start start 2020-11-20T12:04:54.166 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.161+0000 7f7068872700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f70640ad000 0x7f70640a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.166 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.161+0000 7f7068872700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f70640ad3e0 0x7f70640a1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.166 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.161+0000 7f7068872700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70640aaf50 0x7f7064149980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.166 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.161+0000 7f7068872700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70640a3410 con 0x7f70640ad3e0 2020-11-20T12:04:54.166 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.161+0000 7f7068872700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70640a1b30 con 0x7f70640aaf50 2020-11-20T12:04:54.167 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.161+0000 7f7068872700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7064149440 con 0x7f70640ad000 2020-11-20T12:04:54.167 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.161+0000 7f706359e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70640aaf50 0x7f7064149980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.167 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.161+0000 7f706359e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70640aaf50 0x7f7064149980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48248/0 (socket says 172.21.15.189:48248) 2020-11-20T12:04:54.167 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.161+0000 7f706359e700 1 -- 172.21.15.189:0/4224713529 learned_addr learned my addr 172.21.15.189:0/4224713529 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:54.167 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.161+0000 7f7062d9d700 1 --2- 172.21.15.189:0/4224713529 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f70640ad000 0x7f70640a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.168 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.161+0000 7f706259c700 1 --2- 172.21.15.189:0/4224713529 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f70640ad3e0 0x7f70640a1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.168 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.161+0000 7f706359e700 1 -- 172.21.15.189:0/4224713529 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f70640ad000 msgr2=0x7f70640a4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.168 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.161+0000 7f706359e700 1 --2- 172.21.15.189:0/4224713529 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f70640ad000 0x7f70640a4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.168 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.161+0000 7f706359e700 1 -- 172.21.15.189:0/4224713529 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f70640ad3e0 msgr2=0x7f70640a1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.168 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.161+0000 7f706359e700 1 --2- 172.21.15.189:0/4224713529 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f70640ad3e0 0x7f70640a1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.169 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.161+0000 7f706359e700 1 -- 172.21.15.189:0/4224713529 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70641495b0 con 0x7f70640aaf50 2020-11-20T12:04:54.169 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.161+0000 7f7062d9d700 1 --2- 172.21.15.189:0/4224713529 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f70640ad000 0x7f70640a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:54.169 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.161+0000 7f706359e700 1 --2- 172.21.15.189:0/4224713529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70640aaf50 0x7f7064149980 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f7054006e30 tx=0x7f7054015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:54.169 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.161+0000 7f705affd700 1 -- 172.21.15.189:0/4224713529 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7054006b20 con 0x7f70640aaf50 2020-11-20T12:04:54.169 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.161+0000 7f705affd700 1 -- 172.21.15.189:0/4224713529 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705401d5d0 con 0x7f70640aaf50 2020-11-20T12:04:54.170 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.161+0000 7f705affd700 1 -- 172.21.15.189:0/4224713529 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705401d950 con 0x7f70640aaf50 2020-11-20T12:04:54.170 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.161+0000 7f7068872700 1 -- 172.21.15.189:0/4224713529 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70640a0df0 con 0x7f70640aaf50 2020-11-20T12:04:54.170 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.161+0000 7f7068872700 1 -- 172.21.15.189:0/4224713529 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70640a24c0 con 0x7f70640aaf50 2020-11-20T12:04:54.170 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.161+0000 7f7068872700 1 -- 172.21.15.189:0/4224713529 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70640a5370 con 0x7f70640aaf50 2020-11-20T12:04:54.171 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.161+0000 7f705affd700 1 -- 172.21.15.189:0/4224713529 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f7054025030 con 0x7f70640aaf50 2020-11-20T12:04:54.171 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.165+0000 7f705affd700 1 --2- 172.21.15.189:0/4224713529 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f70400557c0 0x7f7040057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.171 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.165+0000 7f705affd700 1 -- 172.21.15.189:0/4224713529 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7054070620 con 0x7f70640aaf50 2020-11-20T12:04:54.171 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.165+0000 7f7062d9d700 1 --2- 172.21.15.189:0/4224713529 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f70400557c0 0x7f7040057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.171 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.165+0000 7f7062d9d700 1 --2- 172.21.15.189:0/4224713529 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f70400557c0 0x7f7040057c80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f704c00bac0 tx=0x7f704c008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:54.172 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.169+0000 7f705affd700 1 -- 172.21.15.189:0/4224713529 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7054046520 con 0x7f70640aaf50 2020-11-20T12:04:54.317 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.317+0000 7f7068872700 1 -- 172.21.15.189:0/4224713529 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70641495b0 con 0x7f70640aaf50 2020-11-20T12:04:54.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.317+0000 7f705affd700 1 -- 172.21.15.189:0/4224713529 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16839 (secure 0 0 0) 0x7f70641495b0 con 0x7f70640aaf50 2020-11-20T12:04:54.319 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:54.319 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":21,"fsid":"cda18d0e-64b3-41bb-964a-084069c3e648","created":"2020-11-20T12:04:37.538706+0000","modified":"2020-11-20T12:04:54.011613+0000","last_up_change":"2020-11-20T12:04:43.006631+0000","last_in_change":"2020-11-20T12:04:43.006631+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:41.395395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:48.550454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:52.101837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:53.018196+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c81214e1-4f84-4aaa-8ceb-99c398b2a89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6811","nonce":11760},{"type":"v1","addr":"172.21.15.189:6814","nonce":11760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":11760},{"type":"v1","addr":"172.21.15.189:6818","nonce":11760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":11760},{"type":"v1","addr":"172.21.15.189:6828","nonce":11760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6821","nonce":11760},{"type":"v1","addr":"172.21.15.189:6824","nonce":11760}]},"public_addr":"172.21.15.189:6814/11760","cluster_addr":"172.21.15.189:6818/11760","heartbeat_back_addr":"172.21.15.189:6828/11760","heartbeat_front_addr":"172.21.15.189:6824/11760","state":["exists","up"]},{"osd":1,"uuid":"d2f3ec7d-d403-484f-b5ed-b2e6b079d63d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6805","nonce":11761},{"type":"v1","addr":"172.21.15.189:6807","nonce":11761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":11761},{"type":"v1","addr":"172.21.15.189:6812","nonce":11761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6817","nonce":11761},{"type":"v1","addr":"172.21.15.189:6820","nonce":11761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6813","nonce":11761},{"type":"v1","addr":"172.21.15.189:6815","nonce":11761}]},"public_addr":"172.21.15.189:6807/11761","cluster_addr":"172.21.15.189:6812/11761","heartbeat_back_addr":"172.21.15.189:6820/11761","heartbeat_front_addr":"172.21.15.189:6815/11761","state":["exists","up"]},{"osd":2,"uuid":"e6cf4790-000c-43eb-8844-175f04c60503","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6819","nonce":11751},{"type":"v1","addr":"172.21.15.189:6822","nonce":11751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":11751},{"type":"v1","addr":"172.21.15.189:6825","nonce":11751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":11751},{"type":"v1","addr":"172.21.15.189:6831","nonce":11751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6827","nonce":11751},{"type":"v1","addr":"172.21.15.189:6829","nonce":11751}]},"public_addr":"172.21.15.189:6822/11751","cluster_addr":"172.21.15.189:6825/11751","heartbeat_back_addr":"172.21.15.189:6831/11751","heartbeat_front_addr":"172.21.15.189:6829/11751","state":["exists","up"]},{"osd":3,"uuid":"ba49dfac-f2b9-476e-853d-42f05e21a77a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":11762},{"type":"v1","addr":"172.21.15.189:6801","nonce":11762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":11762},{"type":"v1","addr":"172.21.15.189:6803","nonce":11762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":11762},{"type":"v1","addr":"172.21.15.189:6809","nonce":11762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":11762},{"type":"v1","addr":"172.21.15.189:6806","nonce":11762}]},"public_addr":"172.21.15.189:6801/11762","cluster_addr":"172.21.15.189:6803/11762","heartbeat_back_addr":"172.21.15.189:6809/11762","heartbeat_front_addr":"172.21.15.189:6806/11762","state":["exists","up"]},{"osd":4,"uuid":"3ca7b438-38b2-4689-8b18-fb5a8ddf6aee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":12114},{"type":"v1","addr":"172.21.15.47:6825","nonce":12114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":12114},{"type":"v1","addr":"172.21.15.47:6827","nonce":12114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":12114},{"type":"v1","addr":"172.21.15.47:6831","nonce":12114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":12114},{"type":"v1","addr":"172.21.15.47:6829","nonce":12114}]},"public_addr":"172.21.15.47:6825/12114","cluster_addr":"172.21.15.47:6827/12114","heartbeat_back_addr":"172.21.15.47:6831/12114","heartbeat_front_addr":"172.21.15.47:6829/12114","state":["exists","up"]},{"osd":5,"uuid":"6a45fca7-83d3-44fd-a1d4-b01e4201a101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":12111},{"type":"v1","addr":"172.21.15.47:6801","nonce":12111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":12111},{"type":"v1","addr":"172.21.15.47:6803","nonce":12111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":12111},{"type":"v1","addr":"172.21.15.47:6807","nonce":12111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":12111},{"type":"v1","addr":"172.21.15.47:6805","nonce":12111}]},"public_addr":"172.21.15.47:6801/12111","cluster_addr":"172.21.15.47:6803/12111","heartbeat_back_addr":"172.21.15.47:6807/12111","heartbeat_front_addr":"172.21.15.47:6805/12111","state":["exists","up"]},{"osd":6,"uuid":"c4b1b9ed-61f8-4677-9755-dff59a30e447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":12113},{"type":"v1","addr":"172.21.15.47:6817","nonce":12113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":12113},{"type":"v1","addr":"172.21.15.47:6819","nonce":12113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":12113},{"type":"v1","addr":"172.21.15.47:6823","nonce":12113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":12113},{"type":"v1","addr":"172.21.15.47:6821","nonce":12113}]},"public_addr":"172.21.15.47:6817/12113","cluster_addr":"172.21.15.47:6819/12113","heartbeat_back_addr":"172.21.15.47:6823/12113","heartbeat_front_addr":"172.21.15.47:6821/12113","state":["exists","up"]},{"osd":7,"uuid":"4608e4ed-cc40-4c5c-ba8b-3779bbc62f65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":12112},{"type":"v1","addr":"172.21.15.47:6809","nonce":12112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":12112},{"type":"v1","addr":"172.21.15.47:6811","nonce":12112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":12112},{"type":"v1","addr":"172.21.15.47:6815","nonce":12112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":12112},{"type":"v1","addr":"172.21.15.47:6813","nonce":12112}]},"public_addr":"172.21.15.47:6809/12112","cluster_addr":"172.21.15.47:6811/12112","heartbeat_back_addr":"172.21.15.47:6815/12112","heartbeat_front_addr":"172.21.15.47:6813/12112","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.435265+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.472776+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.388370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.417366+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.102392+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.116529+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.167430+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.148139+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:04:54.321 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.321+0000 7f7068872700 1 -- 172.21.15.189:0/4224713529 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f70400557c0 msgr2=0x7f7040057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.321 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.321+0000 7f7068872700 1 --2- 172.21.15.189:0/4224713529 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f70400557c0 0x7f7040057c80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f704c00bac0 tx=0x7f704c008040).stop 2020-11-20T12:04:54.321 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.321+0000 7f7068872700 1 -- 172.21.15.189:0/4224713529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70640aaf50 msgr2=0x7f7064149980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.321 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.321+0000 7f7068872700 1 --2- 172.21.15.189:0/4224713529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70640aaf50 0x7f7064149980 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f7054006e30 tx=0x7f7054015000).stop 2020-11-20T12:04:54.322 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.321+0000 7f7068872700 1 -- 172.21.15.189:0/4224713529 shutdown_connections 2020-11-20T12:04:54.322 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.321+0000 7f7068872700 1 --2- 172.21.15.189:0/4224713529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70640aaf50 0x7f7064149980 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.322 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.321+0000 7f7068872700 1 --2- 172.21.15.189:0/4224713529 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f70640ad000 0x7f70640a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.322 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.321+0000 7f7068872700 1 --2- 172.21.15.189:0/4224713529 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f70400557c0 0x7f7040057c80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.322 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.321+0000 7f7068872700 1 --2- 172.21.15.189:0/4224713529 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f70640ad3e0 0x7f70640a1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.323 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.321+0000 7f7068872700 1 -- 172.21.15.189:0/4224713529 >> 172.21.15.189:0/4224713529 conn(0x7f7064062080 msgr2=0x7f706405ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:54.323 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.321+0000 7f7068872700 1 -- 172.21.15.189:0/4224713529 shutdown_connections 2020-11-20T12:04:54.323 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.321+0000 7f7068872700 1 -- 172.21.15.189:0/4224713529 wait complete. 2020-11-20T12:04:54.334 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:04:54.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.449+0000 7f498759e700 1 Processor -- start 2020-11-20T12:04:54.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.449+0000 7f498759e700 1 -- start start 2020-11-20T12:04:54.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.449+0000 7f498759e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4988067500 0x7f4988133460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.449+0000 7f498759e700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f49880a8780 0x7f4988133950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.449+0000 7f498759e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49880679a0 con 0x7f49880612b0 2020-11-20T12:04:54.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.449+0000 7f498759e700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4988067c40 con 0x7f49880a8780 2020-11-20T12:04:54.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.449+0000 7f498759e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4988067ee0 con 0x7f4988067500 2020-11-20T12:04:54.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.449+0000 7f4985d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4988067500 0x7f4988133460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.449+0000 7f4985d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4988067500 0x7f4988133460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48254/0 (socket says 172.21.15.189:48254) 2020-11-20T12:04:54.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.449+0000 7f4985d9b700 1 -- 172.21.15.189:0/2301371697 learned_addr learned my addr 172.21.15.189:0/2301371697 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:54.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.449+0000 7f4986d9d700 1 --2- 172.21.15.189:0/2301371697 >> v2:172.21.15.47:3300/0 conn(0x7f49880a8780 0x7f4988133950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.449+0000 7f4985d9b700 1 -- 172.21.15.189:0/2301371697 >> v2:172.21.15.47:3300/0 conn(0x7f49880a8780 msgr2=0x7f4988133950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:54.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.449+0000 7f4985d9b700 1 --2- 172.21.15.189:0/2301371697 >> v2:172.21.15.47:3300/0 conn(0x7f49880a8780 0x7f4988133950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.449+0000 7f4985d9b700 1 -- 172.21.15.189:0/2301371697 >> v1:172.21.15.47:6789/0 conn(0x7f49880612b0 legacy=0x7f49880a8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.449+0000 7f4985d9b700 1 -- 172.21.15.189:0/2301371697 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4988061690 con 0x7f4988067500 2020-11-20T12:04:54.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.449+0000 7f4985d9b700 1 --2- 172.21.15.189:0/2301371697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4988067500 0x7f4988133460 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f4988067000 tx=0x7f4978009290).ready entity=mon.1 client_cookie=f723dfb7cbdda2d4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:54.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.449+0000 7f498559a700 1 -- 172.21.15.189:0/2301371697 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497800bc50 con 0x7f4988067500 2020-11-20T12:04:54.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.449+0000 7f498559a700 1 -- 172.21.15.189:0/2301371697 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f497800bdb0 con 0x7f4988067500 2020-11-20T12:04:54.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.449+0000 7f498559a700 1 -- 172.21.15.189:0/2301371697 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4978011610 con 0x7f4988067500 2020-11-20T12:04:54.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.449+0000 7f498759e700 1 -- 172.21.15.189:0/2301371697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4988067500 msgr2=0x7f4988133460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.449+0000 7f498759e700 1 --2- 172.21.15.189:0/2301371697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4988067500 0x7f4988133460 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f4988067000 tx=0x7f4978009290).stop 2020-11-20T12:04:54.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.449+0000 7f498759e700 1 -- 172.21.15.189:0/2301371697 shutdown_connections 2020-11-20T12:04:54.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.449+0000 7f498759e700 1 --2- 172.21.15.189:0/2301371697 >> v2:172.21.15.47:3300/0 conn(0x7f49880a8780 0x7f4988133950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.449+0000 7f498759e700 1 --2- 172.21.15.189:0/2301371697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4988067500 0x7f4988133460 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.449+0000 7f498759e700 1 -- 172.21.15.189:0/2301371697 >> 172.21.15.189:0/2301371697 conn(0x7f49880625f0 msgr2=0x7f4988060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:54.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.453+0000 7f498759e700 1 -- 172.21.15.189:0/2301371697 shutdown_connections 2020-11-20T12:04:54.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.453+0000 7f498759e700 1 -- 172.21.15.189:0/2301371697 wait complete. 2020-11-20T12:04:54.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.453+0000 7f498759e700 1 Processor -- start 2020-11-20T12:04:54.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.453+0000 7f498759e700 1 -- start start 2020-11-20T12:04:54.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.453+0000 7f498759e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f49880612b0 0x7f49880a66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.453+0000 7f498759e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4988067500 0x7f49880a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.459 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.453+0000 7f498759e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49880a8780 0x7f498814ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.459 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.453+0000 7f498759e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49880a5c20 con 0x7f49880a8780 2020-11-20T12:04:54.459 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.453+0000 7f498759e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49880a5240 con 0x7f4988067500 2020-11-20T12:04:54.459 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.453+0000 7f498759e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49880a54e0 con 0x7f49880612b0 2020-11-20T12:04:54.459 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.453+0000 7f4985d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4988067500 0x7f49880a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.460 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.453+0000 7f4985d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4988067500 0x7f49880a1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48260/0 (socket says 172.21.15.189:48260) 2020-11-20T12:04:54.460 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.453+0000 7f4985d9b700 1 -- 172.21.15.189:0/2723300367 learned_addr learned my addr 172.21.15.189:0/2723300367 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:54.460 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.453+0000 7f4986d9d700 1 --2- 172.21.15.189:0/2723300367 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49880a8780 0x7f498814ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.460 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.453+0000 7f498659c700 1 --2- 172.21.15.189:0/2723300367 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f49880612b0 0x7f49880a66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.460 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.457+0000 7f4985d9b700 1 -- 172.21.15.189:0/2723300367 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f49880612b0 msgr2=0x7f49880a66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.461 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.457+0000 7f4985d9b700 1 --2- 172.21.15.189:0/2723300367 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f49880612b0 0x7f49880a66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.461 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.457+0000 7f4985d9b700 1 -- 172.21.15.189:0/2723300367 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49880a8780 msgr2=0x7f498814ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.461 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.457+0000 7f4985d9b700 1 --2- 172.21.15.189:0/2723300367 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49880a8780 0x7f498814ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.461 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.457+0000 7f4985d9b700 1 -- 172.21.15.189:0/2723300367 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49880a5780 con 0x7f4988067500 2020-11-20T12:04:54.462 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.457+0000 7f4985d9b700 1 --2- 172.21.15.189:0/2723300367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4988067500 0x7f49880a1a10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f4988066e50 tx=0x7f4978016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:54.463 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.457+0000 7f497e7fc700 1 -- 172.21.15.189:0/2723300367 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497801b5e0 con 0x7f4988067500 2020-11-20T12:04:54.463 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.457+0000 7f497e7fc700 1 -- 172.21.15.189:0/2723300367 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f497801b740 con 0x7f4988067500 2020-11-20T12:04:54.463 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.457+0000 7f498759e700 1 -- 172.21.15.189:0/2723300367 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4988134630 con 0x7f4988067500 2020-11-20T12:04:54.464 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.457+0000 7f497e7fc700 1 -- 172.21.15.189:0/2723300367 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497801bac0 con 0x7f4988067500 2020-11-20T12:04:54.464 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.457+0000 7f498759e700 1 -- 172.21.15.189:0/2723300367 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f498814b260 con 0x7f4988067500 2020-11-20T12:04:54.464 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.457+0000 7f498759e700 1 -- 172.21.15.189:0/2723300367 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49880a5910 con 0x7f4988067500 2020-11-20T12:04:54.464 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.457+0000 7f497e7fc700 1 -- 172.21.15.189:0/2723300367 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f4978017070 con 0x7f4988067500 2020-11-20T12:04:54.464 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.457+0000 7f497e7fc700 1 --2- 172.21.15.189:0/2723300367 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f49580557c0 0x7f4958057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.465 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.457+0000 7f497e7fc700 1 -- 172.21.15.189:0/2723300367 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4978070660 con 0x7f4988067500 2020-11-20T12:04:54.465 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.457+0000 7f498659c700 1 --2- 172.21.15.189:0/2723300367 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f49580557c0 0x7f4958057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.465 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.461+0000 7f498659c700 1 --2- 172.21.15.189:0/2723300367 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f49580557c0 0x7f4958057c80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f49700035f0 tx=0x7f4970007b90).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:54.466 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.465+0000 7f497e7fc700 1 -- 172.21.15.189:0/2723300367 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4978046580 con 0x7f4988067500 2020-11-20T12:04:54.620 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.621+0000 7f498759e700 1 -- 172.21.15.189:0/2723300367 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49880652d0 con 0x7f4988067500 2020-11-20T12:04:54.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.621+0000 7f497e7fc700 1 -- 172.21.15.189:0/2723300367 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16839 (secure 0 0 0) 0x7f4978015020 con 0x7f4988067500 2020-11-20T12:04:54.622 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:54.623 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":21,"fsid":"cda18d0e-64b3-41bb-964a-084069c3e648","created":"2020-11-20T12:04:37.538706+0000","modified":"2020-11-20T12:04:54.011613+0000","last_up_change":"2020-11-20T12:04:43.006631+0000","last_in_change":"2020-11-20T12:04:43.006631+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:41.395395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:48.550454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:52.101837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:53.018196+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c81214e1-4f84-4aaa-8ceb-99c398b2a89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6811","nonce":11760},{"type":"v1","addr":"172.21.15.189:6814","nonce":11760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":11760},{"type":"v1","addr":"172.21.15.189:6818","nonce":11760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":11760},{"type":"v1","addr":"172.21.15.189:6828","nonce":11760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6821","nonce":11760},{"type":"v1","addr":"172.21.15.189:6824","nonce":11760}]},"public_addr":"172.21.15.189:6814/11760","cluster_addr":"172.21.15.189:6818/11760","heartbeat_back_addr":"172.21.15.189:6828/11760","heartbeat_front_addr":"172.21.15.189:6824/11760","state":["exists","up"]},{"osd":1,"uuid":"d2f3ec7d-d403-484f-b5ed-b2e6b079d63d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6805","nonce":11761},{"type":"v1","addr":"172.21.15.189:6807","nonce":11761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":11761},{"type":"v1","addr":"172.21.15.189:6812","nonce":11761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6817","nonce":11761},{"type":"v1","addr":"172.21.15.189:6820","nonce":11761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6813","nonce":11761},{"type":"v1","addr":"172.21.15.189:6815","nonce":11761}]},"public_addr":"172.21.15.189:6807/11761","cluster_addr":"172.21.15.189:6812/11761","heartbeat_back_addr":"172.21.15.189:6820/11761","heartbeat_front_addr":"172.21.15.189:6815/11761","state":["exists","up"]},{"osd":2,"uuid":"e6cf4790-000c-43eb-8844-175f04c60503","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6819","nonce":11751},{"type":"v1","addr":"172.21.15.189:6822","nonce":11751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":11751},{"type":"v1","addr":"172.21.15.189:6825","nonce":11751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":11751},{"type":"v1","addr":"172.21.15.189:6831","nonce":11751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6827","nonce":11751},{"type":"v1","addr":"172.21.15.189:6829","nonce":11751}]},"public_addr":"172.21.15.189:6822/11751","cluster_addr":"172.21.15.189:6825/11751","heartbeat_back_addr":"172.21.15.189:6831/11751","heartbeat_front_addr":"172.21.15.189:6829/11751","state":["exists","up"]},{"osd":3,"uuid":"ba49dfac-f2b9-476e-853d-42f05e21a77a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":11762},{"type":"v1","addr":"172.21.15.189:6801","nonce":11762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":11762},{"type":"v1","addr":"172.21.15.189:6803","nonce":11762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":11762},{"type":"v1","addr":"172.21.15.189:6809","nonce":11762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":11762},{"type":"v1","addr":"172.21.15.189:6806","nonce":11762}]},"public_addr":"172.21.15.189:6801/11762","cluster_addr":"172.21.15.189:6803/11762","heartbeat_back_addr":"172.21.15.189:6809/11762","heartbeat_front_addr":"172.21.15.189:6806/11762","state":["exists","up"]},{"osd":4,"uuid":"3ca7b438-38b2-4689-8b18-fb5a8ddf6aee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":12114},{"type":"v1","addr":"172.21.15.47:6825","nonce":12114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":12114},{"type":"v1","addr":"172.21.15.47:6827","nonce":12114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":12114},{"type":"v1","addr":"172.21.15.47:6831","nonce":12114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":12114},{"type":"v1","addr":"172.21.15.47:6829","nonce":12114}]},"public_addr":"172.21.15.47:6825/12114","cluster_addr":"172.21.15.47:6827/12114","heartbeat_back_addr":"172.21.15.47:6831/12114","heartbeat_front_addr":"172.21.15.47:6829/12114","state":["exists","up"]},{"osd":5,"uuid":"6a45fca7-83d3-44fd-a1d4-b01e4201a101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":12111},{"type":"v1","addr":"172.21.15.47:6801","nonce":12111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":12111},{"type":"v1","addr":"172.21.15.47:6803","nonce":12111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":12111},{"type":"v1","addr":"172.21.15.47:6807","nonce":12111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":12111},{"type":"v1","addr":"172.21.15.47:6805","nonce":12111}]},"public_addr":"172.21.15.47:6801/12111","cluster_addr":"172.21.15.47:6803/12111","heartbeat_back_addr":"172.21.15.47:6807/12111","heartbeat_front_addr":"172.21.15.47:6805/12111","state":["exists","up"]},{"osd":6,"uuid":"c4b1b9ed-61f8-4677-9755-dff59a30e447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":12113},{"type":"v1","addr":"172.21.15.47:6817","nonce":12113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":12113},{"type":"v1","addr":"172.21.15.47:6819","nonce":12113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":12113},{"type":"v1","addr":"172.21.15.47:6823","nonce":12113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":12113},{"type":"v1","addr":"172.21.15.47:6821","nonce":12113}]},"public_addr":"172.21.15.47:6817/12113","cluster_addr":"172.21.15.47:6819/12113","heartbeat_back_addr":"172.21.15.47:6823/12113","heartbeat_front_addr":"172.21.15.47:6821/12113","state":["exists","up"]},{"osd":7,"uuid":"4608e4ed-cc40-4c5c-ba8b-3779bbc62f65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":12112},{"type":"v1","addr":"172.21.15.47:6809","nonce":12112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":12112},{"type":"v1","addr":"172.21.15.47:6811","nonce":12112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":12112},{"type":"v1","addr":"172.21.15.47:6815","nonce":12112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":12112},{"type":"v1","addr":"172.21.15.47:6813","nonce":12112}]},"public_addr":"172.21.15.47:6809/12112","cluster_addr":"172.21.15.47:6811/12112","heartbeat_back_addr":"172.21.15.47:6815/12112","heartbeat_front_addr":"172.21.15.47:6813/12112","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.435265+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.472776+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.388370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.417366+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.102392+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.116529+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.167430+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.148139+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:04:54.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.621+0000 7f498759e700 1 -- 172.21.15.189:0/2723300367 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f49580557c0 msgr2=0x7f4958057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.621+0000 7f498759e700 1 --2- 172.21.15.189:0/2723300367 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f49580557c0 0x7f4958057c80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f49700035f0 tx=0x7f4970007b90).stop 2020-11-20T12:04:54.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.621+0000 7f498759e700 1 -- 172.21.15.189:0/2723300367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4988067500 msgr2=0x7f49880a1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.621+0000 7f498759e700 1 --2- 172.21.15.189:0/2723300367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4988067500 0x7f49880a1a10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f4988066e50 tx=0x7f4978016000).stop 2020-11-20T12:04:54.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.625+0000 7f498759e700 1 -- 172.21.15.189:0/2723300367 shutdown_connections 2020-11-20T12:04:54.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.625+0000 7f498759e700 1 --2- 172.21.15.189:0/2723300367 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f49580557c0 0x7f4958057c80 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.625+0000 7f498759e700 1 --2- 172.21.15.189:0/2723300367 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f49880a8780 0x7f498814ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.625+0000 7f498759e700 1 --2- 172.21.15.189:0/2723300367 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f49880612b0 0x7f49880a66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.626 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.625+0000 7f498759e700 1 --2- 172.21.15.189:0/2723300367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4988067500 0x7f49880a1a10 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.626 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.625+0000 7f498759e700 1 -- 172.21.15.189:0/2723300367 >> 172.21.15.189:0/2723300367 conn(0x7f49880625f0 msgr2=0x7f49880ad0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:54.626 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.625+0000 7f498759e700 1 -- 172.21.15.189:0/2723300367 shutdown_connections 2020-11-20T12:04:54.626 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.625+0000 7f498759e700 1 -- 172.21.15.189:0/2723300367 wait complete. 2020-11-20T12:04:54.638 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2020-11-20T12:04:54.768 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.769+0000 7f26fd005700 1 Processor -- start 2020-11-20T12:04:54.769 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.769+0000 7f26fd005700 1 -- start start 2020-11-20T12:04:54.770 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.769+0000 7f26fd005700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f26f80669b0 0x7f26f80a8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.770 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.769+0000 7f26fd005700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26f8066d90 0x7f26f8067290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.771 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.769+0000 7f26fd005700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26f80a9950 con 0x7f26f8067780 2020-11-20T12:04:54.771 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.769+0000 7f26fd005700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26f80a9bf0 con 0x7f26f80669b0 2020-11-20T12:04:54.771 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.769+0000 7f26fd005700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f80aa090 con 0x7f26f8066d90 2020-11-20T12:04:54.772 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.769+0000 7f26effff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26f8066d90 0x7f26f8067290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.772 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.769+0000 7f26f7fff700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f26f80669b0 0x7f26f80a8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.773 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.769+0000 7f26effff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26f8066d90 0x7f26f8067290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.189:41302/0 (socket says 172.21.15.189:41302) 2020-11-20T12:04:54.773 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.769+0000 7f26fc804700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f26f8067780 0x7f26f8063a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.189:44554/0 (socket says 172.21.15.189:44554) 2020-11-20T12:04:54.773 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.769+0000 7f26effff700 1 -- 172.21.15.189:0/1062717944 learned_addr learned my addr 172.21.15.189:0/1062717944 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:54.773 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.769+0000 7f26effff700 1 -- 172.21.15.189:0/1062717944 >> v2:172.21.15.47:3300/0 conn(0x7f26f80669b0 msgr2=0x7f26f80a8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:54.774 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.769+0000 7f26effff700 1 --2- 172.21.15.189:0/1062717944 >> v2:172.21.15.47:3300/0 conn(0x7f26f80669b0 0x7f26f80a8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.774 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.769+0000 7f26effff700 1 -- 172.21.15.189:0/1062717944 >> v1:172.21.15.47:6789/0 conn(0x7f26f8067780 legacy=0x7f26f8063a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.774 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.769+0000 7f26effff700 1 -- 172.21.15.189:0/1062717944 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f8060ff0 con 0x7f26f8066d90 2020-11-20T12:04:54.774 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.769+0000 7f26effff700 1 --2- 172.21.15.189:0/1062717944 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26f8066d90 0x7f26f8067290 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f26e0006fd0 tx=0x7f26e0006eb0).ready entity=mon.2 client_cookie=f6aaa3c81f267ef7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:54.774 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.773+0000 7f26f77fe700 1 -- 172.21.15.189:0/1062717944 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e0009810 con 0x7f26f8066d90 2020-11-20T12:04:54.775 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.773+0000 7f26f77fe700 1 -- 172.21.15.189:0/1062717944 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e0009970 con 0x7f26f8066d90 2020-11-20T12:04:54.775 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.773+0000 7f26f77fe700 1 -- 172.21.15.189:0/1062717944 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e0009cf0 con 0x7f26f8066d90 2020-11-20T12:04:54.776 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.773+0000 7f26fd005700 1 -- 172.21.15.189:0/1062717944 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26f8066d90 msgr2=0x7f26f8067290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.776 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.773+0000 7f26fd005700 1 --2- 172.21.15.189:0/1062717944 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26f8066d90 0x7f26f8067290 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f26e0006fd0 tx=0x7f26e0006eb0).stop 2020-11-20T12:04:54.776 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.773+0000 7f26fd005700 1 -- 172.21.15.189:0/1062717944 shutdown_connections 2020-11-20T12:04:54.776 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.773+0000 7f26fd005700 1 --2- 172.21.15.189:0/1062717944 >> v2:172.21.15.47:3300/0 conn(0x7f26f80669b0 0x7f26f80a8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.777 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.773+0000 7f26fd005700 1 --2- 172.21.15.189:0/1062717944 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26f8066d90 0x7f26f8067290 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.777 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.773+0000 7f26fd005700 1 -- 172.21.15.189:0/1062717944 >> 172.21.15.189:0/1062717944 conn(0x7f26f80ad8e0 msgr2=0x7f26f80adcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:54.777 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.773+0000 7f26fd005700 1 -- 172.21.15.189:0/1062717944 shutdown_connections 2020-11-20T12:04:54.777 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.773+0000 7f26fd005700 1 -- 172.21.15.189:0/1062717944 wait complete. 2020-11-20T12:04:54.778 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.777+0000 7f26fd005700 1 Processor -- start 2020-11-20T12:04:54.778 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.777+0000 7f26fd005700 1 -- start start 2020-11-20T12:04:54.780 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.777+0000 7f26fd005700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26f80669b0 0x7f26f81410d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.780 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.777+0000 7f26fd005700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26f8066d90 0x7f26f80ac870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.780 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.777+0000 7f26fd005700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26f8067780 0x7f26f8151980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.780 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.777+0000 7f26fd005700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f80ad180 con 0x7f26f8066d90 2020-11-20T12:04:54.781 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.777+0000 7f26fd005700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f8151240 con 0x7f26f80669b0 2020-11-20T12:04:54.781 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.777+0000 7f26fd005700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f81514e0 con 0x7f26f8067780 2020-11-20T12:04:54.781 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.777+0000 7f26f7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26f80669b0 0x7f26f81410d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.781 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.777+0000 7f26fc804700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26f8067780 0x7f26f8151980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.781 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.777+0000 7f26effff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26f8066d90 0x7f26f80ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.782 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.777+0000 7f26f7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26f80669b0 0x7f26f81410d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48274/0 (socket says 172.21.15.189:48274) 2020-11-20T12:04:54.782 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.777+0000 7f26f7fff700 1 -- 172.21.15.189:0/4224857625 learned_addr learned my addr 172.21.15.189:0/4224857625 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:54.782 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.777+0000 7f26f7fff700 1 -- 172.21.15.189:0/4224857625 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26f8067780 msgr2=0x7f26f8151980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.782 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.777+0000 7f26f7fff700 1 --2- 172.21.15.189:0/4224857625 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26f8067780 0x7f26f8151980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.782 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.777+0000 7f26f7fff700 1 -- 172.21.15.189:0/4224857625 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26f8066d90 msgr2=0x7f26f80ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:54.783 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.777+0000 7f26f7fff700 1 --2- 172.21.15.189:0/4224857625 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26f8066d90 0x7f26f80ac870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:54.783 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.777+0000 7f26f7fff700 1 -- 172.21.15.189:0/4224857625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f813c450 con 0x7f26f80669b0 2020-11-20T12:04:54.783 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.777+0000 7f26effff700 1 --2- 172.21.15.189:0/4224857625 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26f8066d90 0x7f26f80ac870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:04:54.783 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.777+0000 7f26fc804700 1 --2- 172.21.15.189:0/4224857625 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26f8067780 0x7f26f8151980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:04:54.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.777+0000 7f26f7fff700 1 --2- 172.21.15.189:0/4224857625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26f80669b0 0x7f26f81410d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f26e800ebd0 tx=0x7f26e8006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:54.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.777+0000 7f26f4ff9700 1 -- 172.21.15.189:0/4224857625 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e800b810 con 0x7f26f80669b0 2020-11-20T12:04:54.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.781+0000 7f26fd005700 1 -- 172.21.15.189:0/4224857625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f814dbd0 con 0x7f26f80669b0 2020-11-20T12:04:54.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.781+0000 7f26f4ff9700 1 -- 172.21.15.189:0/4224857625 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e800b970 con 0x7f26f80669b0 2020-11-20T12:04:54.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.781+0000 7f26f4ff9700 1 -- 172.21.15.189:0/4224857625 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e800bcf0 con 0x7f26f80669b0 2020-11-20T12:04:54.785 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.781+0000 7f26fd005700 1 -- 172.21.15.189:0/4224857625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f8151e70 con 0x7f26f80669b0 2020-11-20T12:04:54.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.781+0000 7f26fd005700 1 -- 172.21.15.189:0/4224857625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f81516a0 con 0x7f26f80669b0 2020-11-20T12:04:54.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.781+0000 7f26f4ff9700 1 -- 172.21.15.189:0/4224857625 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f26e8012600 con 0x7f26f80669b0 2020-11-20T12:04:54.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.781+0000 7f26f4ff9700 1 --2- 172.21.15.189:0/4224857625 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f26c80557c0 0x7f26c8057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:54.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.781+0000 7f26f4ff9700 1 -- 172.21.15.189:0/4224857625 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f26e806e900 con 0x7f26f80669b0 2020-11-20T12:04:54.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.781+0000 7f26effff700 1 --2- 172.21.15.189:0/4224857625 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f26c80557c0 0x7f26c8057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:54.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.785+0000 7f26effff700 1 --2- 172.21.15.189:0/4224857625 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f26c80557c0 0x7f26c8057c80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f26e000b1b0 tx=0x7f26e0015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:54.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.789+0000 7f26f4ff9700 1 -- 172.21.15.189:0/4224857625 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f26e8044740 con 0x7f26f80669b0 2020-11-20T12:04:54.990 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:54.989+0000 7f26fd005700 1 -- 172.21.15.189:0/4224857625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f26f80a9180 con 0x7f26f80669b0 2020-11-20T12:04:55.023 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.021+0000 7f26f4ff9700 1 -- 172.21.15.189:0/4224857625 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) v1 ==== 118+0+0 (secure 0 0 0) 0x7f26e8047d60 con 0x7f26f80669b0 2020-11-20T12:04:55.024 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.025+0000 7f26fd005700 1 -- 172.21.15.189:0/4224857625 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f26c80557c0 msgr2=0x7f26c8057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:55.025 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.025+0000 7f26fd005700 1 --2- 172.21.15.189:0/4224857625 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f26c80557c0 0x7f26c8057c80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f26e000b1b0 tx=0x7f26e0015000).stop 2020-11-20T12:04:55.025 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.025+0000 7f26fd005700 1 -- 172.21.15.189:0/4224857625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26f80669b0 msgr2=0x7f26f81410d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:55.025 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.025+0000 7f26fd005700 1 --2- 172.21.15.189:0/4224857625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26f80669b0 0x7f26f81410d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f26e800ebd0 tx=0x7f26e8006df0).stop 2020-11-20T12:04:55.025 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.025+0000 7f26fd005700 1 -- 172.21.15.189:0/4224857625 shutdown_connections 2020-11-20T12:04:55.026 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.025+0000 7f26fd005700 1 --2- 172.21.15.189:0/4224857625 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26f8067780 0x7f26f8151980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:55.026 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.025+0000 7f26fd005700 1 --2- 172.21.15.189:0/4224857625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26f80669b0 0x7f26f81410d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:55.026 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.025+0000 7f26fd005700 1 --2- 172.21.15.189:0/4224857625 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f26c80557c0 0x7f26c8057c80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:55.026 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.025+0000 7f26fd005700 1 --2- 172.21.15.189:0/4224857625 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26f8066d90 0x7f26f80ac870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:55.026 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.025+0000 7f26fd005700 1 -- 172.21.15.189:0/4224857625 >> 172.21.15.189:0/4224857625 conn(0x7f26f80ad8e0 msgr2=0x7f26f80adcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:55.027 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.025+0000 7f26fd005700 1 -- 172.21.15.189:0/4224857625 shutdown_connections 2020-11-20T12:04:55.027 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.025+0000 7f26fd005700 1 -- 172.21.15.189:0/4224857625 wait complete. 2020-11-20T12:04:55.039 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2020-11-20T12:04:55.159 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.157+0000 7f4e5d614700 1 Processor -- start 2020-11-20T12:04:55.159 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.157+0000 7f4e5d614700 1 -- start start 2020-11-20T12:04:55.162 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.157+0000 7f4e5d614700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4e580612b0 0x7f4e580a0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:55.162 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e5d614700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4e581340b0 0x7f4e5813a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:55.162 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e5d614700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e58067b80 con 0x7f4e5813a090 2020-11-20T12:04:55.163 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e5d614700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e58067e20 con 0x7f4e581340b0 2020-11-20T12:04:55.163 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e5d614700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e580680c0 con 0x7f4e580612b0 2020-11-20T12:04:55.163 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e57fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4e580612b0 0x7f4e580a0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:55.163 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e57fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4e580612b0 0x7f4e580a0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48282/0 (socket says 172.21.15.189:48282) 2020-11-20T12:04:55.163 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e57fff700 1 -- 172.21.15.189:0/1977831347 learned_addr learned my addr 172.21.15.189:0/1977831347 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:55.164 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e5ce13700 1 --2- 172.21.15.189:0/1977831347 >> v2:172.21.15.47:3300/0 conn(0x7f4e581340b0 0x7f4e5813a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:55.164 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e57fff700 1 -- 172.21.15.189:0/1977831347 >> v2:172.21.15.47:3300/0 conn(0x7f4e581340b0 msgr2=0x7f4e5813a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:55.164 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e57fff700 1 --2- 172.21.15.189:0/1977831347 >> v2:172.21.15.47:3300/0 conn(0x7f4e581340b0 0x7f4e5813a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:55.164 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e57fff700 1 -- 172.21.15.189:0/1977831347 >> v1:172.21.15.47:6789/0 conn(0x7f4e5813a090 legacy=0x7f4e581339a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:55.164 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e57fff700 1 -- 172.21.15.189:0/1977831347 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e581344b0 con 0x7f4e580612b0 2020-11-20T12:04:55.165 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e5ce13700 1 --2- 172.21.15.189:0/1977831347 >> v2:172.21.15.47:3300/0 conn(0x7f4e581340b0 0x7f4e5813a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:55.165 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e57fff700 1 --2- 172.21.15.189:0/1977831347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e580612b0 0x7f4e580a0730 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f4e580ae2f0 tx=0x7f4e40006eb0).ready entity=mon.1 client_cookie=3a56867423d13a51 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:55.166 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e56ffd700 1 -- 172.21.15.189:0/1977831347 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e400098d0 con 0x7f4e580612b0 2020-11-20T12:04:55.166 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e56ffd700 1 -- 172.21.15.189:0/1977831347 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e40009a30 con 0x7f4e580612b0 2020-11-20T12:04:55.166 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e56ffd700 1 -- 172.21.15.189:0/1977831347 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e40009db0 con 0x7f4e580612b0 2020-11-20T12:04:55.166 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e5d614700 1 -- 172.21.15.189:0/1977831347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e580612b0 msgr2=0x7f4e580a0730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:55.166 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e5d614700 1 --2- 172.21.15.189:0/1977831347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e580612b0 0x7f4e580a0730 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f4e580ae2f0 tx=0x7f4e40006eb0).stop 2020-11-20T12:04:55.167 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e5d614700 1 -- 172.21.15.189:0/1977831347 shutdown_connections 2020-11-20T12:04:55.167 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e5d614700 1 --2- 172.21.15.189:0/1977831347 >> v2:172.21.15.47:3300/0 conn(0x7f4e581340b0 0x7f4e5813a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:55.167 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e5d614700 1 --2- 172.21.15.189:0/1977831347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e580612b0 0x7f4e580a0730 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:55.167 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e5d614700 1 -- 172.21.15.189:0/1977831347 >> 172.21.15.189:0/1977831347 conn(0x7f4e58065910 msgr2=0x7f4e58060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:55.167 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e5d614700 1 -- 172.21.15.189:0/1977831347 shutdown_connections 2020-11-20T12:04:55.168 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e5d614700 1 -- 172.21.15.189:0/1977831347 wait complete. 2020-11-20T12:04:55.170 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e5d614700 1 Processor -- start 2020-11-20T12:04:55.170 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e5d614700 1 -- start start 2020-11-20T12:04:55.171 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e5d614700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e580612b0 0x7f4e580a5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:55.171 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e5d614700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e581340b0 0x7f4e580a3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:55.171 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e5d614700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4e5813a090 0x7f4e5814d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:55.171 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e5d614700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e580ac9b0 con 0x7f4e581340b0 2020-11-20T12:04:55.172 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e5d614700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e580a6850 con 0x7f4e580612b0 2020-11-20T12:04:55.172 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.161+0000 7f4e5d614700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e580a4650 con 0x7f4e5813a090 2020-11-20T12:04:55.172 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.165+0000 7f4e57fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e580612b0 0x7f4e580a5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:55.172 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.165+0000 7f4e57fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e580612b0 0x7f4e580a5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48286/0 (socket says 172.21.15.189:48286) 2020-11-20T12:04:55.172 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.165+0000 7f4e57fff700 1 -- 172.21.15.189:0/1473519358 learned_addr learned my addr 172.21.15.189:0/1473519358 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:55.173 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.165+0000 7f4e577fe700 1 --2- 172.21.15.189:0/1473519358 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e581340b0 0x7f4e580a3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:55.173 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.165+0000 7f4e5ce13700 1 --2- 172.21.15.189:0/1473519358 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4e5813a090 0x7f4e5814d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:55.173 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.165+0000 7f4e57fff700 1 -- 172.21.15.189:0/1473519358 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4e5813a090 msgr2=0x7f4e5814d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:55.173 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.165+0000 7f4e57fff700 1 --2- 172.21.15.189:0/1473519358 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4e5813a090 0x7f4e5814d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:55.173 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.165+0000 7f4e57fff700 1 -- 172.21.15.189:0/1473519358 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e581340b0 msgr2=0x7f4e580a3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:55.174 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.165+0000 7f4e57fff700 1 --2- 172.21.15.189:0/1473519358 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e581340b0 0x7f4e580a3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:55.174 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.165+0000 7f4e57fff700 1 -- 172.21.15.189:0/1473519358 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e580a4880 con 0x7f4e580612b0 2020-11-20T12:04:55.174 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.165+0000 7f4e577fe700 1 --2- 172.21.15.189:0/1473519358 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e581340b0 0x7f4e580a3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:04:55.174 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.165+0000 7f4e57fff700 1 --2- 172.21.15.189:0/1473519358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e580612b0 0x7f4e580a5c10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f4e4000b1b0 tx=0x7f4e40015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:55.174 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.165+0000 7f4e4ffff700 1 -- 172.21.15.189:0/1473519358 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4001e460 con 0x7f4e580612b0 2020-11-20T12:04:55.175 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.165+0000 7f4e4ffff700 1 -- 172.21.15.189:0/1473519358 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4001e5c0 con 0x7f4e580612b0 2020-11-20T12:04:55.175 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.165+0000 7f4e4ffff700 1 -- 172.21.15.189:0/1473519358 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4001e960 con 0x7f4e580612b0 2020-11-20T12:04:55.175 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.165+0000 7f4e5d614700 1 -- 172.21.15.189:0/1473519358 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e580a4b20 con 0x7f4e580612b0 2020-11-20T12:04:55.175 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.165+0000 7f4e5d614700 1 -- 172.21.15.189:0/1473519358 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e580a15b0 con 0x7f4e580612b0 2020-11-20T12:04:55.175 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.165+0000 7f4e4ffff700 1 -- 172.21.15.189:0/1473519358 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f4e40025030 con 0x7f4e580612b0 2020-11-20T12:04:55.175 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.165+0000 7f4e5d614700 1 -- 172.21.15.189:0/1473519358 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e580a4420 con 0x7f4e580612b0 2020-11-20T12:04:55.176 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.165+0000 7f4e4ffff700 1 --2- 172.21.15.189:0/1473519358 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4e340557c0 0x7f4e34057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:55.176 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.165+0000 7f4e4ffff700 1 -- 172.21.15.189:0/1473519358 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4e40070260 con 0x7f4e580612b0 2020-11-20T12:04:55.176 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.165+0000 7f4e577fe700 1 --2- 172.21.15.189:0/1473519358 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4e340557c0 0x7f4e34057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:55.176 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.165+0000 7f4e577fe700 1 --2- 172.21.15.189:0/1473519358 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4e340557c0 0x7f4e34057c80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f4e580a77b0 tx=0x7f4e48009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:55.177 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.173+0000 7f4e4ffff700 1 -- 172.21.15.189:0/1473519358 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4e40046100 con 0x7f4e580612b0 2020-11-20T12:04:55.376 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:55.377+0000 7f4e5d614700 1 -- 172.21.15.189:0/1473519358 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f4e58067840 con 0x7f4e580612b0 2020-11-20T12:04:56.031 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.029+0000 7f4e4ffff700 1 -- 172.21.15.189:0/1473519358 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v6) v1 ==== 115+0+0 (secure 0 0 0) 0x7f4e40023020 con 0x7f4e580612b0 2020-11-20T12:04:56.035 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.033+0000 7f4e5d614700 1 -- 172.21.15.189:0/1473519358 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4e340557c0 msgr2=0x7f4e34057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:56.035 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.033+0000 7f4e5d614700 1 --2- 172.21.15.189:0/1473519358 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4e340557c0 0x7f4e34057c80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f4e580a77b0 tx=0x7f4e48009040).stop 2020-11-20T12:04:56.035 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.033+0000 7f4e5d614700 1 -- 172.21.15.189:0/1473519358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e580612b0 msgr2=0x7f4e580a5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:56.036 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.033+0000 7f4e5d614700 1 --2- 172.21.15.189:0/1473519358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e580612b0 0x7f4e580a5c10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f4e4000b1b0 tx=0x7f4e40015000).stop 2020-11-20T12:04:56.036 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.033+0000 7f4e5d614700 1 -- 172.21.15.189:0/1473519358 shutdown_connections 2020-11-20T12:04:56.036 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.033+0000 7f4e5d614700 1 --2- 172.21.15.189:0/1473519358 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4e5813a090 0x7f4e5814d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.036 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.033+0000 7f4e5d614700 1 --2- 172.21.15.189:0/1473519358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e580612b0 0x7f4e580a5c10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.036 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.033+0000 7f4e5d614700 1 --2- 172.21.15.189:0/1473519358 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4e340557c0 0x7f4e34057c80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.037 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.033+0000 7f4e5d614700 1 --2- 172.21.15.189:0/1473519358 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4e581340b0 0x7f4e580a3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.037 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.033+0000 7f4e5d614700 1 -- 172.21.15.189:0/1473519358 >> 172.21.15.189:0/1473519358 conn(0x7f4e58065910 msgr2=0x7f4e58136b30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:56.037 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.033+0000 7f4e5d614700 1 -- 172.21.15.189:0/1473519358 shutdown_connections 2020-11-20T12:04:56.037 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.033+0000 7f4e5d614700 1 -- 172.21.15.189:0/1473519358 wait complete. 2020-11-20T12:04:56.052 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:04:56.175 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.173+0000 7f0b06e12700 1 Processor -- start 2020-11-20T12:04:56.176 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.173+0000 7f0b06e12700 1 -- start start 2020-11-20T12:04:56.177 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.177+0000 7f0b06e12700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0b0013a920 0x7f0b0013add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:56.177 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.177+0000 7f0b06e12700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0b00137140 0x7f0b001375d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:56.177 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.177+0000 7f0b06e12700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b0013b2c0 con 0x7f0b00062930 2020-11-20T12:04:56.178 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.177+0000 7f0b06e12700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b001381f0 con 0x7f0b00137140 2020-11-20T12:04:56.178 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.177+0000 7f0b06e12700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b000ad3f0 con 0x7f0b0013a920 2020-11-20T12:04:56.178 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.177+0000 7f0afffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0b0013a920 0x7f0b0013add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:56.178 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.177+0000 7f0b04bae700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0b00062930 0x7f0b00067400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49916/0 (socket says 172.21.15.189:49916) 2020-11-20T12:04:56.179 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.177+0000 7f0b04bae700 1 -- 172.21.15.189:0/3882361060 learned_addr learned my addr 172.21.15.189:0/3882361060 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:04:56.179 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.177+0000 7f0b053af700 1 --2- 172.21.15.189:0/3882361060 >> v2:172.21.15.47:3300/0 conn(0x7f0b00137140 0x7f0b001375d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:56.180 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.177+0000 7f0afffff700 1 -- 172.21.15.189:0/3882361060 >> v2:172.21.15.47:3300/0 conn(0x7f0b00137140 msgr2=0x7f0b001375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:56.180 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.177+0000 7f0afffff700 1 --2- 172.21.15.189:0/3882361060 >> v2:172.21.15.47:3300/0 conn(0x7f0b00137140 0x7f0b001375d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.181 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.177+0000 7f0afffff700 1 -- 172.21.15.189:0/3882361060 >> v1:172.21.15.189:6789/0 conn(0x7f0b00062930 legacy=0x7f0b00067400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:56.181 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.177+0000 7f0afffff700 1 -- 172.21.15.189:0/3882361060 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b00066bb0 con 0x7f0b0013a920 2020-11-20T12:04:56.181 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.177+0000 7f0afffff700 1 --2- 172.21.15.189:0/3882361060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b0013a920 0x7f0b0013add0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f0b000664c0 tx=0x7f0af4006df0).ready entity=mon.1 client_cookie=737b260de613e8bf server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:56.181 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.177+0000 7f0aff7fe700 1 -- 172.21.15.189:0/3882361060 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af40097b0 con 0x7f0b0013a920 2020-11-20T12:04:56.182 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.177+0000 7f0aff7fe700 1 -- 172.21.15.189:0/3882361060 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0af4009910 con 0x7f0b0013a920 2020-11-20T12:04:56.182 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.177+0000 7f0aff7fe700 1 -- 172.21.15.189:0/3882361060 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af4009c90 con 0x7f0b0013a920 2020-11-20T12:04:56.182 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.177+0000 7f0b06e12700 1 -- 172.21.15.189:0/3882361060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b0013a920 msgr2=0x7f0b0013add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:56.182 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.177+0000 7f0b06e12700 1 --2- 172.21.15.189:0/3882361060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b0013a920 0x7f0b0013add0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f0b000664c0 tx=0x7f0af4006df0).stop 2020-11-20T12:04:56.182 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.177+0000 7f0b06e12700 1 -- 172.21.15.189:0/3882361060 shutdown_connections 2020-11-20T12:04:56.183 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.177+0000 7f0b06e12700 1 --2- 172.21.15.189:0/3882361060 >> v2:172.21.15.47:3300/0 conn(0x7f0b00137140 0x7f0b001375d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.183 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.177+0000 7f0b06e12700 1 --2- 172.21.15.189:0/3882361060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b0013a920 0x7f0b0013add0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.183 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.177+0000 7f0b06e12700 1 -- 172.21.15.189:0/3882361060 >> 172.21.15.189:0/3882361060 conn(0x7f0b00065080 msgr2=0x7f0b00062080 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:56.183 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.177+0000 7f0b06e12700 1 -- 172.21.15.189:0/3882361060 shutdown_connections 2020-11-20T12:04:56.183 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.177+0000 7f0b06e12700 1 -- 172.21.15.189:0/3882361060 wait complete. 2020-11-20T12:04:56.185 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.181+0000 7f0b06e12700 1 Processor -- start 2020-11-20T12:04:56.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.181+0000 7f0b06e12700 1 -- start start 2020-11-20T12:04:56.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.181+0000 7f0b06e12700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b00062930 0x7f0b0013ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:56.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.181+0000 7f0b06e12700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b00137140 0x7f0b0014d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:56.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.181+0000 7f0b06e12700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0b0013a920 0x7f0b00134010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:56.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.181+0000 7f0b06e12700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0014d060 con 0x7f0b00137140 2020-11-20T12:04:56.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.181+0000 7f0b06e12700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0014d300 con 0x7f0b00062930 2020-11-20T12:04:56.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.181+0000 7f0b06e12700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b001383b0 con 0x7f0b0013a920 2020-11-20T12:04:56.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.181+0000 7f0b04bae700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b00062930 0x7f0b0013ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:56.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.181+0000 7f0b04bae700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b00062930 0x7f0b0013ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48300/0 (socket says 172.21.15.189:48300) 2020-11-20T12:04:56.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.181+0000 7f0b04bae700 1 -- 172.21.15.189:0/4240161882 learned_addr learned my addr 172.21.15.189:0/4240161882 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:56.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.181+0000 7f0b053af700 1 --2- 172.21.15.189:0/4240161882 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0b0013a920 0x7f0b00134010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:56.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.181+0000 7f0afffff700 1 --2- 172.21.15.189:0/4240161882 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b00137140 0x7f0b0014d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:56.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.181+0000 7f0b04bae700 1 -- 172.21.15.189:0/4240161882 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0b0013a920 msgr2=0x7f0b00134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:56.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.181+0000 7f0b04bae700 1 --2- 172.21.15.189:0/4240161882 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0b0013a920 0x7f0b00134010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.181+0000 7f0b04bae700 1 -- 172.21.15.189:0/4240161882 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b00137140 msgr2=0x7f0b0014d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:56.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.181+0000 7f0b04bae700 1 --2- 172.21.15.189:0/4240161882 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b00137140 0x7f0b0014d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.181+0000 7f0b04bae700 1 -- 172.21.15.189:0/4240161882 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0013bd00 con 0x7f0b00062930 2020-11-20T12:04:56.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.181+0000 7f0afffff700 1 --2- 172.21.15.189:0/4240161882 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b00137140 0x7f0b0014d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:56.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.181+0000 7f0b04bae700 1 --2- 172.21.15.189:0/4240161882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b00062930 0x7f0b0013ef50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f0af000c710 tx=0x7f0af0007c00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:56.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.181+0000 7f0afcff9700 1 -- 172.21.15.189:0/4240161882 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af0007320 con 0x7f0b00062930 2020-11-20T12:04:56.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.181+0000 7f0afcff9700 1 -- 172.21.15.189:0/4240161882 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0af0007480 con 0x7f0b00062930 2020-11-20T12:04:56.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.181+0000 7f0afcff9700 1 -- 172.21.15.189:0/4240161882 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af0005800 con 0x7f0b00062930 2020-11-20T12:04:56.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.181+0000 7f0b06e12700 1 -- 172.21.15.189:0/4240161882 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b0014dc90 con 0x7f0b00062930 2020-11-20T12:04:56.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.181+0000 7f0b06e12700 1 -- 172.21.15.189:0/4240161882 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b0013d6e0 con 0x7f0b00062930 2020-11-20T12:04:56.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.185+0000 7f0b06e12700 1 -- 172.21.15.189:0/4240161882 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b0014d4c0 con 0x7f0b00062930 2020-11-20T12:04:56.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.185+0000 7f0afcff9700 1 -- 172.21.15.189:0/4240161882 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f0af0010030 con 0x7f0b00062930 2020-11-20T12:04:56.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.185+0000 7f0afcff9700 1 --2- 172.21.15.189:0/4240161882 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0ad80557c0 0x7f0ad8057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:56.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.185+0000 7f0afffff700 1 --2- 172.21.15.189:0/4240161882 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0ad80557c0 0x7f0ad8057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:56.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.185+0000 7f0afcff9700 1 -- 172.21.15.189:0/4240161882 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0af006e400 con 0x7f0b00062930 2020-11-20T12:04:56.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.189+0000 7f0afffff700 1 --2- 172.21.15.189:0/4240161882 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0ad80557c0 0x7f0ad8057c80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f0af4006e30 tx=0x7f0af4015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:56.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.193+0000 7f0afcff9700 1 -- 172.21.15.189:0/4240161882 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0af0044190 con 0x7f0b00062930 2020-11-20T12:04:56.394 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.393+0000 7f0b06e12700 1 -- 172.21.15.189:0/4240161882 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b0013d8d0 con 0x7f0b00062930 2020-11-20T12:04:56.395 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.393+0000 7f0afcff9700 1 -- 172.21.15.189:0/4240161882 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2134 (secure 0 0 0) 0x7f0af00477b0 con 0x7f0b00062930 2020-11-20T12:04:56.395 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:56.395 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2660417436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2660417436},{"type":"v1","addr":"172.21.15.47:6833","nonce":2660417436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:04:56.025055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.189:6835/1042140684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1042140684},{"type":"v1","addr":"172.21.15.189:6835","nonce":1042140684}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:04:56.397 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.397+0000 7f0b06e12700 1 -- 172.21.15.189:0/4240161882 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0ad80557c0 msgr2=0x7f0ad8057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:56.397 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.397+0000 7f0b06e12700 1 --2- 172.21.15.189:0/4240161882 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0ad80557c0 0x7f0ad8057c80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f0af4006e30 tx=0x7f0af4015000).stop 2020-11-20T12:04:56.397 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.397+0000 7f0b06e12700 1 -- 172.21.15.189:0/4240161882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b00062930 msgr2=0x7f0b0013ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:56.398 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.397+0000 7f0b06e12700 1 --2- 172.21.15.189:0/4240161882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b00062930 0x7f0b0013ef50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f0af000c710 tx=0x7f0af0007c00).stop 2020-11-20T12:04:56.398 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.397+0000 7f0b06e12700 1 -- 172.21.15.189:0/4240161882 shutdown_connections 2020-11-20T12:04:56.398 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.397+0000 7f0b06e12700 1 --2- 172.21.15.189:0/4240161882 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0b0013a920 0x7f0b00134010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.398 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.397+0000 7f0b06e12700 1 --2- 172.21.15.189:0/4240161882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b00062930 0x7f0b0013ef50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.398 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.397+0000 7f0b06e12700 1 --2- 172.21.15.189:0/4240161882 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0ad80557c0 0x7f0ad8057c80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.399 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.397+0000 7f0b06e12700 1 --2- 172.21.15.189:0/4240161882 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0b00137140 0x7f0b0014d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.399 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.397+0000 7f0b06e12700 1 -- 172.21.15.189:0/4240161882 >> 172.21.15.189:0/4240161882 conn(0x7f0b00065080 msgr2=0x7f0b000abea0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:56.399 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.397+0000 7f0b06e12700 1 -- 172.21.15.189:0/4240161882 shutdown_connections 2020-11-20T12:04:56.399 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.397+0000 7f0b06e12700 1 -- 172.21.15.189:0/4240161882 wait complete. 2020-11-20T12:04:56.400 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 6 2020-11-20T12:04:56.410 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:04:56.532 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f630b474700 1 Processor -- start 2020-11-20T12:04:56.533 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f630b474700 1 -- start start 2020-11-20T12:04:56.543 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f630b474700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6304067b50 0x7f6304136fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:56.543 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f630b474700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f63041374a0 0x7f6304133a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:56.543 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f630b474700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6304068020 con 0x7f63040612b0 2020-11-20T12:04:56.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f630b474700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63040682c0 con 0x7f6304067b50 2020-11-20T12:04:56.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f630b474700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6304137900 con 0x7f63041374a0 2020-11-20T12:04:56.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f630ac73700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f63041374a0 0x7f6304133a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:56.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f630a472700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f63040612b0 0x7f63040a0960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49930/0 (socket says 172.21.15.189:49930) 2020-11-20T12:04:56.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f630a472700 1 -- 172.21.15.189:0/300373297 learned_addr learned my addr 172.21.15.189:0/300373297 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:04:56.545 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f6309c71700 1 --2- 172.21.15.189:0/300373297 >> v2:172.21.15.47:3300/0 conn(0x7f6304067b50 0x7f6304136fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:56.545 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f630ac73700 1 -- 172.21.15.189:0/300373297 >> v2:172.21.15.47:3300/0 conn(0x7f6304067b50 msgr2=0x7f6304136fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:56.545 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f630ac73700 1 --2- 172.21.15.189:0/300373297 >> v2:172.21.15.47:3300/0 conn(0x7f6304067b50 0x7f6304136fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.545 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f630ac73700 1 -- 172.21.15.189:0/300373297 >> v1:172.21.15.189:6789/0 conn(0x7f63040612b0 legacy=0x7f63040a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:56.546 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f630ac73700 1 -- 172.21.15.189:0/300373297 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6304061690 con 0x7f63041374a0 2020-11-20T12:04:56.546 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f6309c71700 1 --2- 172.21.15.189:0/300373297 >> v2:172.21.15.47:3300/0 conn(0x7f6304067b50 0x7f6304136fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:04:56.546 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f630ac73700 1 --2- 172.21.15.189:0/300373297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63041374a0 0x7f6304133a10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f63040ad260 tx=0x7f62fc006df0).ready entity=mon.1 client_cookie=d7121873edc1daa1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:56.546 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f6309470700 1 -- 172.21.15.189:0/300373297 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc0097b0 con 0x7f63041374a0 2020-11-20T12:04:56.547 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f6309470700 1 -- 172.21.15.189:0/300373297 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62fc009910 con 0x7f63041374a0 2020-11-20T12:04:56.547 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f6309470700 1 -- 172.21.15.189:0/300373297 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc009c90 con 0x7f63041374a0 2020-11-20T12:04:56.547 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f630b474700 1 -- 172.21.15.189:0/300373297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63041374a0 msgr2=0x7f6304133a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:56.547 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f630b474700 1 --2- 172.21.15.189:0/300373297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63041374a0 0x7f6304133a10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f63040ad260 tx=0x7f62fc006df0).stop 2020-11-20T12:04:56.547 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f630b474700 1 -- 172.21.15.189:0/300373297 shutdown_connections 2020-11-20T12:04:56.548 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f630b474700 1 --2- 172.21.15.189:0/300373297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63041374a0 0x7f6304133a10 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.548 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f630b474700 1 --2- 172.21.15.189:0/300373297 >> v2:172.21.15.47:3300/0 conn(0x7f6304067b50 0x7f6304136fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.548 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f630b474700 1 -- 172.21.15.189:0/300373297 >> 172.21.15.189:0/300373297 conn(0x7f6304063210 msgr2=0x7f6304060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:56.548 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f630b474700 1 -- 172.21.15.189:0/300373297 shutdown_connections 2020-11-20T12:04:56.548 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f630b474700 1 -- 172.21.15.189:0/300373297 wait complete. 2020-11-20T12:04:56.549 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f630b474700 1 Processor -- start 2020-11-20T12:04:56.549 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.533+0000 7f630b474700 1 -- start start 2020-11-20T12:04:56.549 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.537+0000 7f630b474700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f63040612b0 0x7f63040a92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:56.549 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.537+0000 7f630b474700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6304067b50 0x7f6304127f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:56.549 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.537+0000 7f630b474700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63041374a0 0x7f63040abe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:56.550 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.537+0000 7f630b474700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63040a9c80 con 0x7f63041374a0 2020-11-20T12:04:56.550 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.537+0000 7f630b474700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63040ac6c0 con 0x7f6304067b50 2020-11-20T12:04:56.550 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.537+0000 7f630b474700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63040ab0d0 con 0x7f63040612b0 2020-11-20T12:04:56.550 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.537+0000 7f6309c71700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6304067b50 0x7f6304127f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:56.550 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.537+0000 7f6309c71700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6304067b50 0x7f6304127f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48318/0 (socket says 172.21.15.189:48318) 2020-11-20T12:04:56.551 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.537+0000 7f6309c71700 1 -- 172.21.15.189:0/220898942 learned_addr learned my addr 172.21.15.189:0/220898942 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:56.551 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.537+0000 7f630a472700 1 --2- 172.21.15.189:0/220898942 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f63040612b0 0x7f63040a92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:56.551 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.537+0000 7f630ac73700 1 --2- 172.21.15.189:0/220898942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63041374a0 0x7f63040abe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:56.551 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.537+0000 7f6309c71700 1 -- 172.21.15.189:0/220898942 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f63040612b0 msgr2=0x7f63040a92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:56.552 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.537+0000 7f6309c71700 1 --2- 172.21.15.189:0/220898942 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f63040612b0 0x7f63040a92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.552 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.537+0000 7f6309c71700 1 -- 172.21.15.189:0/220898942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63041374a0 msgr2=0x7f63040abe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:56.552 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.537+0000 7f6309c71700 1 --2- 172.21.15.189:0/220898942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63041374a0 0x7f63040abe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.552 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.537+0000 7f6309c71700 1 -- 172.21.15.189:0/220898942 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63040ac350 con 0x7f6304067b50 2020-11-20T12:04:56.552 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.537+0000 7f6309c71700 1 --2- 172.21.15.189:0/220898942 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6304067b50 0x7f6304127f40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f62f400cb80 tx=0x7f62f4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:56.553 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.537+0000 7f62f27fc700 1 -- 172.21.15.189:0/220898942 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f4007710 con 0x7f6304067b50 2020-11-20T12:04:56.553 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.537+0000 7f62f27fc700 1 -- 172.21.15.189:0/220898942 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f4007870 con 0x7f6304067b50 2020-11-20T12:04:56.553 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.537+0000 7f630b474700 1 -- 172.21.15.189:0/220898942 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6304066540 con 0x7f6304067b50 2020-11-20T12:04:56.553 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.537+0000 7f62f27fc700 1 -- 172.21.15.189:0/220898942 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f4007bf0 con 0x7f6304067b50 2020-11-20T12:04:56.553 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.537+0000 7f630b474700 1 -- 172.21.15.189:0/220898942 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6304138bf0 con 0x7f6304067b50 2020-11-20T12:04:56.554 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.537+0000 7f630b474700 1 -- 172.21.15.189:0/220898942 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6304133b20 con 0x7f6304067b50 2020-11-20T12:04:56.554 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.541+0000 7f62f27fc700 1 -- 172.21.15.189:0/220898942 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f62f401b460 con 0x7f6304067b50 2020-11-20T12:04:56.554 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.541+0000 7f62f27fc700 1 --2- 172.21.15.189:0/220898942 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f62dc0557c0 0x7f62dc057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:56.554 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.541+0000 7f62f27fc700 1 -- 172.21.15.189:0/220898942 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f62f406e780 con 0x7f6304067b50 2020-11-20T12:04:56.554 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.541+0000 7f630a472700 1 --2- 172.21.15.189:0/220898942 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f62dc0557c0 0x7f62dc057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:56.555 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.541+0000 7f630a472700 1 --2- 172.21.15.189:0/220898942 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f62dc0557c0 0x7f62dc057c80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f6300000f40 tx=0x7f6300009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:56.555 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.545+0000 7f62f27fc700 1 -- 172.21.15.189:0/220898942 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f62f40445c0 con 0x7f6304067b50 2020-11-20T12:04:56.633 INFO:tasks.ceph.mds.a.smithi189.stderr:2020-11-20T12:04:56.633+0000 7f610d559700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:04:56.698 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.697+0000 7f630b474700 1 -- 172.21.15.189:0/220898942 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6304145520 con 0x7f6304067b50 2020-11-20T12:04:56.700 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.701+0000 7f62f27fc700 1 -- 172.21.15.189:0/220898942 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16826 (secure 0 0 0) 0x7f62f4047be0 con 0x7f6304067b50 2020-11-20T12:04:56.701 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:56.701 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":22,"fsid":"cda18d0e-64b3-41bb-964a-084069c3e648","created":"2020-11-20T12:04:37.538706+0000","modified":"2020-11-20T12:04:55.015818+0000","last_up_change":"2020-11-20T12:04:43.006631+0000","last_in_change":"2020-11-20T12:04:43.006631+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:41.395395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:48.550454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:52.101837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:53.018196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c81214e1-4f84-4aaa-8ceb-99c398b2a89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6811","nonce":11760},{"type":"v1","addr":"172.21.15.189:6814","nonce":11760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":11760},{"type":"v1","addr":"172.21.15.189:6818","nonce":11760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":11760},{"type":"v1","addr":"172.21.15.189:6828","nonce":11760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6821","nonce":11760},{"type":"v1","addr":"172.21.15.189:6824","nonce":11760}]},"public_addr":"172.21.15.189:6814/11760","cluster_addr":"172.21.15.189:6818/11760","heartbeat_back_addr":"172.21.15.189:6828/11760","heartbeat_front_addr":"172.21.15.189:6824/11760","state":["exists","up"]},{"osd":1,"uuid":"d2f3ec7d-d403-484f-b5ed-b2e6b079d63d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6805","nonce":11761},{"type":"v1","addr":"172.21.15.189:6807","nonce":11761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":11761},{"type":"v1","addr":"172.21.15.189:6812","nonce":11761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6817","nonce":11761},{"type":"v1","addr":"172.21.15.189:6820","nonce":11761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6813","nonce":11761},{"type":"v1","addr":"172.21.15.189:6815","nonce":11761}]},"public_addr":"172.21.15.189:6807/11761","cluster_addr":"172.21.15.189:6812/11761","heartbeat_back_addr":"172.21.15.189:6820/11761","heartbeat_front_addr":"172.21.15.189:6815/11761","state":["exists","up"]},{"osd":2,"uuid":"e6cf4790-000c-43eb-8844-175f04c60503","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6819","nonce":11751},{"type":"v1","addr":"172.21.15.189:6822","nonce":11751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":11751},{"type":"v1","addr":"172.21.15.189:6825","nonce":11751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":11751},{"type":"v1","addr":"172.21.15.189:6831","nonce":11751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6827","nonce":11751},{"type":"v1","addr":"172.21.15.189:6829","nonce":11751}]},"public_addr":"172.21.15.189:6822/11751","cluster_addr":"172.21.15.189:6825/11751","heartbeat_back_addr":"172.21.15.189:6831/11751","heartbeat_front_addr":"172.21.15.189:6829/11751","state":["exists","up"]},{"osd":3,"uuid":"ba49dfac-f2b9-476e-853d-42f05e21a77a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":11762},{"type":"v1","addr":"172.21.15.189:6801","nonce":11762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":11762},{"type":"v1","addr":"172.21.15.189:6803","nonce":11762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":11762},{"type":"v1","addr":"172.21.15.189:6809","nonce":11762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":11762},{"type":"v1","addr":"172.21.15.189:6806","nonce":11762}]},"public_addr":"172.21.15.189:6801/11762","cluster_addr":"172.21.15.189:6803/11762","heartbeat_back_addr":"172.21.15.189:6809/11762","heartbeat_front_addr":"172.21.15.189:6806/11762","state":["exists","up"]},{"osd":4,"uuid":"3ca7b438-38b2-4689-8b18-fb5a8ddf6aee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":12114},{"type":"v1","addr":"172.21.15.47:6825","nonce":12114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":12114},{"type":"v1","addr":"172.21.15.47:6827","nonce":12114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":12114},{"type":"v1","addr":"172.21.15.47:6831","nonce":12114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":12114},{"type":"v1","addr":"172.21.15.47:6829","nonce":12114}]},"public_addr":"172.21.15.47:6825/12114","cluster_addr":"172.21.15.47:6827/12114","heartbeat_back_addr":"172.21.15.47:6831/12114","heartbeat_front_addr":"172.21.15.47:6829/12114","state":["exists","up"]},{"osd":5,"uuid":"6a45fca7-83d3-44fd-a1d4-b01e4201a101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":12111},{"type":"v1","addr":"172.21.15.47:6801","nonce":12111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":12111},{"type":"v1","addr":"172.21.15.47:6803","nonce":12111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":12111},{"type":"v1","addr":"172.21.15.47:6807","nonce":12111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":12111},{"type":"v1","addr":"172.21.15.47:6805","nonce":12111}]},"public_addr":"172.21.15.47:6801/12111","cluster_addr":"172.21.15.47:6803/12111","heartbeat_back_addr":"172.21.15.47:6807/12111","heartbeat_front_addr":"172.21.15.47:6805/12111","state":["exists","up"]},{"osd":6,"uuid":"c4b1b9ed-61f8-4677-9755-dff59a30e447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":12113},{"type":"v1","addr":"172.21.15.47:6817","nonce":12113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":12113},{"type":"v1","addr":"172.21.15.47:6819","nonce":12113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":12113},{"type":"v1","addr":"172.21.15.47:6823","nonce":12113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":12113},{"type":"v1","addr":"172.21.15.47:6821","nonce":12113}]},"public_addr":"172.21.15.47:6817/12113","cluster_addr":"172.21.15.47:6819/12113","heartbeat_back_addr":"172.21.15.47:6823/12113","heartbeat_front_addr":"172.21.15.47:6821/12113","state":["exists","up"]},{"osd":7,"uuid":"4608e4ed-cc40-4c5c-ba8b-3779bbc62f65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":12112},{"type":"v1","addr":"172.21.15.47:6809","nonce":12112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":12112},{"type":"v1","addr":"172.21.15.47:6811","nonce":12112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":12112},{"type":"v1","addr":"172.21.15.47:6815","nonce":12112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":12112},{"type":"v1","addr":"172.21.15.47:6813","nonce":12112}]},"public_addr":"172.21.15.47:6809/12112","cluster_addr":"172.21.15.47:6811/12112","heartbeat_back_addr":"172.21.15.47:6815/12112","heartbeat_front_addr":"172.21.15.47:6813/12112","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.435265+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.472776+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.388370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.417366+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.102392+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.116529+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.167430+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.148139+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:04:56.703 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.701+0000 7f630b474700 1 -- 172.21.15.189:0/220898942 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f62dc0557c0 msgr2=0x7f62dc057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:56.704 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.701+0000 7f630b474700 1 --2- 172.21.15.189:0/220898942 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f62dc0557c0 0x7f62dc057c80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f6300000f40 tx=0x7f6300009040).stop 2020-11-20T12:04:56.704 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.701+0000 7f630b474700 1 -- 172.21.15.189:0/220898942 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6304067b50 msgr2=0x7f6304127f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:56.705 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.701+0000 7f630b474700 1 --2- 172.21.15.189:0/220898942 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6304067b50 0x7f6304127f40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f62f400cb80 tx=0x7f62f4008040).stop 2020-11-20T12:04:56.705 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.701+0000 7f630b474700 1 -- 172.21.15.189:0/220898942 shutdown_connections 2020-11-20T12:04:56.705 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.701+0000 7f630b474700 1 --2- 172.21.15.189:0/220898942 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f62dc0557c0 0x7f62dc057c80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.706 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.701+0000 7f630b474700 1 --2- 172.21.15.189:0/220898942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f63041374a0 0x7f63040abe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.706 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.701+0000 7f630b474700 1 --2- 172.21.15.189:0/220898942 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f63040612b0 0x7f63040a92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.707 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.701+0000 7f630b474700 1 --2- 172.21.15.189:0/220898942 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6304067b50 0x7f6304127f40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.707 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.701+0000 7f630b474700 1 -- 172.21.15.189:0/220898942 >> 172.21.15.189:0/220898942 conn(0x7f6304063210 msgr2=0x7f63040ad2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:56.707 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.701+0000 7f630b474700 1 -- 172.21.15.189:0/220898942 shutdown_connections 2020-11-20T12:04:56.708 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.701+0000 7f630b474700 1 -- 172.21.15.189:0/220898942 wait complete. 2020-11-20T12:04:56.716 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:04:56.717 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-11-20T12:04:56.841 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.841+0000 7fe6e96af700 1 Processor -- start 2020-11-20T12:04:56.842 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.841+0000 7fe6e96af700 1 -- start start 2020-11-20T12:04:56.844 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.841+0000 7fe6e96af700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe6e413bc80 0x7fe6e414a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:56.844 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.841+0000 7fe6e96af700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e4066800 con 0x7fe6e40a8840 2020-11-20T12:04:56.844 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.841+0000 7fe6e96af700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e4066aa0 con 0x7fe6e4066ed0 2020-11-20T12:04:56.844 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.841+0000 7fe6e96af700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e4067450 con 0x7fe6e413bc80 2020-11-20T12:04:56.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.841+0000 7fe6e37fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe6e4066ed0 0x7fe6e413b570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49944/0 (socket says 172.21.15.189:49944) 2020-11-20T12:04:56.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.841+0000 7fe6e37fe700 1 -- 172.21.15.189:0/2083951991 learned_addr learned my addr 172.21.15.189:0/2083951991 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:04:56.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.841+0000 7fe6e8eae700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe6e413bc80 0x7fe6e414a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:56.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.841+0000 7fe6e2ffd700 1 -- 172.21.15.189:0/2083951991 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359976321 0 0) 0x7fe6e4066aa0 con 0x7fe6e4066ed0 2020-11-20T12:04:56.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.841+0000 7fe6e2ffd700 1 -- 172.21.15.189:0/2083951991 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6c8003240 con 0x7fe6e4066ed0 2020-11-20T12:04:56.846 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.841+0000 7fe6e8eae700 1 -- 172.21.15.189:0/2083951991 >> v1:172.21.15.189:6789/0 conn(0x7fe6e4066ed0 legacy=0x7fe6e413b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:56.846 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.841+0000 7fe6e8eae700 1 -- 172.21.15.189:0/2083951991 >> v1:172.21.15.47:6789/0 conn(0x7fe6e40a8840 legacy=0x7fe6e413eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:56.846 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.841+0000 7fe6e8eae700 1 -- 172.21.15.189:0/2083951991 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e40676f0 con 0x7fe6e413bc80 2020-11-20T12:04:56.846 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.841+0000 7fe6e8eae700 1 --2- 172.21.15.189:0/2083951991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6e413bc80 0x7fe6e414a5f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fe6dc00b9a0 tx=0x7fe6dc006df0).ready entity=mon.1 client_cookie=43aa200c9d94f51e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:56.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.841+0000 7fe6e2ffd700 1 -- 172.21.15.189:0/2083951991 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc0097b0 con 0x7fe6e413bc80 2020-11-20T12:04:56.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.841+0000 7fe6e2ffd700 1 -- 172.21.15.189:0/2083951991 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6dc009910 con 0x7fe6e413bc80 2020-11-20T12:04:56.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.841+0000 7fe6e2ffd700 1 -- 172.21.15.189:0/2083951991 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc009c90 con 0x7fe6e413bc80 2020-11-20T12:04:56.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.841+0000 7fe6e96af700 1 -- 172.21.15.189:0/2083951991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6e413bc80 msgr2=0x7fe6e414a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:56.848 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.841+0000 7fe6e96af700 1 --2- 172.21.15.189:0/2083951991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6e413bc80 0x7fe6e414a5f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fe6dc00b9a0 tx=0x7fe6dc006df0).stop 2020-11-20T12:04:56.848 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.841+0000 7fe6e96af700 1 -- 172.21.15.189:0/2083951991 shutdown_connections 2020-11-20T12:04:56.848 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.841+0000 7fe6e96af700 1 --2- 172.21.15.189:0/2083951991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6e413bc80 0x7fe6e414a5f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.848 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.841+0000 7fe6e96af700 1 -- 172.21.15.189:0/2083951991 >> 172.21.15.189:0/2083951991 conn(0x7fe6e4060170 msgr2=0x7fe6e4060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:56.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6e96af700 1 -- 172.21.15.189:0/2083951991 shutdown_connections 2020-11-20T12:04:56.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6e96af700 1 -- 172.21.15.189:0/2083951991 wait complete. 2020-11-20T12:04:56.852 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6e96af700 1 Processor -- start 2020-11-20T12:04:56.853 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6e96af700 1 -- start start 2020-11-20T12:04:56.853 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6e96af700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe6e4066ed0 0x7fe6e413c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:56.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6e96af700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6e40a8840 0x7fe6e413e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:56.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6e96af700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6e413bc80 0x7fe6e413d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:56.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6e96af700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e414dc90 con 0x7fe6e40a8840 2020-11-20T12:04:56.855 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6e96af700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e414df30 con 0x7fe6e413bc80 2020-11-20T12:04:56.855 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6e96af700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e414e1d0 con 0x7fe6e4066ed0 2020-11-20T12:04:56.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6e8eae700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6e413bc80 0x7fe6e413d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:56.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6e8eae700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6e413bc80 0x7fe6e413d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48332/0 (socket says 172.21.15.189:48332) 2020-11-20T12:04:56.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6e8eae700 1 -- 172.21.15.189:0/43081688 learned_addr learned my addr 172.21.15.189:0/43081688 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:56.857 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6e3fff700 1 --2- 172.21.15.189:0/43081688 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe6e4066ed0 0x7fe6e413c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:56.857 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6e37fe700 1 --2- 172.21.15.189:0/43081688 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6e40a8840 0x7fe6e413e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:56.858 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6e8eae700 1 -- 172.21.15.189:0/43081688 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe6e4066ed0 msgr2=0x7fe6e413c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:56.858 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6e8eae700 1 --2- 172.21.15.189:0/43081688 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe6e4066ed0 0x7fe6e413c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.859 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6e8eae700 1 -- 172.21.15.189:0/43081688 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6e40a8840 msgr2=0x7fe6e413e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:56.859 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6e8eae700 1 --2- 172.21.15.189:0/43081688 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6e40a8840 0x7fe6e413e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:56.859 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6e8eae700 1 -- 172.21.15.189:0/43081688 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e414e470 con 0x7fe6e413bc80 2020-11-20T12:04:56.860 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6e37fe700 1 --2- 172.21.15.189:0/43081688 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6e40a8840 0x7fe6e413e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:04:56.860 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6e8eae700 1 --2- 172.21.15.189:0/43081688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6e413bc80 0x7fe6e413d320 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fe6dc006e30 tx=0x7fe6dc015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:56.861 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6dbfff700 1 -- 172.21.15.189:0/43081688 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc009e40 con 0x7fe6e413bc80 2020-11-20T12:04:56.861 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6dbfff700 1 -- 172.21.15.189:0/43081688 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6dc01d5d0 con 0x7fe6e413bc80 2020-11-20T12:04:56.861 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6e96af700 1 -- 172.21.15.189:0/43081688 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e413ccc0 con 0x7fe6e413bc80 2020-11-20T12:04:56.862 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6dbfff700 1 -- 172.21.15.189:0/43081688 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc01d970 con 0x7fe6e413bc80 2020-11-20T12:04:56.862 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6e96af700 1 -- 172.21.15.189:0/43081688 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e414e910 con 0x7fe6e413bc80 2020-11-20T12:04:56.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.845+0000 7fe6e96af700 1 -- 172.21.15.189:0/43081688 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e413ea00 con 0x7fe6e413bc80 2020-11-20T12:04:56.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.849+0000 7fe6dbfff700 1 -- 172.21.15.189:0/43081688 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fe6dc01a040 con 0x7fe6e413bc80 2020-11-20T12:04:56.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.849+0000 7fe6dbfff700 1 --2- 172.21.15.189:0/43081688 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe6b40557c0 0x7fe6b4057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:56.864 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.849+0000 7fe6e3fff700 1 --2- 172.21.15.189:0/43081688 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe6b40557c0 0x7fe6b4057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:56.864 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.849+0000 7fe6dbfff700 1 -- 172.21.15.189:0/43081688 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe6dc06fea0 con 0x7fe6e413bc80 2020-11-20T12:04:56.865 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.849+0000 7fe6e3fff700 1 --2- 172.21.15.189:0/43081688 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe6b40557c0 0x7fe6b4057c80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fe6cc003610 tx=0x7fe6cc000b80).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:56.866 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:56.853+0000 7fe6dbfff700 1 -- 172.21.15.189:0/43081688 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe6e413ea00 con 0x7fe6e413bc80 2020-11-20T12:04:57.007 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.005+0000 7fe6e96af700 1 -- 172.21.15.189:0/43081688 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6e413ea00 con 0x7fe6e413bc80 2020-11-20T12:04:57.008 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.009+0000 7fe6dbfff700 1 -- 172.21.15.189:0/43081688 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16826 (secure 0 0 0) 0x7fe6e413ea00 con 0x7fe6e413bc80 2020-11-20T12:04:57.009 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:57.009 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":22,"fsid":"cda18d0e-64b3-41bb-964a-084069c3e648","created":"2020-11-20T12:04:37.538706+0000","modified":"2020-11-20T12:04:55.015818+0000","last_up_change":"2020-11-20T12:04:43.006631+0000","last_in_change":"2020-11-20T12:04:43.006631+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:41.395395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:48.550454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:52.101837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:53.018196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c81214e1-4f84-4aaa-8ceb-99c398b2a89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6811","nonce":11760},{"type":"v1","addr":"172.21.15.189:6814","nonce":11760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":11760},{"type":"v1","addr":"172.21.15.189:6818","nonce":11760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":11760},{"type":"v1","addr":"172.21.15.189:6828","nonce":11760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6821","nonce":11760},{"type":"v1","addr":"172.21.15.189:6824","nonce":11760}]},"public_addr":"172.21.15.189:6814/11760","cluster_addr":"172.21.15.189:6818/11760","heartbeat_back_addr":"172.21.15.189:6828/11760","heartbeat_front_addr":"172.21.15.189:6824/11760","state":["exists","up"]},{"osd":1,"uuid":"d2f3ec7d-d403-484f-b5ed-b2e6b079d63d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6805","nonce":11761},{"type":"v1","addr":"172.21.15.189:6807","nonce":11761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":11761},{"type":"v1","addr":"172.21.15.189:6812","nonce":11761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6817","nonce":11761},{"type":"v1","addr":"172.21.15.189:6820","nonce":11761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6813","nonce":11761},{"type":"v1","addr":"172.21.15.189:6815","nonce":11761}]},"public_addr":"172.21.15.189:6807/11761","cluster_addr":"172.21.15.189:6812/11761","heartbeat_back_addr":"172.21.15.189:6820/11761","heartbeat_front_addr":"172.21.15.189:6815/11761","state":["exists","up"]},{"osd":2,"uuid":"e6cf4790-000c-43eb-8844-175f04c60503","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6819","nonce":11751},{"type":"v1","addr":"172.21.15.189:6822","nonce":11751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":11751},{"type":"v1","addr":"172.21.15.189:6825","nonce":11751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":11751},{"type":"v1","addr":"172.21.15.189:6831","nonce":11751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6827","nonce":11751},{"type":"v1","addr":"172.21.15.189:6829","nonce":11751}]},"public_addr":"172.21.15.189:6822/11751","cluster_addr":"172.21.15.189:6825/11751","heartbeat_back_addr":"172.21.15.189:6831/11751","heartbeat_front_addr":"172.21.15.189:6829/11751","state":["exists","up"]},{"osd":3,"uuid":"ba49dfac-f2b9-476e-853d-42f05e21a77a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":11762},{"type":"v1","addr":"172.21.15.189:6801","nonce":11762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":11762},{"type":"v1","addr":"172.21.15.189:6803","nonce":11762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":11762},{"type":"v1","addr":"172.21.15.189:6809","nonce":11762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":11762},{"type":"v1","addr":"172.21.15.189:6806","nonce":11762}]},"public_addr":"172.21.15.189:6801/11762","cluster_addr":"172.21.15.189:6803/11762","heartbeat_back_addr":"172.21.15.189:6809/11762","heartbeat_front_addr":"172.21.15.189:6806/11762","state":["exists","up"]},{"osd":4,"uuid":"3ca7b438-38b2-4689-8b18-fb5a8ddf6aee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":12114},{"type":"v1","addr":"172.21.15.47:6825","nonce":12114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":12114},{"type":"v1","addr":"172.21.15.47:6827","nonce":12114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":12114},{"type":"v1","addr":"172.21.15.47:6831","nonce":12114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":12114},{"type":"v1","addr":"172.21.15.47:6829","nonce":12114}]},"public_addr":"172.21.15.47:6825/12114","cluster_addr":"172.21.15.47:6827/12114","heartbeat_back_addr":"172.21.15.47:6831/12114","heartbeat_front_addr":"172.21.15.47:6829/12114","state":["exists","up"]},{"osd":5,"uuid":"6a45fca7-83d3-44fd-a1d4-b01e4201a101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":12111},{"type":"v1","addr":"172.21.15.47:6801","nonce":12111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":12111},{"type":"v1","addr":"172.21.15.47:6803","nonce":12111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":12111},{"type":"v1","addr":"172.21.15.47:6807","nonce":12111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":12111},{"type":"v1","addr":"172.21.15.47:6805","nonce":12111}]},"public_addr":"172.21.15.47:6801/12111","cluster_addr":"172.21.15.47:6803/12111","heartbeat_back_addr":"172.21.15.47:6807/12111","heartbeat_front_addr":"172.21.15.47:6805/12111","state":["exists","up"]},{"osd":6,"uuid":"c4b1b9ed-61f8-4677-9755-dff59a30e447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":12113},{"type":"v1","addr":"172.21.15.47:6817","nonce":12113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":12113},{"type":"v1","addr":"172.21.15.47:6819","nonce":12113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":12113},{"type":"v1","addr":"172.21.15.47:6823","nonce":12113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":12113},{"type":"v1","addr":"172.21.15.47:6821","nonce":12113}]},"public_addr":"172.21.15.47:6817/12113","cluster_addr":"172.21.15.47:6819/12113","heartbeat_back_addr":"172.21.15.47:6823/12113","heartbeat_front_addr":"172.21.15.47:6821/12113","state":["exists","up"]},{"osd":7,"uuid":"4608e4ed-cc40-4c5c-ba8b-3779bbc62f65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":12112},{"type":"v1","addr":"172.21.15.47:6809","nonce":12112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":12112},{"type":"v1","addr":"172.21.15.47:6811","nonce":12112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":12112},{"type":"v1","addr":"172.21.15.47:6815","nonce":12112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":12112},{"type":"v1","addr":"172.21.15.47:6813","nonce":12112}]},"public_addr":"172.21.15.47:6809/12112","cluster_addr":"172.21.15.47:6811/12112","heartbeat_back_addr":"172.21.15.47:6815/12112","heartbeat_front_addr":"172.21.15.47:6813/12112","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.435265+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.472776+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.388370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.417366+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.102392+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.116529+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.167430+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.148139+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:04:57.011 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.009+0000 7fe6e96af700 1 -- 172.21.15.189:0/43081688 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe6b40557c0 msgr2=0x7fe6b4057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.011 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.009+0000 7fe6e96af700 1 --2- 172.21.15.189:0/43081688 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe6b40557c0 0x7fe6b4057c80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fe6cc003610 tx=0x7fe6cc000b80).stop 2020-11-20T12:04:57.011 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.009+0000 7fe6e96af700 1 -- 172.21.15.189:0/43081688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6e413bc80 msgr2=0x7fe6e413d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.011 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.009+0000 7fe6e96af700 1 --2- 172.21.15.189:0/43081688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6e413bc80 0x7fe6e413d320 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fe6dc006e30 tx=0x7fe6dc015000).stop 2020-11-20T12:04:57.011 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.009+0000 7fe6e96af700 1 -- 172.21.15.189:0/43081688 shutdown_connections 2020-11-20T12:04:57.012 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.009+0000 7fe6e96af700 1 --2- 172.21.15.189:0/43081688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6e413bc80 0x7fe6e413d320 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.012 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.009+0000 7fe6e96af700 1 --2- 172.21.15.189:0/43081688 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe6e4066ed0 0x7fe6e413c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.012 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.009+0000 7fe6e96af700 1 --2- 172.21.15.189:0/43081688 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe6b40557c0 0x7fe6b4057c80 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.012 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.009+0000 7fe6e96af700 1 --2- 172.21.15.189:0/43081688 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6e40a8840 0x7fe6e413e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.013 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.009+0000 7fe6e96af700 1 -- 172.21.15.189:0/43081688 >> 172.21.15.189:0/43081688 conn(0x7fe6e4060170 msgr2=0x7fe6e405ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:57.013 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.009+0000 7fe6e96af700 1 -- 172.21.15.189:0/43081688 shutdown_connections 2020-11-20T12:04:57.013 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.009+0000 7fe6e96af700 1 -- 172.21.15.189:0/43081688 wait complete. 2020-11-20T12:04:57.025 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:41.395395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:48.550454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:04:52.101837+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:04:53.018196+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:04:57.026 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:04:57.150 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.149+0000 7fae3dec7700 1 Processor -- start 2020-11-20T12:04:57.151 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.149+0000 7fae3dec7700 1 -- start start 2020-11-20T12:04:57.152 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.149+0000 7fae3dec7700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fae38062930 0x7fae380b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:57.152 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.149+0000 7fae3dec7700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae38066fb0 con 0x7fae380b17c0 2020-11-20T12:04:57.152 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.149+0000 7fae3dec7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae38067250 con 0x7fae38066a30 2020-11-20T12:04:57.152 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.149+0000 7fae3dec7700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae380666b0 con 0x7fae38062930 2020-11-20T12:04:57.153 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.153+0000 7fae37fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fae38066a30 0x7fae381373e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:49960/0 (socket says 172.21.15.189:49960) 2020-11-20T12:04:57.153 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.153+0000 7fae37fff700 1 -- 172.21.15.189:0/2668307323 learned_addr learned my addr 172.21.15.189:0/2668307323 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:04:57.153 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.153+0000 7fae3cec5700 1 --2- 172.21.15.189:0/2668307323 >> v2:172.21.15.47:3300/0 conn(0x7fae38062930 0x7fae380b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:57.154 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.153+0000 7fae377fe700 1 -- 172.21.15.189:0/2668307323 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098469414 0 0) 0x7fae38067250 con 0x7fae38066a30 2020-11-20T12:04:57.155 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.153+0000 7fae377fe700 1 -- 172.21.15.189:0/2668307323 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae18003240 con 0x7fae38066a30 2020-11-20T12:04:57.155 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.153+0000 7fae3cec5700 1 -- 172.21.15.189:0/2668307323 >> v1:172.21.15.189:6789/0 conn(0x7fae38066a30 legacy=0x7fae381373e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.155 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.153+0000 7fae3cec5700 1 -- 172.21.15.189:0/2668307323 >> v1:172.21.15.47:6789/0 conn(0x7fae380b17c0 legacy=0x7fae38137af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.155 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.153+0000 7fae3cec5700 1 -- 172.21.15.189:0/2668307323 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae380b0e30 con 0x7fae38062930 2020-11-20T12:04:57.156 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.153+0000 7fae3cec5700 1 --2- 172.21.15.189:0/2668307323 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae38062930 0x7fae380b12d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fae2800b9a0 tx=0x7fae28006df0).ready entity=mon.0 client_cookie=99bf75416a5c5ca server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:57.156 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.153+0000 7fae377fe700 1 -- 172.21.15.189:0/2668307323 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae280097b0 con 0x7fae38062930 2020-11-20T12:04:57.156 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.153+0000 7fae377fe700 1 -- 172.21.15.189:0/2668307323 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae28009910 con 0x7fae38062930 2020-11-20T12:04:57.156 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.153+0000 7fae377fe700 1 -- 172.21.15.189:0/2668307323 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae28009c90 con 0x7fae38062930 2020-11-20T12:04:57.156 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.153+0000 7fae3dec7700 1 -- 172.21.15.189:0/2668307323 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae38062930 msgr2=0x7fae380b12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.157 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.153+0000 7fae3dec7700 1 --2- 172.21.15.189:0/2668307323 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae38062930 0x7fae380b12d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fae2800b9a0 tx=0x7fae28006df0).stop 2020-11-20T12:04:57.157 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.153+0000 7fae3dec7700 1 -- 172.21.15.189:0/2668307323 shutdown_connections 2020-11-20T12:04:57.157 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.153+0000 7fae3dec7700 1 --2- 172.21.15.189:0/2668307323 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae38062930 0x7fae380b12d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.157 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.153+0000 7fae3dec7700 1 -- 172.21.15.189:0/2668307323 >> 172.21.15.189:0/2668307323 conn(0x7fae38065080 msgr2=0x7fae38062080 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:57.157 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.153+0000 7fae3dec7700 1 -- 172.21.15.189:0/2668307323 shutdown_connections 2020-11-20T12:04:57.158 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.153+0000 7fae3dec7700 1 -- 172.21.15.189:0/2668307323 wait complete. 2020-11-20T12:04:57.159 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.153+0000 7fae3dec7700 1 Processor -- start 2020-11-20T12:04:57.160 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.157+0000 7fae3dec7700 1 -- start start 2020-11-20T12:04:57.160 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.157+0000 7fae3dec7700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fae38062930 0x7fae380a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:57.160 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.157+0000 7fae3dec7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae38066a30 0x7fae380a94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:57.161 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.157+0000 7fae3dec7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae380b17c0 0x7fae38149ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:57.161 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.157+0000 7fae3dec7700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3813e280 con 0x7fae38066a30 2020-11-20T12:04:57.161 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.157+0000 7fae3dec7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae380a9d50 con 0x7fae380b17c0 2020-11-20T12:04:57.161 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.157+0000 7fae3dec7700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae380a9ff0 con 0x7fae38062930 2020-11-20T12:04:57.162 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.157+0000 7fae3d6c6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae380b17c0 0x7fae38149ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:57.162 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.157+0000 7fae3d6c6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae380b17c0 0x7fae38149ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48346/0 (socket says 172.21.15.189:48346) 2020-11-20T12:04:57.162 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.157+0000 7fae3cec5700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fae38062930 0x7fae380a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:57.162 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.157+0000 7fae3d6c6700 1 -- 172.21.15.189:0/107078134 learned_addr learned my addr 172.21.15.189:0/107078134 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:57.162 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.157+0000 7fae37fff700 1 --2- 172.21.15.189:0/107078134 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae38066a30 0x7fae380a94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:57.163 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.157+0000 7fae3d6c6700 1 -- 172.21.15.189:0/107078134 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fae38062930 msgr2=0x7fae380a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.163 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.157+0000 7fae3d6c6700 1 --2- 172.21.15.189:0/107078134 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fae38062930 0x7fae380a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.163 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.157+0000 7fae3d6c6700 1 -- 172.21.15.189:0/107078134 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae38066a30 msgr2=0x7fae380a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.163 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.157+0000 7fae3d6c6700 1 --2- 172.21.15.189:0/107078134 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae38066a30 0x7fae380a94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.163 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.157+0000 7fae3d6c6700 1 -- 172.21.15.189:0/107078134 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae380aa290 con 0x7fae380b17c0 2020-11-20T12:04:57.164 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.157+0000 7fae37fff700 1 --2- 172.21.15.189:0/107078134 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae38066a30 0x7fae380a94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:57.164 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.157+0000 7fae3d6c6700 1 --2- 172.21.15.189:0/107078134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae380b17c0 0x7fae38149ed0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fae3000ca90 tx=0x7fae30001360).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:57.164 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.157+0000 7fae34ff9700 1 -- 172.21.15.189:0/107078134 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3000a7c0 con 0x7fae380b17c0 2020-11-20T12:04:57.164 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.157+0000 7fae34ff9700 1 -- 172.21.15.189:0/107078134 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3000a920 con 0x7fae380b17c0 2020-11-20T12:04:57.165 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.157+0000 7fae3dec7700 1 -- 172.21.15.189:0/107078134 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae380a99a0 con 0x7fae380b17c0 2020-11-20T12:04:57.165 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.157+0000 7fae34ff9700 1 -- 172.21.15.189:0/107078134 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3000acc0 con 0x7fae380b17c0 2020-11-20T12:04:57.165 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.157+0000 7fae3dec7700 1 -- 172.21.15.189:0/107078134 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae3813e810 con 0x7fae380b17c0 2020-11-20T12:04:57.165 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.161+0000 7fae3dec7700 1 -- 172.21.15.189:0/107078134 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae380a9bc0 con 0x7fae380b17c0 2020-11-20T12:04:57.166 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.161+0000 7fae34ff9700 1 -- 172.21.15.189:0/107078134 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fae30019030 con 0x7fae380b17c0 2020-11-20T12:04:57.167 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.161+0000 7fae34ff9700 1 --2- 172.21.15.189:0/107078134 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fae140557c0 0x7fae14057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:57.167 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.161+0000 7fae34ff9700 1 -- 172.21.15.189:0/107078134 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fae3006deb0 con 0x7fae380b17c0 2020-11-20T12:04:57.167 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.161+0000 7fae3cec5700 1 --2- 172.21.15.189:0/107078134 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fae140557c0 0x7fae14057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:57.167 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.161+0000 7fae3cec5700 1 --2- 172.21.15.189:0/107078134 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fae140557c0 0x7fae14057c80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fae28006e30 tx=0x7fae28015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:57.168 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.165+0000 7fae34ff9700 1 -- 172.21.15.189:0/107078134 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fae30043e00 con 0x7fae380b17c0 2020-11-20T12:04:57.320 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.317+0000 7fae3dec7700 1 -- 172.21.15.189:0/107078134 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fae380adfb0 con 0x7fae380b17c0 2020-11-20T12:04:57.321 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.317+0000 7fae34ff9700 1 -- 172.21.15.189:0/107078134 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v22) v1 ==== 110+0+10 (secure 0 0 0) 0x7fae30047420 con 0x7fae380b17c0 2020-11-20T12:04:57.321 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.317+0000 7fae3dec7700 1 -- 172.21.15.189:0/107078134 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fae140557c0 msgr2=0x7fae14057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.321 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.317+0000 7fae3dec7700 1 --2- 172.21.15.189:0/107078134 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fae140557c0 0x7fae14057c80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fae28006e30 tx=0x7fae28015000).stop 2020-11-20T12:04:57.321 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.317+0000 7fae3dec7700 1 -- 172.21.15.189:0/107078134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae380b17c0 msgr2=0x7fae38149ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.321 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.317+0000 7fae3dec7700 1 --2- 172.21.15.189:0/107078134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae380b17c0 0x7fae38149ed0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fae3000ca90 tx=0x7fae30001360).stop 2020-11-20T12:04:57.322 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.317+0000 7fae3dec7700 1 -- 172.21.15.189:0/107078134 shutdown_connections 2020-11-20T12:04:57.322 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.317+0000 7fae3dec7700 1 --2- 172.21.15.189:0/107078134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae380b17c0 0x7fae38149ed0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.322 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.317+0000 7fae3dec7700 1 --2- 172.21.15.189:0/107078134 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fae38062930 0x7fae380a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.322 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.317+0000 7fae3dec7700 1 --2- 172.21.15.189:0/107078134 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fae140557c0 0x7fae14057c80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.322 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.317+0000 7fae3dec7700 1 --2- 172.21.15.189:0/107078134 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae38066a30 0x7fae380a94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.323 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.317+0000 7fae3dec7700 1 -- 172.21.15.189:0/107078134 >> 172.21.15.189:0/107078134 conn(0x7fae38065080 msgr2=0x7fae380ac660 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:57.323 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.317+0000 7fae3dec7700 1 -- 172.21.15.189:0/107078134 shutdown_connections 2020-11-20T12:04:57.323 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.317+0000 7fae3dec7700 1 -- 172.21.15.189:0/107078134 wait complete. 2020-11-20T12:04:57.323 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2020-11-20T12:04:57.335 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:04:57.466 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.465+0000 7f212171f700 1 Processor -- start 2020-11-20T12:04:57.466 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.465+0000 7f212171f700 1 -- start start 2020-11-20T12:04:57.467 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.465+0000 7f212171f700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f211c0ae0d0 0x7f211c13aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:57.467 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.465+0000 7f212171f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f211c060880 0x7f211c1451e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:57.468 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.465+0000 7f212171f700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f211c066960 con 0x7f211c13af60 2020-11-20T12:04:57.468 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.465+0000 7f212171f700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f211c13a680 con 0x7f211c0ae0d0 2020-11-20T12:04:57.468 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.465+0000 7f212171f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f211c067110 con 0x7f211c060880 2020-11-20T12:04:57.468 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.465+0000 7f211b7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f211c060880 0x7f211c1451e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:57.469 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.465+0000 7f211b7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f211c060880 0x7f211c1451e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48354/0 (socket says 172.21.15.189:48354) 2020-11-20T12:04:57.469 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.465+0000 7f211b7fe700 1 -- 172.21.15.189:0/1476031612 learned_addr learned my addr 172.21.15.189:0/1476031612 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:57.470 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.465+0000 7f211affd700 1 --2- 172.21.15.189:0/1476031612 >> v2:172.21.15.47:3300/0 conn(0x7f211c0ae0d0 0x7f211c13aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:57.471 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.465+0000 7f211b7fe700 1 -- 172.21.15.189:0/1476031612 >> v2:172.21.15.47:3300/0 conn(0x7f211c0ae0d0 msgr2=0x7f211c13aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:57.472 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.465+0000 7f211b7fe700 1 --2- 172.21.15.189:0/1476031612 >> v2:172.21.15.47:3300/0 conn(0x7f211c0ae0d0 0x7f211c13aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.472 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.465+0000 7f211b7fe700 1 -- 172.21.15.189:0/1476031612 >> v1:172.21.15.47:6789/0 conn(0x7f211c13af60 legacy=0x7f211c060170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.472 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.469+0000 7f211b7fe700 1 -- 172.21.15.189:0/1476031612 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f211c0673b0 con 0x7f211c060880 2020-11-20T12:04:57.472 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.469+0000 7f211b7fe700 1 --2- 172.21.15.189:0/1476031612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f211c060880 0x7f211c1451e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f2114009d50 tx=0x7f2114009290).ready entity=mon.1 client_cookie=314e4e7cd2cd5d02 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:57.472 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.469+0000 7f211a7fc700 1 -- 172.21.15.189:0/1476031612 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211400bb80 con 0x7f211c060880 2020-11-20T12:04:57.473 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.469+0000 7f211a7fc700 1 -- 172.21.15.189:0/1476031612 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f211400bce0 con 0x7f211c060880 2020-11-20T12:04:57.473 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.469+0000 7f211a7fc700 1 -- 172.21.15.189:0/1476031612 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2114011610 con 0x7f211c060880 2020-11-20T12:04:57.473 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.469+0000 7f212171f700 1 -- 172.21.15.189:0/1476031612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f211c060880 msgr2=0x7f211c1451e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.473 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.469+0000 7f212171f700 1 --2- 172.21.15.189:0/1476031612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f211c060880 0x7f211c1451e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f2114009d50 tx=0x7f2114009290).stop 2020-11-20T12:04:57.474 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.469+0000 7f212171f700 1 -- 172.21.15.189:0/1476031612 shutdown_connections 2020-11-20T12:04:57.474 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.469+0000 7f212171f700 1 --2- 172.21.15.189:0/1476031612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f211c060880 0x7f211c1451e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.474 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.469+0000 7f212171f700 1 --2- 172.21.15.189:0/1476031612 >> v2:172.21.15.47:3300/0 conn(0x7f211c0ae0d0 0x7f211c13aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.474 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.469+0000 7f212171f700 1 -- 172.21.15.189:0/1476031612 >> 172.21.15.189:0/1476031612 conn(0x7f211c0b1850 msgr2=0x7f211c0ad820 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:57.474 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.469+0000 7f212171f700 1 -- 172.21.15.189:0/1476031612 shutdown_connections 2020-11-20T12:04:57.475 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.469+0000 7f212171f700 1 -- 172.21.15.189:0/1476031612 wait complete. 2020-11-20T12:04:57.475 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.469+0000 7f212171f700 1 Processor -- start 2020-11-20T12:04:57.475 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.469+0000 7f212171f700 1 -- start start 2020-11-20T12:04:57.478 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.473+0000 7f212171f700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f211c060880 0x7f211c0ac0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:57.478 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.473+0000 7f212171f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f211c0ae0d0 0x7f211c0aa500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:57.478 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.473+0000 7f212171f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f211c13af60 0x7f211c14d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:57.478 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.473+0000 7f212171f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211c0a9300 con 0x7f211c13af60 2020-11-20T12:04:57.479 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.473+0000 7f212171f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211c0a95a0 con 0x7f211c0ae0d0 2020-11-20T12:04:57.479 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.473+0000 7f212171f700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f211c0a9840 con 0x7f211c060880 2020-11-20T12:04:57.479 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.473+0000 7f2113fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f211c0ae0d0 0x7f211c0aa500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:57.479 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.473+0000 7f2113fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f211c0ae0d0 0x7f211c0aa500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48358/0 (socket says 172.21.15.189:48358) 2020-11-20T12:04:57.479 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.473+0000 7f211affd700 1 --2- 172.21.15.189:0/524208564 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f211c060880 0x7f211c0ac0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:57.480 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.473+0000 7f2113fff700 1 -- 172.21.15.189:0/524208564 learned_addr learned my addr 172.21.15.189:0/524208564 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:57.480 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.473+0000 7f211b7fe700 1 --2- 172.21.15.189:0/524208564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f211c13af60 0x7f211c14d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:57.480 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.473+0000 7f2113fff700 1 -- 172.21.15.189:0/524208564 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f211c060880 msgr2=0x7f211c0ac0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.480 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.473+0000 7f2113fff700 1 --2- 172.21.15.189:0/524208564 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f211c060880 0x7f211c0ac0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.480 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.473+0000 7f2113fff700 1 -- 172.21.15.189:0/524208564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f211c13af60 msgr2=0x7f211c14d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.481 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.473+0000 7f2113fff700 1 --2- 172.21.15.189:0/524208564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f211c13af60 0x7f211c14d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.481 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.473+0000 7f2113fff700 1 -- 172.21.15.189:0/524208564 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f211c0aa9f0 con 0x7f211c0ae0d0 2020-11-20T12:04:57.481 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.473+0000 7f2113fff700 1 --2- 172.21.15.189:0/524208564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f211c0ae0d0 0x7f211c0aa500 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f210400c6e0 tx=0x7f2104007c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:57.481 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.473+0000 7f2112ffd700 1 -- 172.21.15.189:0/524208564 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2104007020 con 0x7f211c0ae0d0 2020-11-20T12:04:57.482 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.473+0000 7f2112ffd700 1 -- 172.21.15.189:0/524208564 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2104007180 con 0x7f211c0ae0d0 2020-11-20T12:04:57.482 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.473+0000 7f2112ffd700 1 -- 172.21.15.189:0/524208564 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f210400e560 con 0x7f211c0ae0d0 2020-11-20T12:04:57.482 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.473+0000 7f212171f700 1 -- 172.21.15.189:0/524208564 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f211c0a8ed0 con 0x7f211c0ae0d0 2020-11-20T12:04:57.482 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.473+0000 7f212171f700 1 -- 172.21.15.189:0/524208564 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f211c14d840 con 0x7f211c0ae0d0 2020-11-20T12:04:57.483 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.473+0000 7f212171f700 1 -- 172.21.15.189:0/524208564 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f211c0ac5a0 con 0x7f211c0ae0d0 2020-11-20T12:04:57.483 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.477+0000 7f2112ffd700 1 -- 172.21.15.189:0/524208564 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f2104010030 con 0x7f211c0ae0d0 2020-11-20T12:04:57.483 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.477+0000 7f2112ffd700 1 --2- 172.21.15.189:0/524208564 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f20f80557c0 0x7f20f8057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:57.483 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.477+0000 7f2112ffd700 1 -- 172.21.15.189:0/524208564 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f210406dee0 con 0x7f211c0ae0d0 2020-11-20T12:04:57.484 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.477+0000 7f211affd700 1 --2- 172.21.15.189:0/524208564 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f20f80557c0 0x7f20f8057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:57.484 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.477+0000 7f211affd700 1 --2- 172.21.15.189:0/524208564 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f20f80557c0 0x7f20f8057c80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f210c006fd0 tx=0x7f210c006df0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:57.485 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.481+0000 7f2112ffd700 1 -- 172.21.15.189:0/524208564 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f211c0ac5a0 con 0x7f211c0ae0d0 2020-11-20T12:04:57.635 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.633+0000 7f212171f700 1 -- 172.21.15.189:0/524208564 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f211c0ac5a0 con 0x7f211c0ae0d0 2020-11-20T12:04:57.636 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.637+0000 7f2112ffd700 1 -- 172.21.15.189:0/524208564 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v22) v1 ==== 92+0+10 (secure 0 0 0) 0x7f211c0ac5a0 con 0x7f211c0ae0d0 2020-11-20T12:04:57.636 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 8 2020-11-20T12:04:57.638 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.637+0000 7f212171f700 1 -- 172.21.15.189:0/524208564 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f20f80557c0 msgr2=0x7f20f8057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.639 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.637+0000 7f212171f700 1 --2- 172.21.15.189:0/524208564 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f20f80557c0 0x7f20f8057c80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f210c006fd0 tx=0x7f210c006df0).stop 2020-11-20T12:04:57.639 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.637+0000 7f212171f700 1 -- 172.21.15.189:0/524208564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f211c0ae0d0 msgr2=0x7f211c0aa500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.639 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.637+0000 7f212171f700 1 --2- 172.21.15.189:0/524208564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f211c0ae0d0 0x7f211c0aa500 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f210400c6e0 tx=0x7f2104007c50).stop 2020-11-20T12:04:57.639 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.637+0000 7f212171f700 1 -- 172.21.15.189:0/524208564 shutdown_connections 2020-11-20T12:04:57.640 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.637+0000 7f212171f700 1 --2- 172.21.15.189:0/524208564 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f20f80557c0 0x7f20f8057c80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.640 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.637+0000 7f212171f700 1 --2- 172.21.15.189:0/524208564 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f211c13af60 0x7f211c14d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.640 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.637+0000 7f212171f700 1 --2- 172.21.15.189:0/524208564 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f211c060880 0x7f211c0ac0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.640 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.637+0000 7f212171f700 1 --2- 172.21.15.189:0/524208564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f211c0ae0d0 0x7f211c0aa500 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.640 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.637+0000 7f212171f700 1 -- 172.21.15.189:0/524208564 >> 172.21.15.189:0/524208564 conn(0x7f211c0b1850 msgr2=0x7f211c137830 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:57.641 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.637+0000 7f212171f700 1 -- 172.21.15.189:0/524208564 shutdown_connections 2020-11-20T12:04:57.641 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.637+0000 7f212171f700 1 -- 172.21.15.189:0/524208564 wait complete. 2020-11-20T12:04:57.655 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:04:57.783 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.781+0000 7f1e632a0700 1 Processor -- start 2020-11-20T12:04:57.783 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.781+0000 7f1e632a0700 1 -- start start 2020-11-20T12:04:57.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.781+0000 7f1e632a0700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1e5c063e30 0x7f1e5c0682e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:57.785 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.781+0000 7f1e632a0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1e5c13b0e0 0x7f1e5c0a7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:57.785 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.781+0000 7f1e632a0700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e5c0687d0 con 0x7f1e5c063a50 2020-11-20T12:04:57.785 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.781+0000 7f1e632a0700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e5c0a8550 con 0x7f1e5c13b0e0 2020-11-20T12:04:57.785 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.781+0000 7f1e632a0700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5c0a87f0 con 0x7f1e5c063e30 2020-11-20T12:04:57.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e62a9f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1e5c13b0e0 0x7f1e5c0a7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:57.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e62a9f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1e5c13b0e0 0x7f1e5c0a7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48368/0 (socket says 172.21.15.189:48368) 2020-11-20T12:04:57.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e62a9f700 1 -- 172.21.15.189:0/1437026877 learned_addr learned my addr 172.21.15.189:0/1437026877 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:57.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e61a9d700 1 --2- 172.21.15.189:0/1437026877 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1e5c063e30 0x7f1e5c0682e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:57.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e62a9f700 1 -- 172.21.15.189:0/1437026877 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1e5c063e30 msgr2=0x7f1e5c0682e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:57.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e62a9f700 1 --2- 172.21.15.189:0/1437026877 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1e5c063e30 0x7f1e5c0682e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e62a9f700 1 -- 172.21.15.189:0/1437026877 >> v1:172.21.15.47:6789/0 conn(0x7f1e5c063a50 legacy=0x7f1e5c13a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e62a9f700 1 -- 172.21.15.189:0/1437026877 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e5c0a8280 con 0x7f1e5c13b0e0 2020-11-20T12:04:57.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e62a9f700 1 --2- 172.21.15.189:0/1437026877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e5c13b0e0 0x7f1e5c0a7370 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f1e54006fd0 tx=0x7f1e54006eb0).ready entity=mon.1 client_cookie=2544bf0d73a3cbee server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:57.790 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e6129c700 1 -- 172.21.15.189:0/1437026877 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e540098a0 con 0x7f1e5c13b0e0 2020-11-20T12:04:57.790 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e6129c700 1 -- 172.21.15.189:0/1437026877 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e54009a00 con 0x7f1e5c13b0e0 2020-11-20T12:04:57.791 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e6129c700 1 -- 172.21.15.189:0/1437026877 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e54009d80 con 0x7f1e5c13b0e0 2020-11-20T12:04:57.791 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e632a0700 1 -- 172.21.15.189:0/1437026877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e5c13b0e0 msgr2=0x7f1e5c0a7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.791 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e632a0700 1 --2- 172.21.15.189:0/1437026877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e5c13b0e0 0x7f1e5c0a7370 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f1e54006fd0 tx=0x7f1e54006eb0).stop 2020-11-20T12:04:57.791 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e632a0700 1 -- 172.21.15.189:0/1437026877 shutdown_connections 2020-11-20T12:04:57.791 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e632a0700 1 --2- 172.21.15.189:0/1437026877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e5c13b0e0 0x7f1e5c0a7370 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.792 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e632a0700 1 --2- 172.21.15.189:0/1437026877 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1e5c063e30 0x7f1e5c0682e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.792 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e632a0700 1 -- 172.21.15.189:0/1437026877 >> 172.21.15.189:0/1437026877 conn(0x7f1e5c0ad8e0 msgr2=0x7f1e5c0adcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:57.792 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e632a0700 1 -- 172.21.15.189:0/1437026877 shutdown_connections 2020-11-20T12:04:57.792 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e632a0700 1 -- 172.21.15.189:0/1437026877 wait complete. 2020-11-20T12:04:57.792 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e632a0700 1 Processor -- start 2020-11-20T12:04:57.793 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e632a0700 1 -- start start 2020-11-20T12:04:57.793 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e632a0700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1e5c063a50 0x7f1e5c09eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:57.793 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e632a0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e5c063e30 0x7f1e5c0a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:57.793 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e632a0700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e5c13b0e0 0x7f1e5c149590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:57.793 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e632a0700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5c14a4f0 con 0x7f1e5c13b0e0 2020-11-20T12:04:57.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e632a0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5c14cee0 con 0x7f1e5c063e30 2020-11-20T12:04:57.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.785+0000 7f1e632a0700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5c14d180 con 0x7f1e5c063a50 2020-11-20T12:04:57.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.789+0000 7f1e61a9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e5c063e30 0x7f1e5c0a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:57.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.789+0000 7f1e61a9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e5c063e30 0x7f1e5c0a0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48372/0 (socket says 172.21.15.189:48372) 2020-11-20T12:04:57.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.789+0000 7f1e61a9d700 1 -- 172.21.15.189:0/602997574 learned_addr learned my addr 172.21.15.189:0/602997574 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:57.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.789+0000 7f1e62a9f700 1 --2- 172.21.15.189:0/602997574 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e5c13b0e0 0x7f1e5c149590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:57.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.789+0000 7f1e6229e700 1 --2- 172.21.15.189:0/602997574 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1e5c063a50 0x7f1e5c09eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:57.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.789+0000 7f1e61a9d700 1 -- 172.21.15.189:0/602997574 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1e5c063a50 msgr2=0x7f1e5c09eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.789+0000 7f1e61a9d700 1 --2- 172.21.15.189:0/602997574 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1e5c063a50 0x7f1e5c09eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.789+0000 7f1e61a9d700 1 -- 172.21.15.189:0/602997574 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e5c13b0e0 msgr2=0x7f1e5c149590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.796 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.789+0000 7f1e61a9d700 1 --2- 172.21.15.189:0/602997574 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e5c13b0e0 0x7f1e5c149590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.796 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.789+0000 7f1e61a9d700 1 -- 172.21.15.189:0/602997574 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e5c14d420 con 0x7f1e5c063e30 2020-11-20T12:04:57.796 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.789+0000 7f1e6229e700 1 --2- 172.21.15.189:0/602997574 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1e5c063a50 0x7f1e5c09eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:57.796 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.789+0000 7f1e61a9d700 1 --2- 172.21.15.189:0/602997574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e5c063e30 0x7f1e5c0a0630 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f1e4800eba0 tx=0x7f1e48006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:57.796 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.789+0000 7f1e5a7fc700 1 -- 172.21.15.189:0/602997574 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e4800c9b0 con 0x7f1e5c063e30 2020-11-20T12:04:57.797 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.789+0000 7f1e632a0700 1 -- 172.21.15.189:0/602997574 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e5c09cb00 con 0x7f1e5c063e30 2020-11-20T12:04:57.797 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.789+0000 7f1e5a7fc700 1 -- 172.21.15.189:0/602997574 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e4800cb10 con 0x7f1e5c063e30 2020-11-20T12:04:57.797 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.789+0000 7f1e632a0700 1 -- 172.21.15.189:0/602997574 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e5c14d8c0 con 0x7f1e5c063e30 2020-11-20T12:04:57.797 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.789+0000 7f1e5a7fc700 1 -- 172.21.15.189:0/602997574 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e4800ce90 con 0x7f1e5c063e30 2020-11-20T12:04:57.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.789+0000 7f1e632a0700 1 -- 172.21.15.189:0/602997574 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e5c0a7600 con 0x7f1e5c063e30 2020-11-20T12:04:57.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.793+0000 7f1e5a7fc700 1 -- 172.21.15.189:0/602997574 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f1e48012600 con 0x7f1e5c063e30 2020-11-20T12:04:57.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.793+0000 7f1e5a7fc700 1 --2- 172.21.15.189:0/602997574 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1e3c0557c0 0x7f1e3c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:57.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.793+0000 7f1e5a7fc700 1 -- 172.21.15.189:0/602997574 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1e4806eb20 con 0x7f1e5c063e30 2020-11-20T12:04:57.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.793+0000 7f1e6229e700 1 --2- 172.21.15.189:0/602997574 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1e3c0557c0 0x7f1e3c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:57.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.793+0000 7f1e6229e700 1 --2- 172.21.15.189:0/602997574 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1e3c0557c0 0x7f1e3c057c80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f1e50001c60 tx=0x7f1e50009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:57.801 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.801+0000 7f1e5a7fc700 1 -- 172.21.15.189:0/602997574 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1e480449c0 con 0x7f1e5c063e30 2020-11-20T12:04:57.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.949+0000 7f1e632a0700 1 -- 172.21.15.189:0/602997574 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1e5c0446d0 con 0x7f1e5c063e30 2020-11-20T12:04:57.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.949+0000 7f1e5a7fc700 1 -- 172.21.15.189:0/602997574 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+10 (secure 0 0 0) 0x7f1e48047fe0 con 0x7f1e5c063e30 2020-11-20T12:04:57.951 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 8 2020-11-20T12:04:57.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.949+0000 7f1e632a0700 1 -- 172.21.15.189:0/602997574 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1e3c0557c0 msgr2=0x7f1e3c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.949+0000 7f1e632a0700 1 --2- 172.21.15.189:0/602997574 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1e3c0557c0 0x7f1e3c057c80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f1e50001c60 tx=0x7f1e50009040).stop 2020-11-20T12:04:57.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.949+0000 7f1e632a0700 1 -- 172.21.15.189:0/602997574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e5c063e30 msgr2=0x7f1e5c0a0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:57.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.949+0000 7f1e632a0700 1 --2- 172.21.15.189:0/602997574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e5c063e30 0x7f1e5c0a0630 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f1e4800eba0 tx=0x7f1e48006df0).stop 2020-11-20T12:04:57.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.949+0000 7f1e632a0700 1 -- 172.21.15.189:0/602997574 shutdown_connections 2020-11-20T12:04:57.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.949+0000 7f1e632a0700 1 --2- 172.21.15.189:0/602997574 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1e3c0557c0 0x7f1e3c057c80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.949+0000 7f1e632a0700 1 --2- 172.21.15.189:0/602997574 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1e5c13b0e0 0x7f1e5c149590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.954 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.953+0000 7f1e632a0700 1 --2- 172.21.15.189:0/602997574 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1e5c063a50 0x7f1e5c09eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.954 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.953+0000 7f1e632a0700 1 --2- 172.21.15.189:0/602997574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e5c063e30 0x7f1e5c0a0630 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:57.954 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.953+0000 7f1e632a0700 1 -- 172.21.15.189:0/602997574 >> 172.21.15.189:0/602997574 conn(0x7f1e5c0ad8e0 msgr2=0x7f1e5c0aa280 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:57.954 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.953+0000 7f1e632a0700 1 -- 172.21.15.189:0/602997574 shutdown_connections 2020-11-20T12:04:57.955 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:57.953+0000 7f1e632a0700 1 -- 172.21.15.189:0/602997574 wait complete. 2020-11-20T12:04:57.970 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:04:58.097 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.097+0000 7f4c03d4e700 1 Processor -- start 2020-11-20T12:04:58.098 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.097+0000 7f4c03d4e700 1 -- start start 2020-11-20T12:04:58.100 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.097+0000 7f4c03d4e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4bfc0ae350 0x7f4bfc0640d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.100 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.097+0000 7f4c03d4e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4bfc0ae730 0x7f4bfc13a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.100 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.097+0000 7f4c03d4e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bfc134e50 con 0x7f4bfc0a70a0 2020-11-20T12:04:58.101 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.097+0000 7f4c03d4e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bfc0645c0 con 0x7f4bfc0ae350 2020-11-20T12:04:58.101 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.097+0000 7f4c03d4e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bfc0aa060 con 0x7f4bfc0ae730 2020-11-20T12:04:58.101 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.097+0000 7f4c01aea700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4bfc0ae350 0x7f4bfc0640d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.101 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.097+0000 7f4c022eb700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4bfc0a70a0 0x7f4bfc137400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50002/0 (socket says 172.21.15.189:50002) 2020-11-20T12:04:58.102 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.097+0000 7f4c022eb700 1 -- 172.21.15.189:0/2574058978 learned_addr learned my addr 172.21.15.189:0/2574058978 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:04:58.102 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.097+0000 7f4c012e9700 1 --2- 172.21.15.189:0/2574058978 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4bfc0ae730 0x7f4bfc13a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.102 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.097+0000 7f4c00ae8700 1 -- 172.21.15.189:0/2574058978 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 543548418 0 0) 0x7f4bfc134e50 con 0x7f4bfc0a70a0 2020-11-20T12:04:58.102 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.097+0000 7f4c00ae8700 1 -- 172.21.15.189:0/2574058978 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4be4003240 con 0x7f4bfc0a70a0 2020-11-20T12:04:58.102 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.097+0000 7f4c01aea700 1 -- 172.21.15.189:0/2574058978 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4bfc0ae730 msgr2=0x7f4bfc13a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:58.103 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.097+0000 7f4c01aea700 1 --2- 172.21.15.189:0/2574058978 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4bfc0ae730 0x7f4bfc13a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.103 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.097+0000 7f4c01aea700 1 -- 172.21.15.189:0/2574058978 >> v1:172.21.15.189:6789/0 conn(0x7f4bfc0a70a0 legacy=0x7f4bfc137400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.103 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.097+0000 7f4c01aea700 1 -- 172.21.15.189:0/2574058978 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bfc0a85e0 con 0x7f4bfc0ae350 2020-11-20T12:04:58.103 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.097+0000 7f4c01aea700 1 --2- 172.21.15.189:0/2574058978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bfc0ae350 0x7f4bfc0640d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f4bec00b9a0 tx=0x7f4bec006df0).ready entity=mon.1 client_cookie=dbb186c8594d014c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:58.104 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.097+0000 7f4c00ae8700 1 -- 172.21.15.189:0/2574058978 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bec009710 con 0x7f4bfc0ae350 2020-11-20T12:04:58.104 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.097+0000 7f4c00ae8700 1 -- 172.21.15.189:0/2574058978 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bec009870 con 0x7f4bfc0ae350 2020-11-20T12:04:58.104 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.097+0000 7f4c00ae8700 1 -- 172.21.15.189:0/2574058978 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bec009bf0 con 0x7f4bfc0ae350 2020-11-20T12:04:58.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.101+0000 7f4c03d4e700 1 -- 172.21.15.189:0/2574058978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bfc0ae350 msgr2=0x7f4bfc0640d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.101+0000 7f4c03d4e700 1 --2- 172.21.15.189:0/2574058978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bfc0ae350 0x7f4bfc0640d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f4bec00b9a0 tx=0x7f4bec006df0).stop 2020-11-20T12:04:58.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.101+0000 7f4c03d4e700 1 -- 172.21.15.189:0/2574058978 shutdown_connections 2020-11-20T12:04:58.109 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.101+0000 7f4c03d4e700 1 --2- 172.21.15.189:0/2574058978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bfc0ae350 0x7f4bfc0640d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.109 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.101+0000 7f4c03d4e700 1 --2- 172.21.15.189:0/2574058978 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4bfc0ae730 0x7f4bfc13a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.109 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.101+0000 7f4c03d4e700 1 -- 172.21.15.189:0/2574058978 >> 172.21.15.189:0/2574058978 conn(0x7f4bfc062580 msgr2=0x7f4bfc0adc40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:58.109 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.101+0000 7f4c03d4e700 1 -- 172.21.15.189:0/2574058978 shutdown_connections 2020-11-20T12:04:58.110 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.101+0000 7f4c03d4e700 1 -- 172.21.15.189:0/2574058978 wait complete. 2020-11-20T12:04:58.110 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.101+0000 7f4c03d4e700 1 Processor -- start 2020-11-20T12:04:58.110 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.101+0000 7f4c03d4e700 1 -- start start 2020-11-20T12:04:58.110 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.105+0000 7f4c03d4e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bfc0a70a0 0x7f4bfc139710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.105+0000 7f4c03d4e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4bfc0ae350 0x7f4bfc12fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.105+0000 7f4c03d4e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4bfc0ae730 0x7f4bfc130360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.105+0000 7f4c03d4e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bfc1313d0 con 0x7f4bfc0ae730 2020-11-20T12:04:58.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.105+0000 7f4c03d4e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bfc131670 con 0x7f4bfc0a70a0 2020-11-20T12:04:58.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.105+0000 7f4c03d4e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bfc1310e0 con 0x7f4bfc0ae350 2020-11-20T12:04:58.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.105+0000 7f4c01aea700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bfc0a70a0 0x7f4bfc139710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.105+0000 7f4c01aea700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bfc0a70a0 0x7f4bfc139710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48386/0 (socket says 172.21.15.189:48386) 2020-11-20T12:04:58.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.105+0000 7f4c01aea700 1 -- 172.21.15.189:0/282331159 learned_addr learned my addr 172.21.15.189:0/282331159 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:58.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.105+0000 7f4c012e9700 1 --2- 172.21.15.189:0/282331159 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4bfc0ae350 0x7f4bfc12fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.105+0000 7f4c022eb700 1 --2- 172.21.15.189:0/282331159 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4bfc0ae730 0x7f4bfc130360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.114 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.105+0000 7f4c01aea700 1 -- 172.21.15.189:0/282331159 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4bfc0ae350 msgr2=0x7f4bfc12fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.114 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.105+0000 7f4c01aea700 1 --2- 172.21.15.189:0/282331159 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4bfc0ae350 0x7f4bfc12fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.114 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.105+0000 7f4c01aea700 1 -- 172.21.15.189:0/282331159 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4bfc0ae730 msgr2=0x7f4bfc130360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.114 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.105+0000 7f4c01aea700 1 --2- 172.21.15.189:0/282331159 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4bfc0ae730 0x7f4bfc130360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.114 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.105+0000 7f4c01aea700 1 -- 172.21.15.189:0/282331159 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bfc1379c0 con 0x7f4bfc0a70a0 2020-11-20T12:04:58.115 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.105+0000 7f4c012e9700 1 --2- 172.21.15.189:0/282331159 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4bfc0ae350 0x7f4bfc12fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:58.115 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.105+0000 7f4c022eb700 1 --2- 172.21.15.189:0/282331159 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4bfc0ae730 0x7f4bfc130360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:58.115 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.105+0000 7f4c01aea700 1 --2- 172.21.15.189:0/282331159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bfc0a70a0 0x7f4bfc139710 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f4bec009410 tx=0x7f4bec015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:58.115 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.105+0000 7f4be9ffb700 1 -- 172.21.15.189:0/282331159 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bec011ea0 con 0x7f4bfc0a70a0 2020-11-20T12:04:58.115 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.105+0000 7f4be9ffb700 1 -- 172.21.15.189:0/282331159 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bec01d450 con 0x7f4bfc0a70a0 2020-11-20T12:04:58.116 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.105+0000 7f4be9ffb700 1 -- 172.21.15.189:0/282331159 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bec01d7f0 con 0x7f4bfc0a70a0 2020-11-20T12:04:58.116 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.105+0000 7f4c03d4e700 1 -- 172.21.15.189:0/282331159 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bfc133410 con 0x7f4bfc0a70a0 2020-11-20T12:04:58.116 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.105+0000 7f4c03d4e700 1 -- 172.21.15.189:0/282331159 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bfc0a4af0 con 0x7f4bfc0a70a0 2020-11-20T12:04:58.116 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.105+0000 7f4be9ffb700 1 -- 172.21.15.189:0/282331159 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f4bec00f070 con 0x7f4bfc0a70a0 2020-11-20T12:04:58.116 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.105+0000 7f4c03d4e700 1 -- 172.21.15.189:0/282331159 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bfc137b50 con 0x7f4bfc0a70a0 2020-11-20T12:04:58.117 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.109+0000 7f4be9ffb700 1 --2- 172.21.15.189:0/282331159 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4bd40557c0 0x7f4bd4057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.117 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.109+0000 7f4be9ffb700 1 -- 172.21.15.189:0/282331159 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4bec06fa80 con 0x7f4bfc0a70a0 2020-11-20T12:04:58.117 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.109+0000 7f4c012e9700 1 --2- 172.21.15.189:0/282331159 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4bd40557c0 0x7f4bd4057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.118 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.109+0000 7f4c012e9700 1 --2- 172.21.15.189:0/282331159 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4bd40557c0 0x7f4bd4057c80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f4bfc12dc40 tx=0x7f4bf8008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:58.118 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.113+0000 7f4be9ffb700 1 -- 172.21.15.189:0/282331159 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4bec045920 con 0x7f4bfc0a70a0 2020-11-20T12:04:58.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.265+0000 7f4c03d4e700 1 -- 172.21.15.189:0/282331159 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4bfc044f60 con 0x7f4bfc0a70a0 2020-11-20T12:04:58.269 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.269+0000 7f4be9ffb700 1 -- 172.21.15.189:0/282331159 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+10 (secure 0 0 0) 0x7f4bec022070 con 0x7f4bfc0a70a0 2020-11-20T12:04:58.269 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 8 2020-11-20T12:04:58.271 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.269+0000 7f4c03d4e700 1 -- 172.21.15.189:0/282331159 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4bd40557c0 msgr2=0x7f4bd4057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.272 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.269+0000 7f4c03d4e700 1 --2- 172.21.15.189:0/282331159 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4bd40557c0 0x7f4bd4057c80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f4bfc12dc40 tx=0x7f4bf8008040).stop 2020-11-20T12:04:58.272 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.269+0000 7f4c03d4e700 1 -- 172.21.15.189:0/282331159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bfc0a70a0 msgr2=0x7f4bfc139710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.272 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.269+0000 7f4c03d4e700 1 --2- 172.21.15.189:0/282331159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bfc0a70a0 0x7f4bfc139710 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f4bec009410 tx=0x7f4bec015000).stop 2020-11-20T12:04:58.272 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.269+0000 7f4c03d4e700 1 -- 172.21.15.189:0/282331159 shutdown_connections 2020-11-20T12:04:58.272 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.269+0000 7f4c03d4e700 1 --2- 172.21.15.189:0/282331159 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4bd40557c0 0x7f4bd4057c80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.273 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.269+0000 7f4c03d4e700 1 --2- 172.21.15.189:0/282331159 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4bfc0ae730 0x7f4bfc130360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.273 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.269+0000 7f4c03d4e700 1 --2- 172.21.15.189:0/282331159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bfc0a70a0 0x7f4bfc139710 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.273 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.269+0000 7f4c03d4e700 1 --2- 172.21.15.189:0/282331159 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4bfc0ae350 0x7f4bfc12fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.273 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.269+0000 7f4c03d4e700 1 -- 172.21.15.189:0/282331159 >> 172.21.15.189:0/282331159 conn(0x7f4bfc062580 msgr2=0x7f4bfc0a6980 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:58.273 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.269+0000 7f4c03d4e700 1 -- 172.21.15.189:0/282331159 shutdown_connections 2020-11-20T12:04:58.274 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.269+0000 7f4c03d4e700 1 -- 172.21.15.189:0/282331159 wait complete. 2020-11-20T12:04:58.288 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2020-11-20T12:04:58.288 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2020-11-20T12:04:58.289 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2020-11-20T12:04:58.442 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.441+0000 7f65568a0700 1 Processor -- start 2020-11-20T12:04:58.442 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.441+0000 7f65568a0700 1 -- start start 2020-11-20T12:04:58.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.441+0000 7f65568a0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f65500aeb80 0x7f655013a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.441+0000 7f65568a0700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f65500aef60 0x7f655013e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.441+0000 7f65568a0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f655013dcb0 con 0x7f655013ade0 2020-11-20T12:04:58.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.441+0000 7f65568a0700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f655013eb40 con 0x7f65500aeb80 2020-11-20T12:04:58.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.441+0000 7f65568a0700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f655013df10 con 0x7f65500aef60 2020-11-20T12:04:58.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.441+0000 7f655589e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f65500aeb80 0x7f655013a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.441+0000 7f655609f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f655013ade0 0x7f65500a1960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50016/0 (socket says 172.21.15.189:50016) 2020-11-20T12:04:58.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.441+0000 7f655609f700 1 -- 172.21.15.189:0/1789940859 learned_addr learned my addr 172.21.15.189:0/1789940859 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:04:58.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.441+0000 7f655509d700 1 --2- 172.21.15.189:0/1789940859 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f65500aef60 0x7f655013e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.445+0000 7f655589e700 1 -- 172.21.15.189:0/1789940859 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f65500aef60 msgr2=0x7f655013e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:58.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.441+0000 7f655489c700 1 -- 172.21.15.189:0/1789940859 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803235516 0 0) 0x7f655013dcb0 con 0x7f655013ade0 2020-11-20T12:04:58.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.445+0000 7f655589e700 1 --2- 172.21.15.189:0/1789940859 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f65500aef60 0x7f655013e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.445+0000 7f655589e700 1 -- 172.21.15.189:0/1789940859 >> v1:172.21.15.189:6789/0 conn(0x7f655013ade0 legacy=0x7f65500a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.445+0000 7f655589e700 1 -- 172.21.15.189:0/1789940859 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6550137fa0 con 0x7f65500aeb80 2020-11-20T12:04:58.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.445+0000 7f655589e700 1 --2- 172.21.15.189:0/1789940859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65500aeb80 0x7f655013a640 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f654c00b9a0 tx=0x7f654c006df0).ready entity=mon.1 client_cookie=fa599ad3b8251037 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:58.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.445+0000 7f655489c700 1 -- 172.21.15.189:0/1789940859 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654c0097b0 con 0x7f65500aeb80 2020-11-20T12:04:58.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.445+0000 7f655489c700 1 -- 172.21.15.189:0/1789940859 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f654c009910 con 0x7f65500aeb80 2020-11-20T12:04:58.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.445+0000 7f655489c700 1 -- 172.21.15.189:0/1789940859 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654c009c90 con 0x7f65500aeb80 2020-11-20T12:04:58.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.445+0000 7f65568a0700 1 -- 172.21.15.189:0/1789940859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65500aeb80 msgr2=0x7f655013a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.445+0000 7f65568a0700 1 --2- 172.21.15.189:0/1789940859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65500aeb80 0x7f655013a640 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f654c00b9a0 tx=0x7f654c006df0).stop 2020-11-20T12:04:58.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.445+0000 7f65568a0700 1 -- 172.21.15.189:0/1789940859 shutdown_connections 2020-11-20T12:04:58.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.445+0000 7f65568a0700 1 --2- 172.21.15.189:0/1789940859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65500aeb80 0x7f655013a640 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.445+0000 7f65568a0700 1 --2- 172.21.15.189:0/1789940859 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f65500aef60 0x7f655013e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.445+0000 7f65568a0700 1 -- 172.21.15.189:0/1789940859 >> 172.21.15.189:0/1789940859 conn(0x7f6550062080 msgr2=0x7f6550062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:58.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.445+0000 7f65568a0700 1 -- 172.21.15.189:0/1789940859 shutdown_connections 2020-11-20T12:04:58.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.445+0000 7f65568a0700 1 -- 172.21.15.189:0/1789940859 wait complete. 2020-11-20T12:04:58.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.445+0000 7f65568a0700 1 Processor -- start 2020-11-20T12:04:58.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.445+0000 7f65568a0700 1 -- start start 2020-11-20T12:04:58.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f65568a0700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f65500aeb80 0x7f6550139670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f65568a0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65500aef60 0x7f65500a4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f65568a0700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f655013ade0 0x7f6550064380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f65568a0700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65500a4610 con 0x7f655013ade0 2020-11-20T12:04:58.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f65568a0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65500a5990 con 0x7f65500aef60 2020-11-20T12:04:58.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f65568a0700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65500a2dd0 con 0x7f65500aeb80 2020-11-20T12:04:58.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f655509d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65500aef60 0x7f65500a4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f655509d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65500aef60 0x7f65500a4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48400/0 (socket says 172.21.15.189:48400) 2020-11-20T12:04:58.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f655509d700 1 -- 172.21.15.189:0/4168518330 learned_addr learned my addr 172.21.15.189:0/4168518330 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:58.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f655589e700 1 --2- 172.21.15.189:0/4168518330 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f65500aeb80 0x7f6550139670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f655609f700 1 --2- 172.21.15.189:0/4168518330 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f655013ade0 0x7f6550064380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f655509d700 1 -- 172.21.15.189:0/4168518330 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f65500aeb80 msgr2=0x7f6550139670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f655509d700 1 --2- 172.21.15.189:0/4168518330 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f65500aeb80 0x7f6550139670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f655509d700 1 -- 172.21.15.189:0/4168518330 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f655013ade0 msgr2=0x7f6550064380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f655509d700 1 --2- 172.21.15.189:0/4168518330 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f655013ade0 0x7f6550064380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f655509d700 1 -- 172.21.15.189:0/4168518330 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65500a3070 con 0x7f65500aef60 2020-11-20T12:04:58.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f655509d700 1 --2- 172.21.15.189:0/4168518330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65500aef60 0x7f65500a4a00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f654000eba0 tx=0x7f6540006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:58.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f653dffb700 1 -- 172.21.15.189:0/4168518330 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654000ca60 con 0x7f65500aef60 2020-11-20T12:04:58.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f653dffb700 1 -- 172.21.15.189:0/4168518330 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f654000cbc0 con 0x7f65500aef60 2020-11-20T12:04:58.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f65568a0700 1 -- 172.21.15.189:0/4168518330 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65500a2a50 con 0x7f65500aef60 2020-11-20T12:04:58.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f653dffb700 1 -- 172.21.15.189:0/4168518330 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65400123f0 con 0x7f65500aef60 2020-11-20T12:04:58.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f65568a0700 1 -- 172.21.15.189:0/4168518330 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6550064d90 con 0x7f65500aef60 2020-11-20T12:04:58.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f653dffb700 1 -- 172.21.15.189:0/4168518330 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f65400127a0 con 0x7f65500aef60 2020-11-20T12:04:58.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f65568a0700 1 -- 172.21.15.189:0/4168518330 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65500a2c40 con 0x7f65500aef60 2020-11-20T12:04:58.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f653dffb700 1 --2- 172.21.15.189:0/4168518330 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f65280557c0 0x7f6528057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f653dffb700 1 -- 172.21.15.189:0/4168518330 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f654006ebd0 con 0x7f65500aef60 2020-11-20T12:04:58.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.449+0000 7f655589e700 1 --2- 172.21.15.189:0/4168518330 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f65280557c0 0x7f6528057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.453+0000 7f655589e700 1 --2- 172.21.15.189:0/4168518330 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f65280557c0 0x7f6528057c80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f654c006e30 tx=0x7f654c015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:58.459 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.457+0000 7f653dffb700 1 -- 172.21.15.189:0/4168518330 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f65400449f0 con 0x7f65500aef60 2020-11-20T12:04:58.643 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.641+0000 7f65568a0700 1 -- 172.21.15.189:0/4168518330 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f65500446d0 con 0x7f65500aef60 2020-11-20T12:04:58.648 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.649+0000 7f653dffb700 1 -- 172.21.15.189:0/4168518330 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) v1 ==== 74+0+133248 (secure 0 0 0) 0x7f6540048010 con 0x7f65500aef60 2020-11-20T12:04:58.649 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:58.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.649+0000 7f65568a0700 1 -- 172.21.15.189:0/4168518330 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f65280557c0 msgr2=0x7f6528057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.649+0000 7f65568a0700 1 --2- 172.21.15.189:0/4168518330 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f65280557c0 0x7f6528057c80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f654c006e30 tx=0x7f654c015000).stop 2020-11-20T12:04:58.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.649+0000 7f65568a0700 1 -- 172.21.15.189:0/4168518330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65500aef60 msgr2=0x7f65500a4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.649+0000 7f65568a0700 1 --2- 172.21.15.189:0/4168518330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65500aef60 0x7f65500a4a00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f654000eba0 tx=0x7f6540006df0).stop 2020-11-20T12:04:58.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.649+0000 7f65568a0700 1 -- 172.21.15.189:0/4168518330 shutdown_connections 2020-11-20T12:04:58.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.649+0000 7f65568a0700 1 --2- 172.21.15.189:0/4168518330 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f65280557c0 0x7f6528057c80 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.649+0000 7f65568a0700 1 --2- 172.21.15.189:0/4168518330 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f655013ade0 0x7f6550064380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.649+0000 7f65568a0700 1 --2- 172.21.15.189:0/4168518330 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f65500aeb80 0x7f6550139670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.649+0000 7f65568a0700 1 --2- 172.21.15.189:0/4168518330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65500aef60 0x7f65500a4a00 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.649+0000 7f65568a0700 1 -- 172.21.15.189:0/4168518330 >> 172.21.15.189:0/4168518330 conn(0x7f6550062080 msgr2=0x7f65500a92e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:58.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.649+0000 7f65568a0700 1 -- 172.21.15.189:0/4168518330 shutdown_connections 2020-11-20T12:04:58.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.649+0000 7f65568a0700 1 -- 172.21.15.189:0/4168518330 wait complete. 2020-11-20T12:04:58.665 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":4,"active_gid":4105,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":11671},{"type":"v1","addr":"172.21.15.189:6833","nonce":11671}]},"active_addr":"172.21.15.189:6833/11671","active_change":"2020-11-20T12:04:41.319722+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[{"gid":4104,"name":"x","mgr_features":4540138297136906239,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":2518100196}]},{"addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":1260947910}]},{"addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":255982973}]}]}} 2020-11-20T12:04:58.669 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2020-11-20T12:04:58.669 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2020-11-20T12:04:58.669 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:04:58.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.785+0000 7fbae0608700 1 Processor -- start 2020-11-20T12:04:58.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.785+0000 7fbae0608700 1 -- start start 2020-11-20T12:04:58.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.785+0000 7fbae0608700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbad80678c0 0x7fbad80a8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.785+0000 7fbae0608700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad80aeb40 con 0x7fbad8067ca0 2020-11-20T12:04:58.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.785+0000 7fbae0608700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad80ae8a0 con 0x7fbad80ae390 2020-11-20T12:04:58.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.785+0000 7fbae0608700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad8137690 con 0x7fbad80678c0 2020-11-20T12:04:58.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.785+0000 7fbadeba5700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbad80ae390 0x7fbad8137b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50032/0 (socket says 172.21.15.189:50032) 2020-11-20T12:04:58.796 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.785+0000 7fbadeba5700 1 -- 172.21.15.189:0/3363791807 learned_addr learned my addr 172.21.15.189:0/3363791807 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:04:58.796 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.785+0000 7fbade3a4700 1 --2- 172.21.15.189:0/3363791807 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbad80678c0 0x7fbad80a8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.796 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.785+0000 7fbadd3a2700 1 -- 172.21.15.189:0/3363791807 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723099123 0 0) 0x7fbad80ae8a0 con 0x7fbad80ae390 2020-11-20T12:04:58.796 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.785+0000 7fbadd3a2700 1 -- 172.21.15.189:0/3363791807 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbac0003240 con 0x7fbad80ae390 2020-11-20T12:04:58.796 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbadd3a2700 1 -- 172.21.15.189:0/3363791807 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1237633218 0 0) 0x7fbac0003240 con 0x7fbad80ae390 2020-11-20T12:04:58.797 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbadd3a2700 1 -- 172.21.15.189:0/3363791807 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbad80678c0 msgr2=0x7fbad80a8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:58.797 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbadd3a2700 1 --2- 172.21.15.189:0/3363791807 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbad80678c0 0x7fbad80a8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.797 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbadd3a2700 1 -- 172.21.15.189:0/3363791807 >> v1:172.21.15.47:6789/0 conn(0x7fbad8067ca0 legacy=0x7fbad813b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.797 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbadd3a2700 1 -- 172.21.15.189:0/3363791807 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad80a9170 con 0x7fbad80ae390 2020-11-20T12:04:58.797 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbadd3a2700 1 -- 172.21.15.189:0/3363791807 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7fbad0001860 con 0x7fbad80ae390 2020-11-20T12:04:58.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbadd3a2700 1 -- 172.21.15.189:0/3363791807 >> v1:172.21.15.189:6789/0 conn(0x7fbad80ae390 legacy=0x7fbad8137b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbadd3a2700 1 --2- 172.21.15.189:0/3363791807 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbad80678c0 0x7fbac0003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbadd3a2700 1 --2- 172.21.15.189:0/3363791807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbac0003c20 0x7fbac00044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbadd3a2700 1 --2- 172.21.15.189:0/3363791807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbac00049d0 0x7fbac0004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbadd3a2700 1 -- 172.21.15.189:0/3363791807 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad8137690 con 0x7fbac00049d0 2020-11-20T12:04:58.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbadd3a2700 1 -- 172.21.15.189:0/3363791807 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac0004140 con 0x7fbac0003c20 2020-11-20T12:04:58.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbadd3a2700 1 -- 172.21.15.189:0/3363791807 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac00042b0 con 0x7fbad80678c0 2020-11-20T12:04:58.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbaddba3700 1 --2- 172.21.15.189:0/3363791807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbac0003c20 0x7fbac00044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbade3a4700 1 --2- 172.21.15.189:0/3363791807 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbad80678c0 0x7fbac0003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbadeba5700 1 --2- 172.21.15.189:0/3363791807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbac00049d0 0x7fbac0004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbaddba3700 1 -- 172.21.15.189:0/3363791807 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbad80678c0 msgr2=0x7fbac0003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:58.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbaddba3700 1 --2- 172.21.15.189:0/3363791807 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbad80678c0 0x7fbac0003730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.801 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbaddba3700 1 -- 172.21.15.189:0/3363791807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbac00049d0 msgr2=0x7fbac0004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:58.801 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbaddba3700 1 --2- 172.21.15.189:0/3363791807 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbac00049d0 0x7fbac0004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.801 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbaddba3700 1 -- 172.21.15.189:0/3363791807 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac0003240 con 0x7fbac0003c20 2020-11-20T12:04:58.801 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbaddba3700 1 --2- 172.21.15.189:0/3363791807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbac0003c20 0x7fbac00044e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fbad8065a90 tx=0x7fbac8007b90).ready entity=mon.1 client_cookie=238d345b089c5b12 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:58.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbadd3a2700 1 -- 172.21.15.189:0/3363791807 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac8007020 con 0x7fbac0003c20 2020-11-20T12:04:58.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbadd3a2700 1 -- 172.21.15.189:0/3363791807 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac8007180 con 0x7fbac0003c20 2020-11-20T12:04:58.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbadd3a2700 1 -- 172.21.15.189:0/3363791807 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac800c980 con 0x7fbac0003c20 2020-11-20T12:04:58.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbae0608700 1 -- 172.21.15.189:0/3363791807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbac0003c20 msgr2=0x7fbac00044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbae0608700 1 --2- 172.21.15.189:0/3363791807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbac0003c20 0x7fbac00044e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fbad8065a90 tx=0x7fbac8007b90).stop 2020-11-20T12:04:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbadeba5700 1 -- 172.21.15.189:0/3363791807 reap_dead start 2020-11-20T12:04:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbae0608700 1 -- 172.21.15.189:0/3363791807 shutdown_connections 2020-11-20T12:04:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbae0608700 1 -- 172.21.15.189:0/3363791807 >> 172.21.15.189:0/3363791807 conn(0x7fbad80625a0 msgr2=0x7fbad80adc80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbae0608700 1 -- 172.21.15.189:0/3363791807 shutdown_connections 2020-11-20T12:04:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbae0608700 1 -- 172.21.15.189:0/3363791807 wait complete. 2020-11-20T12:04:58.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.789+0000 7fbae0608700 1 Processor -- start 2020-11-20T12:04:58.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.793+0000 7fbae0608700 1 -- start start 2020-11-20T12:04:58.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.793+0000 7fbae0608700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbad80a0a90 0x7fbad80a72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.793+0000 7fbae0608700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbad80a5c20 0x7fbad8139d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.793+0000 7fbae0608700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbad8138870 0x7fbad8138cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.793+0000 7fbae0608700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad8066cb0 con 0x7fbad80a5c20 2020-11-20T12:04:58.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.793+0000 7fbae0608700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad80a7b70 con 0x7fbad80a0a90 2020-11-20T12:04:58.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.793+0000 7fbae0608700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad813a220 con 0x7fbad8138870 2020-11-20T12:04:58.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.793+0000 7fbade3a4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbad80a0a90 0x7fbad80a72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.806 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.793+0000 7fbade3a4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbad80a0a90 0x7fbad80a72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48418/0 (socket says 172.21.15.189:48418) 2020-11-20T12:04:58.806 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.793+0000 7fbade3a4700 1 -- 172.21.15.189:0/1795319468 learned_addr learned my addr 172.21.15.189:0/1795319468 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:58.806 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.793+0000 7fbaddba3700 1 --2- 172.21.15.189:0/1795319468 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbad80a5c20 0x7fbad8139d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.806 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.793+0000 7fbadeba5700 1 --2- 172.21.15.189:0/1795319468 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbad8138870 0x7fbad8138cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.807 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.793+0000 7fbade3a4700 1 -- 172.21.15.189:0/1795319468 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbad8138870 msgr2=0x7fbad8138cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.807 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.793+0000 7fbade3a4700 1 --2- 172.21.15.189:0/1795319468 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbad8138870 0x7fbad8138cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.807 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.793+0000 7fbade3a4700 1 -- 172.21.15.189:0/1795319468 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbad80a5c20 msgr2=0x7fbad8139d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.807 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.793+0000 7fbade3a4700 1 --2- 172.21.15.189:0/1795319468 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbad80a5c20 0x7fbad8139d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.807 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.793+0000 7fbade3a4700 1 -- 172.21.15.189:0/1795319468 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad80a78d0 con 0x7fbad80a0a90 2020-11-20T12:04:58.808 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.793+0000 7fbade3a4700 1 --2- 172.21.15.189:0/1795319468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbad80a0a90 0x7fbad80a72e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fbad40067d0 tx=0x7fbad4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:58.808 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.793+0000 7fbac67fc700 1 -- 172.21.15.189:0/1795319468 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad400e8a0 con 0x7fbad80a0a90 2020-11-20T12:04:58.808 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.793+0000 7fbac67fc700 1 -- 172.21.15.189:0/1795319468 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad400ea00 con 0x7fbad80a0a90 2020-11-20T12:04:58.809 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.793+0000 7fbae0608700 1 -- 172.21.15.189:0/1795319468 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad80a5780 con 0x7fbad80a0a90 2020-11-20T12:04:58.809 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.793+0000 7fbac67fc700 1 -- 172.21.15.189:0/1795319468 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad400ed80 con 0x7fbad80a0a90 2020-11-20T12:04:58.809 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.793+0000 7fbae0608700 1 -- 172.21.15.189:0/1795319468 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad80a1250 con 0x7fbad80a0a90 2020-11-20T12:04:58.812 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.797+0000 7fbae0608700 1 -- 172.21.15.189:0/1795319468 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbad8044f60 con 0x7fbad80a0a90 2020-11-20T12:04:58.812 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.797+0000 7fbac67fc700 1 -- 172.21.15.189:0/1795319468 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fbad4024070 con 0x7fbad80a0a90 2020-11-20T12:04:58.812 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.797+0000 7fbac67fc700 1 --2- 172.21.15.189:0/1795319468 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fbab00557c0 0x7fbab0057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:58.813 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.797+0000 7fbac67fc700 1 -- 172.21.15.189:0/1795319468 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbad406eff0 con 0x7fbad80a0a90 2020-11-20T12:04:58.813 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.797+0000 7fbaddba3700 1 --2- 172.21.15.189:0/1795319468 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fbab00557c0 0x7fbab0057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:58.814 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.797+0000 7fbaddba3700 1 --2- 172.21.15.189:0/1795319468 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fbab00557c0 0x7fbab0057c80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fbac8013000 tx=0x7fbac8016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:58.814 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.805+0000 7fbac67fc700 1 -- 172.21.15.189:0/1795319468 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbad4044da0 con 0x7fbad80a0a90 2020-11-20T12:04:58.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.949+0000 7fbae0608700 1 -- 172.21.15.189:0/1795319468 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbad8138240 con 0x7fbad80a0a90 2020-11-20T12:04:58.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.953+0000 7fbac67fc700 1 -- 172.21.15.189:0/1795319468 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16826 (secure 0 0 0) 0x7fbad4018020 con 0x7fbad80a0a90 2020-11-20T12:04:58.955 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:58.955 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":22,"fsid":"cda18d0e-64b3-41bb-964a-084069c3e648","created":"2020-11-20T12:04:37.538706+0000","modified":"2020-11-20T12:04:55.015818+0000","last_up_change":"2020-11-20T12:04:43.006631+0000","last_in_change":"2020-11-20T12:04:43.006631+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:41.395395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:48.550454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:52.101837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:53.018196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c81214e1-4f84-4aaa-8ceb-99c398b2a89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6811","nonce":11760},{"type":"v1","addr":"172.21.15.189:6814","nonce":11760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":11760},{"type":"v1","addr":"172.21.15.189:6818","nonce":11760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":11760},{"type":"v1","addr":"172.21.15.189:6828","nonce":11760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6821","nonce":11760},{"type":"v1","addr":"172.21.15.189:6824","nonce":11760}]},"public_addr":"172.21.15.189:6814/11760","cluster_addr":"172.21.15.189:6818/11760","heartbeat_back_addr":"172.21.15.189:6828/11760","heartbeat_front_addr":"172.21.15.189:6824/11760","state":["exists","up"]},{"osd":1,"uuid":"d2f3ec7d-d403-484f-b5ed-b2e6b079d63d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6805","nonce":11761},{"type":"v1","addr":"172.21.15.189:6807","nonce":11761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":11761},{"type":"v1","addr":"172.21.15.189:6812","nonce":11761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6817","nonce":11761},{"type":"v1","addr":"172.21.15.189:6820","nonce":11761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6813","nonce":11761},{"type":"v1","addr":"172.21.15.189:6815","nonce":11761}]},"public_addr":"172.21.15.189:6807/11761","cluster_addr":"172.21.15.189:6812/11761","heartbeat_back_addr":"172.21.15.189:6820/11761","heartbeat_front_addr":"172.21.15.189:6815/11761","state":["exists","up"]},{"osd":2,"uuid":"e6cf4790-000c-43eb-8844-175f04c60503","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6819","nonce":11751},{"type":"v1","addr":"172.21.15.189:6822","nonce":11751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":11751},{"type":"v1","addr":"172.21.15.189:6825","nonce":11751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":11751},{"type":"v1","addr":"172.21.15.189:6831","nonce":11751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6827","nonce":11751},{"type":"v1","addr":"172.21.15.189:6829","nonce":11751}]},"public_addr":"172.21.15.189:6822/11751","cluster_addr":"172.21.15.189:6825/11751","heartbeat_back_addr":"172.21.15.189:6831/11751","heartbeat_front_addr":"172.21.15.189:6829/11751","state":["exists","up"]},{"osd":3,"uuid":"ba49dfac-f2b9-476e-853d-42f05e21a77a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":11762},{"type":"v1","addr":"172.21.15.189:6801","nonce":11762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":11762},{"type":"v1","addr":"172.21.15.189:6803","nonce":11762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":11762},{"type":"v1","addr":"172.21.15.189:6809","nonce":11762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":11762},{"type":"v1","addr":"172.21.15.189:6806","nonce":11762}]},"public_addr":"172.21.15.189:6801/11762","cluster_addr":"172.21.15.189:6803/11762","heartbeat_back_addr":"172.21.15.189:6809/11762","heartbeat_front_addr":"172.21.15.189:6806/11762","state":["exists","up"]},{"osd":4,"uuid":"3ca7b438-38b2-4689-8b18-fb5a8ddf6aee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":12114},{"type":"v1","addr":"172.21.15.47:6825","nonce":12114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":12114},{"type":"v1","addr":"172.21.15.47:6827","nonce":12114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":12114},{"type":"v1","addr":"172.21.15.47:6831","nonce":12114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":12114},{"type":"v1","addr":"172.21.15.47:6829","nonce":12114}]},"public_addr":"172.21.15.47:6825/12114","cluster_addr":"172.21.15.47:6827/12114","heartbeat_back_addr":"172.21.15.47:6831/12114","heartbeat_front_addr":"172.21.15.47:6829/12114","state":["exists","up"]},{"osd":5,"uuid":"6a45fca7-83d3-44fd-a1d4-b01e4201a101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":12111},{"type":"v1","addr":"172.21.15.47:6801","nonce":12111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":12111},{"type":"v1","addr":"172.21.15.47:6803","nonce":12111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":12111},{"type":"v1","addr":"172.21.15.47:6807","nonce":12111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":12111},{"type":"v1","addr":"172.21.15.47:6805","nonce":12111}]},"public_addr":"172.21.15.47:6801/12111","cluster_addr":"172.21.15.47:6803/12111","heartbeat_back_addr":"172.21.15.47:6807/12111","heartbeat_front_addr":"172.21.15.47:6805/12111","state":["exists","up"]},{"osd":6,"uuid":"c4b1b9ed-61f8-4677-9755-dff59a30e447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":12113},{"type":"v1","addr":"172.21.15.47:6817","nonce":12113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":12113},{"type":"v1","addr":"172.21.15.47:6819","nonce":12113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":12113},{"type":"v1","addr":"172.21.15.47:6823","nonce":12113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":12113},{"type":"v1","addr":"172.21.15.47:6821","nonce":12113}]},"public_addr":"172.21.15.47:6817/12113","cluster_addr":"172.21.15.47:6819/12113","heartbeat_back_addr":"172.21.15.47:6823/12113","heartbeat_front_addr":"172.21.15.47:6821/12113","state":["exists","up"]},{"osd":7,"uuid":"4608e4ed-cc40-4c5c-ba8b-3779bbc62f65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":12112},{"type":"v1","addr":"172.21.15.47:6809","nonce":12112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":12112},{"type":"v1","addr":"172.21.15.47:6811","nonce":12112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":12112},{"type":"v1","addr":"172.21.15.47:6815","nonce":12112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":12112},{"type":"v1","addr":"172.21.15.47:6813","nonce":12112}]},"public_addr":"172.21.15.47:6809/12112","cluster_addr":"172.21.15.47:6811/12112","heartbeat_back_addr":"172.21.15.47:6815/12112","heartbeat_front_addr":"172.21.15.47:6813/12112","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.435265+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.472776+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.388370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.417366+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.102392+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.116529+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.167430+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.148139+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:04:58.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.953+0000 7fbae0608700 1 -- 172.21.15.189:0/1795319468 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fbab00557c0 msgr2=0x7fbab0057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.953+0000 7fbae0608700 1 --2- 172.21.15.189:0/1795319468 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fbab00557c0 0x7fbab0057c80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fbac8013000 tx=0x7fbac8016000).stop 2020-11-20T12:04:58.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.953+0000 7fbae0608700 1 -- 172.21.15.189:0/1795319468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbad80a0a90 msgr2=0x7fbad80a72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:58.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.953+0000 7fbae0608700 1 --2- 172.21.15.189:0/1795319468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbad80a0a90 0x7fbad80a72e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fbad40067d0 tx=0x7fbad4008040).stop 2020-11-20T12:04:58.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.953+0000 7fbae0608700 1 -- 172.21.15.189:0/1795319468 shutdown_connections 2020-11-20T12:04:58.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.953+0000 7fbae0608700 1 --2- 172.21.15.189:0/1795319468 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbad8138870 0x7fbad8138cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.953+0000 7fbae0608700 1 --2- 172.21.15.189:0/1795319468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbad80a0a90 0x7fbad80a72e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.953+0000 7fbae0608700 1 --2- 172.21.15.189:0/1795319468 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fbab00557c0 0x7fbab0057c80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.953+0000 7fbae0608700 1 --2- 172.21.15.189:0/1795319468 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbad80a5c20 0x7fbad8139d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:58.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.953+0000 7fbae0608700 1 -- 172.21.15.189:0/1795319468 >> 172.21.15.189:0/1795319468 conn(0x7fbad80625a0 msgr2=0x7fbad8137a50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:58.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.953+0000 7fbae0608700 1 -- 172.21.15.189:0/1795319468 shutdown_connections 2020-11-20T12:04:58.960 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:58.953+0000 7fbae0608700 1 -- 172.21.15.189:0/1795319468 wait complete. 2020-11-20T12:04:58.968 INFO:tasks.ceph.ceph_manager.ceph:all up! 2020-11-20T12:04:58.969 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:04:59.094 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.093+0000 7fdc64b92700 1 Processor -- start 2020-11-20T12:04:59.094 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.093+0000 7fdc64b92700 1 -- start start 2020-11-20T12:04:59.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.093+0000 7fdc64b92700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdc600ae770 0x7fdc60065b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.093+0000 7fdc64b92700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdc600aa330 0x7fdc600a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.093+0000 7fdc64b92700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc60067f50 con 0x7fdc600ae390 2020-11-20T12:04:59.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.093+0000 7fdc64b92700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc600673b0 con 0x7fdc600aa330 2020-11-20T12:04:59.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.093+0000 7fdc64b92700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc600a10f0 con 0x7fdc600ae770 2020-11-20T12:04:59.097 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.093+0000 7fdc5ed9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdc600aa330 0x7fdc600a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.097 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.093+0000 7fdc5e59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdc600ae390 0x7fdc60137af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50048/0 (socket says 172.21.15.189:50048) 2020-11-20T12:04:59.097 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.097+0000 7fdc5e59c700 1 -- 172.21.15.189:0/1691998562 learned_addr learned my addr 172.21.15.189:0/1691998562 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:04:59.097 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.097+0000 7fdc57fff700 1 --2- 172.21.15.189:0/1691998562 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdc600ae770 0x7fdc60065b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.099 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.097+0000 7fdc5ed9d700 1 -- 172.21.15.189:0/1691998562 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdc600ae770 msgr2=0x7fdc60065b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:59.100 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.097+0000 7fdc5ed9d700 1 --2- 172.21.15.189:0/1691998562 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdc600ae770 0x7fdc60065b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.100 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.097+0000 7fdc5ed9d700 1 -- 172.21.15.189:0/1691998562 >> v1:172.21.15.189:6789/0 conn(0x7fdc600ae390 legacy=0x7fdc60137af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.100 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.097+0000 7fdc5ed9d700 1 -- 172.21.15.189:0/1691998562 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc600a13b0 con 0x7fdc600aa330 2020-11-20T12:04:59.100 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.097+0000 7fdc5dd9b700 1 -- 172.21.15.189:0/1691998562 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562985781 0 0) 0x7fdc60067f50 con 0x7fdc600ae390 2020-11-20T12:04:59.101 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.097+0000 7fdc5ed9d700 1 --2- 172.21.15.189:0/1691998562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc600aa330 0x7fdc600a0c00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fdc58009c90 tx=0x7fdc580091d0).ready entity=mon.1 client_cookie=7ca707448101ef68 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:59.101 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.097+0000 7fdc5dd9b700 1 -- 172.21.15.189:0/1691998562 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5800bb00 con 0x7fdc600aa330 2020-11-20T12:04:59.101 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.097+0000 7fdc5dd9b700 1 -- 172.21.15.189:0/1691998562 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc5800bc60 con 0x7fdc600aa330 2020-11-20T12:04:59.101 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.097+0000 7fdc5dd9b700 1 -- 172.21.15.189:0/1691998562 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc58011400 con 0x7fdc600aa330 2020-11-20T12:04:59.102 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.097+0000 7fdc64b92700 1 -- 172.21.15.189:0/1691998562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc600aa330 msgr2=0x7fdc600a0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.102 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.097+0000 7fdc64b92700 1 --2- 172.21.15.189:0/1691998562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc600aa330 0x7fdc600a0c00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fdc58009c90 tx=0x7fdc580091d0).stop 2020-11-20T12:04:59.102 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.097+0000 7fdc64b92700 1 -- 172.21.15.189:0/1691998562 shutdown_connections 2020-11-20T12:04:59.102 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.097+0000 7fdc64b92700 1 --2- 172.21.15.189:0/1691998562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc600aa330 0x7fdc600a0c00 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.102 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.097+0000 7fdc64b92700 1 --2- 172.21.15.189:0/1691998562 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdc600ae770 0x7fdc60065b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.103 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.097+0000 7fdc64b92700 1 -- 172.21.15.189:0/1691998562 >> 172.21.15.189:0/1691998562 conn(0x7fdc600625a0 msgr2=0x7fdc600adc80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:59.103 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.097+0000 7fdc64b92700 1 -- 172.21.15.189:0/1691998562 shutdown_connections 2020-11-20T12:04:59.103 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.097+0000 7fdc64b92700 1 -- 172.21.15.189:0/1691998562 wait complete. 2020-11-20T12:04:59.105 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.101+0000 7fdc64b92700 1 Processor -- start 2020-11-20T12:04:59.106 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.101+0000 7fdc64b92700 1 -- start start 2020-11-20T12:04:59.106 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.101+0000 7fdc64b92700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc600ae390 0x7fdc60149730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.106 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.101+0000 7fdc64b92700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdc600ae770 0x7fdc600a3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.106 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.101+0000 7fdc64b92700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc600a2300 0x7fdc6013afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.106 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.101+0000 7fdc64b92700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc600a4b10 con 0x7fdc600ae390 2020-11-20T12:04:59.107 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.101+0000 7fdc64b92700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6013b760 con 0x7fdc600a2300 2020-11-20T12:04:59.107 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.101+0000 7fdc64b92700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc60133b50 con 0x7fdc600ae770 2020-11-20T12:04:59.107 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.101+0000 7fdc5ed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc600a2300 0x7fdc6013afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.107 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.101+0000 7fdc5ed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc600a2300 0x7fdc6013afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48436/0 (socket says 172.21.15.189:48436) 2020-11-20T12:04:59.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.101+0000 7fdc5ed9d700 1 -- 172.21.15.189:0/176807190 learned_addr learned my addr 172.21.15.189:0/176807190 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:59.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.101+0000 7fdc5e59c700 1 --2- 172.21.15.189:0/176807190 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc600ae390 0x7fdc60149730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.101+0000 7fdc57fff700 1 --2- 172.21.15.189:0/176807190 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdc600ae770 0x7fdc600a3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.101+0000 7fdc5ed9d700 1 -- 172.21.15.189:0/176807190 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdc600ae770 msgr2=0x7fdc600a3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.101+0000 7fdc5ed9d700 1 --2- 172.21.15.189:0/176807190 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdc600ae770 0x7fdc600a3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.109 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.101+0000 7fdc5ed9d700 1 -- 172.21.15.189:0/176807190 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc600ae390 msgr2=0x7fdc60149730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.109 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.101+0000 7fdc5ed9d700 1 --2- 172.21.15.189:0/176807190 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc600ae390 0x7fdc60149730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.109 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.101+0000 7fdc5ed9d700 1 -- 172.21.15.189:0/176807190 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc600a1bc0 con 0x7fdc600a2300 2020-11-20T12:04:59.109 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.101+0000 7fdc5e59c700 1 --2- 172.21.15.189:0/176807190 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc600ae390 0x7fdc60149730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:59.110 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.101+0000 7fdc5ed9d700 1 --2- 172.21.15.189:0/176807190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc600a2300 0x7fdc6013afb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fdc58011000 tx=0x7fdc58016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:59.110 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.101+0000 7fdc567fc700 1 -- 172.21.15.189:0/176807190 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc580108b0 con 0x7fdc600a2300 2020-11-20T12:04:59.110 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.101+0000 7fdc567fc700 1 -- 172.21.15.189:0/176807190 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc58010a10 con 0x7fdc600a2300 2020-11-20T12:04:59.110 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.105+0000 7fdc64b92700 1 -- 172.21.15.189:0/176807190 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc600a1e60 con 0x7fdc600a2300 2020-11-20T12:04:59.110 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.105+0000 7fdc567fc700 1 -- 172.21.15.189:0/176807190 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc58010db0 con 0x7fdc600a2300 2020-11-20T12:04:59.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.105+0000 7fdc64b92700 1 -- 172.21.15.189:0/176807190 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc600a2740 con 0x7fdc600a2300 2020-11-20T12:04:59.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.105+0000 7fdc64b92700 1 -- 172.21.15.189:0/176807190 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc60044f60 con 0x7fdc600a2300 2020-11-20T12:04:59.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.105+0000 7fdc567fc700 1 -- 172.21.15.189:0/176807190 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fdc58017070 con 0x7fdc600a2300 2020-11-20T12:04:59.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.105+0000 7fdc567fc700 1 --2- 172.21.15.189:0/176807190 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fdc3c0557c0 0x7fdc3c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.105+0000 7fdc567fc700 1 -- 172.21.15.189:0/176807190 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdc58070cd0 con 0x7fdc600a2300 2020-11-20T12:04:59.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.105+0000 7fdc5e59c700 1 --2- 172.21.15.189:0/176807190 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fdc3c0557c0 0x7fdc3c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.105+0000 7fdc5e59c700 1 --2- 172.21.15.189:0/176807190 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fdc3c0557c0 0x7fdc3c057c80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fdc500011a0 tx=0x7fdc50009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:59.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.113+0000 7fdc567fc700 1 -- 172.21.15.189:0/176807190 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fdc58046b70 con 0x7fdc600a2300 2020-11-20T12:04:59.262 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.261+0000 7fdc64b92700 1 -- 172.21.15.189:0/176807190 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc6013e0f0 con 0x7fdc600a2300 2020-11-20T12:04:59.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.265+0000 7fdc567fc700 1 -- 172.21.15.189:0/176807190 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16826 (secure 0 0 0) 0x7fdc58015030 con 0x7fdc600a2300 2020-11-20T12:04:59.264 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:04:59.265 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":22,"fsid":"cda18d0e-64b3-41bb-964a-084069c3e648","created":"2020-11-20T12:04:37.538706+0000","modified":"2020-11-20T12:04:55.015818+0000","last_up_change":"2020-11-20T12:04:43.006631+0000","last_in_change":"2020-11-20T12:04:43.006631+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:41.395395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:48.550454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:52.101837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:53.018196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c81214e1-4f84-4aaa-8ceb-99c398b2a89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6811","nonce":11760},{"type":"v1","addr":"172.21.15.189:6814","nonce":11760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":11760},{"type":"v1","addr":"172.21.15.189:6818","nonce":11760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":11760},{"type":"v1","addr":"172.21.15.189:6828","nonce":11760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6821","nonce":11760},{"type":"v1","addr":"172.21.15.189:6824","nonce":11760}]},"public_addr":"172.21.15.189:6814/11760","cluster_addr":"172.21.15.189:6818/11760","heartbeat_back_addr":"172.21.15.189:6828/11760","heartbeat_front_addr":"172.21.15.189:6824/11760","state":["exists","up"]},{"osd":1,"uuid":"d2f3ec7d-d403-484f-b5ed-b2e6b079d63d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6805","nonce":11761},{"type":"v1","addr":"172.21.15.189:6807","nonce":11761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":11761},{"type":"v1","addr":"172.21.15.189:6812","nonce":11761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6817","nonce":11761},{"type":"v1","addr":"172.21.15.189:6820","nonce":11761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6813","nonce":11761},{"type":"v1","addr":"172.21.15.189:6815","nonce":11761}]},"public_addr":"172.21.15.189:6807/11761","cluster_addr":"172.21.15.189:6812/11761","heartbeat_back_addr":"172.21.15.189:6820/11761","heartbeat_front_addr":"172.21.15.189:6815/11761","state":["exists","up"]},{"osd":2,"uuid":"e6cf4790-000c-43eb-8844-175f04c60503","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6819","nonce":11751},{"type":"v1","addr":"172.21.15.189:6822","nonce":11751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":11751},{"type":"v1","addr":"172.21.15.189:6825","nonce":11751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":11751},{"type":"v1","addr":"172.21.15.189:6831","nonce":11751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6827","nonce":11751},{"type":"v1","addr":"172.21.15.189:6829","nonce":11751}]},"public_addr":"172.21.15.189:6822/11751","cluster_addr":"172.21.15.189:6825/11751","heartbeat_back_addr":"172.21.15.189:6831/11751","heartbeat_front_addr":"172.21.15.189:6829/11751","state":["exists","up"]},{"osd":3,"uuid":"ba49dfac-f2b9-476e-853d-42f05e21a77a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":11762},{"type":"v1","addr":"172.21.15.189:6801","nonce":11762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":11762},{"type":"v1","addr":"172.21.15.189:6803","nonce":11762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":11762},{"type":"v1","addr":"172.21.15.189:6809","nonce":11762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":11762},{"type":"v1","addr":"172.21.15.189:6806","nonce":11762}]},"public_addr":"172.21.15.189:6801/11762","cluster_addr":"172.21.15.189:6803/11762","heartbeat_back_addr":"172.21.15.189:6809/11762","heartbeat_front_addr":"172.21.15.189:6806/11762","state":["exists","up"]},{"osd":4,"uuid":"3ca7b438-38b2-4689-8b18-fb5a8ddf6aee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":12114},{"type":"v1","addr":"172.21.15.47:6825","nonce":12114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":12114},{"type":"v1","addr":"172.21.15.47:6827","nonce":12114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":12114},{"type":"v1","addr":"172.21.15.47:6831","nonce":12114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":12114},{"type":"v1","addr":"172.21.15.47:6829","nonce":12114}]},"public_addr":"172.21.15.47:6825/12114","cluster_addr":"172.21.15.47:6827/12114","heartbeat_back_addr":"172.21.15.47:6831/12114","heartbeat_front_addr":"172.21.15.47:6829/12114","state":["exists","up"]},{"osd":5,"uuid":"6a45fca7-83d3-44fd-a1d4-b01e4201a101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":12111},{"type":"v1","addr":"172.21.15.47:6801","nonce":12111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":12111},{"type":"v1","addr":"172.21.15.47:6803","nonce":12111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":12111},{"type":"v1","addr":"172.21.15.47:6807","nonce":12111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":12111},{"type":"v1","addr":"172.21.15.47:6805","nonce":12111}]},"public_addr":"172.21.15.47:6801/12111","cluster_addr":"172.21.15.47:6803/12111","heartbeat_back_addr":"172.21.15.47:6807/12111","heartbeat_front_addr":"172.21.15.47:6805/12111","state":["exists","up"]},{"osd":6,"uuid":"c4b1b9ed-61f8-4677-9755-dff59a30e447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":12113},{"type":"v1","addr":"172.21.15.47:6817","nonce":12113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":12113},{"type":"v1","addr":"172.21.15.47:6819","nonce":12113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":12113},{"type":"v1","addr":"172.21.15.47:6823","nonce":12113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":12113},{"type":"v1","addr":"172.21.15.47:6821","nonce":12113}]},"public_addr":"172.21.15.47:6817/12113","cluster_addr":"172.21.15.47:6819/12113","heartbeat_back_addr":"172.21.15.47:6823/12113","heartbeat_front_addr":"172.21.15.47:6821/12113","state":["exists","up"]},{"osd":7,"uuid":"4608e4ed-cc40-4c5c-ba8b-3779bbc62f65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":12112},{"type":"v1","addr":"172.21.15.47:6809","nonce":12112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":12112},{"type":"v1","addr":"172.21.15.47:6811","nonce":12112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":12112},{"type":"v1","addr":"172.21.15.47:6815","nonce":12112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":12112},{"type":"v1","addr":"172.21.15.47:6813","nonce":12112}]},"public_addr":"172.21.15.47:6809/12112","cluster_addr":"172.21.15.47:6811/12112","heartbeat_back_addr":"172.21.15.47:6815/12112","heartbeat_front_addr":"172.21.15.47:6813/12112","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.435265+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.472776+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.388370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.417366+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.102392+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.116529+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.167430+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.148139+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:04:59.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.265+0000 7fdc64b92700 1 -- 172.21.15.189:0/176807190 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fdc3c0557c0 msgr2=0x7fdc3c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.265+0000 7fdc64b92700 1 --2- 172.21.15.189:0/176807190 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fdc3c0557c0 0x7fdc3c057c80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fdc500011a0 tx=0x7fdc50009040).stop 2020-11-20T12:04:59.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.265+0000 7fdc64b92700 1 -- 172.21.15.189:0/176807190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc600a2300 msgr2=0x7fdc6013afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.265+0000 7fdc64b92700 1 --2- 172.21.15.189:0/176807190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc600a2300 0x7fdc6013afb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fdc58011000 tx=0x7fdc58016000).stop 2020-11-20T12:04:59.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.265+0000 7fdc64b92700 1 -- 172.21.15.189:0/176807190 shutdown_connections 2020-11-20T12:04:59.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.265+0000 7fdc64b92700 1 --2- 172.21.15.189:0/176807190 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fdc3c0557c0 0x7fdc3c057c80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.265+0000 7fdc64b92700 1 --2- 172.21.15.189:0/176807190 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdc600ae390 0x7fdc60149730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.265+0000 7fdc64b92700 1 --2- 172.21.15.189:0/176807190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc600a2300 0x7fdc6013afb0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.265+0000 7fdc64b92700 1 --2- 172.21.15.189:0/176807190 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdc600ae770 0x7fdc600a3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.265+0000 7fdc64b92700 1 -- 172.21.15.189:0/176807190 >> 172.21.15.189:0/176807190 conn(0x7fdc600625a0 msgr2=0x7fdc600aae70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:59.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.265+0000 7fdc64b92700 1 -- 172.21.15.189:0/176807190 shutdown_connections 2020-11-20T12:04:59.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.265+0000 7fdc64b92700 1 -- 172.21.15.189:0/176807190 wait complete. 2020-11-20T12:04:59.281 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2020-11-20T12:04:59.403 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.401+0000 7fc50ccc3700 1 Processor -- start 2020-11-20T12:04:59.403 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.401+0000 7fc50ccc3700 1 -- start start 2020-11-20T12:04:59.405 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.401+0000 7fc50ccc3700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc508066ed0 0x7fc50813b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.405 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.401+0000 7fc50ccc3700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc508066800 con 0x7fc50813ba60 2020-11-20T12:04:59.405 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.401+0000 7fc50ccc3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc508066aa0 con 0x7fc5080a8840 2020-11-20T12:04:59.405 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.401+0000 7fc50ccc3700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc508067450 con 0x7fc508066ed0 2020-11-20T12:04:59.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.405+0000 7fc506ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc508066ed0 0x7fc50813b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.405+0000 7fc5077fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc5080a8840 0x7fc50813eb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50064/0 (socket says 172.21.15.189:50064) 2020-11-20T12:04:59.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.405+0000 7fc5077fe700 1 -- 172.21.15.189:0/1696122867 learned_addr learned my addr 172.21.15.189:0/1696122867 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:04:59.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.405+0000 7fc506ffd700 1 -- 172.21.15.189:0/1696122867 >> v1:172.21.15.189:6789/0 conn(0x7fc5080a8840 legacy=0x7fc50813eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.405+0000 7fc506ffd700 1 -- 172.21.15.189:0/1696122867 >> v1:172.21.15.47:6789/0 conn(0x7fc50813ba60 legacy=0x7fc5081495e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.405+0000 7fc506ffd700 1 -- 172.21.15.189:0/1696122867 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5080676f0 con 0x7fc508066ed0 2020-11-20T12:04:59.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.405+0000 7fc506ffd700 1 --2- 172.21.15.189:0/1696122867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc508066ed0 0x7fc50813b570 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fc4f000b9a0 tx=0x7fc4f0006df0).ready entity=mon.1 client_cookie=c9ed17feb1d5d99 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:59.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.405+0000 7fc5067fc700 1 -- 172.21.15.189:0/1696122867 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f00097b0 con 0x7fc508066ed0 2020-11-20T12:04:59.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.405+0000 7fc5067fc700 1 -- 172.21.15.189:0/1696122867 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4f0009910 con 0x7fc508066ed0 2020-11-20T12:04:59.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.405+0000 7fc5067fc700 1 -- 172.21.15.189:0/1696122867 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f0009c90 con 0x7fc508066ed0 2020-11-20T12:04:59.410 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.405+0000 7fc50ccc3700 1 -- 172.21.15.189:0/1696122867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc508066ed0 msgr2=0x7fc50813b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.410 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.405+0000 7fc50ccc3700 1 --2- 172.21.15.189:0/1696122867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc508066ed0 0x7fc50813b570 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fc4f000b9a0 tx=0x7fc4f0006df0).stop 2020-11-20T12:04:59.410 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.405+0000 7fc50ccc3700 1 -- 172.21.15.189:0/1696122867 shutdown_connections 2020-11-20T12:04:59.410 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.405+0000 7fc50ccc3700 1 --2- 172.21.15.189:0/1696122867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc508066ed0 0x7fc50813b570 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.410 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.405+0000 7fc50ccc3700 1 -- 172.21.15.189:0/1696122867 >> 172.21.15.189:0/1696122867 conn(0x7fc508060170 msgr2=0x7fc508060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:59.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.405+0000 7fc50ccc3700 1 -- 172.21.15.189:0/1696122867 shutdown_connections 2020-11-20T12:04:59.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.405+0000 7fc50ccc3700 1 -- 172.21.15.189:0/1696122867 wait complete. 2020-11-20T12:04:59.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.405+0000 7fc50ccc3700 1 Processor -- start 2020-11-20T12:04:59.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.405+0000 7fc50ccc3700 1 -- start start 2020-11-20T12:04:59.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.409+0000 7fc50ccc3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5080a8840 0x7fc50813e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.412 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.409+0000 7fc50ccc3700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc50813ba60 0x7fc5081412c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.412 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.409+0000 7fc50ccc3700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5080a64e0 0x7fc5080a4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.412 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.409+0000 7fc50ccc3700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5080a4440 con 0x7fc5080a64e0 2020-11-20T12:04:59.412 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.409+0000 7fc50ccc3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5080a6bf0 con 0x7fc5080a8840 2020-11-20T12:04:59.412 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.409+0000 7fc50ccc3700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5080a4670 con 0x7fc50813ba60 2020-11-20T12:04:59.413 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.409+0000 7fc5077fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5080a8840 0x7fc50813e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.413 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.409+0000 7fc5077fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5080a8840 0x7fc50813e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48444/0 (socket says 172.21.15.189:48444) 2020-11-20T12:04:59.413 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.409+0000 7fc5077fe700 1 -- 172.21.15.189:0/4012071887 learned_addr learned my addr 172.21.15.189:0/4012071887 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:59.413 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.409+0000 7fc506ffd700 1 --2- 172.21.15.189:0/4012071887 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc50813ba60 0x7fc5081412c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.409+0000 7fc507fff700 1 --2- 172.21.15.189:0/4012071887 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5080a64e0 0x7fc5080a4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.409+0000 7fc5077fe700 1 -- 172.21.15.189:0/4012071887 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc50813ba60 msgr2=0x7fc5081412c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.409+0000 7fc5077fe700 1 --2- 172.21.15.189:0/4012071887 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc50813ba60 0x7fc5081412c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.409+0000 7fc5077fe700 1 -- 172.21.15.189:0/4012071887 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5080a64e0 msgr2=0x7fc5080a4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.409+0000 7fc5077fe700 1 --2- 172.21.15.189:0/4012071887 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5080a64e0 0x7fc5080a4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.409+0000 7fc5077fe700 1 -- 172.21.15.189:0/4012071887 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5080a6940 con 0x7fc5080a8840 2020-11-20T12:04:59.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.409+0000 7fc5077fe700 1 --2- 172.21.15.189:0/4012071887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5080a8840 0x7fc50813e830 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fc4f800ca90 tx=0x7fc4f8001460).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:59.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.409+0000 7fc4ff7fe700 1 -- 172.21.15.189:0/4012071887 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8005b80 con 0x7fc5080a8840 2020-11-20T12:04:59.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.409+0000 7fc4ff7fe700 1 -- 172.21.15.189:0/4012071887 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4f8005ce0 con 0x7fc5080a8840 2020-11-20T12:04:59.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.409+0000 7fc50ccc3700 1 -- 172.21.15.189:0/4012071887 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc508151760 con 0x7fc5080a8840 2020-11-20T12:04:59.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.409+0000 7fc4ff7fe700 1 -- 172.21.15.189:0/4012071887 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8007320 con 0x7fc5080a8840 2020-11-20T12:04:59.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.409+0000 7fc50ccc3700 1 -- 172.21.15.189:0/4012071887 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5080a51a0 con 0x7fc5080a8840 2020-11-20T12:04:59.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.409+0000 7fc50ccc3700 1 -- 172.21.15.189:0/4012071887 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc5080a6d60 con 0x7fc5080a8840 2020-11-20T12:04:59.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.409+0000 7fc4ff7fe700 1 -- 172.21.15.189:0/4012071887 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fc4f8010030 con 0x7fc5080a8840 2020-11-20T12:04:59.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.413+0000 7fc4ff7fe700 1 --2- 172.21.15.189:0/4012071887 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc4e40557c0 0x7fc4e4057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.413+0000 7fc4ff7fe700 1 -- 172.21.15.189:0/4012071887 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc4f806f000 con 0x7fc5080a8840 2020-11-20T12:04:59.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.413+0000 7fc506ffd700 1 --2- 172.21.15.189:0/4012071887 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc4e40557c0 0x7fc4e4057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.413+0000 7fc4ff7fe700 1 --2- 172.21.15.189:0/4012071887 >> [v2:172.21.15.189:6811/11760,v1:172.21.15.189:6814/11760] conn(0x7fc4e405cd10 0x7fc4e405f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.413+0000 7fc4ff7fe700 1 -- 172.21.15.189:0/4012071887 --> [v2:172.21.15.189:6811/11760,v1:172.21.15.189:6814/11760] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc4e405f790 con 0x7fc4e405cd10 2020-11-20T12:04:59.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.413+0000 7fc4ff7fe700 1 -- 172.21.15.189:0/4012071887 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4f806f400 con 0x7fc5080a8840 2020-11-20T12:04:59.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.413+0000 7fc507fff700 1 --2- 172.21.15.189:0/4012071887 >> [v2:172.21.15.189:6811/11760,v1:172.21.15.189:6814/11760] conn(0x7fc4e405cd10 0x7fc4e405f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.413+0000 7fc506ffd700 1 --2- 172.21.15.189:0/4012071887 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc4e40557c0 0x7fc4e4057c80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fc5080672b0 tx=0x7fc4f0015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:59.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.413+0000 7fc507fff700 1 --2- 172.21.15.189:0/4012071887 >> [v2:172.21.15.189:6811/11760,v1:172.21.15.189:6814/11760] conn(0x7fc4e405cd10 0x7fc4e405f130 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:59.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.413+0000 7fc4ff7fe700 1 -- 172.21.15.189:0/4012071887 <== osd.0 v2:172.21.15.189:6811/11760 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7fc4e405f790 con 0x7fc4e405cd10 2020-11-20T12:04:59.432 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.433+0000 7fc50ccc3700 1 -- 172.21.15.189:0/4012071887 --> [v2:172.21.15.189:6811/11760,v1:172.21.15.189:6814/11760] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc5080a3b90 con 0x7fc4e405cd10 2020-11-20T12:04:59.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.433+0000 7fc4ff7fe700 1 -- 172.21.15.189:0/4012071887 <== osd.0 v2:172.21.15.189:6811/11760 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc5080a3b90 con 0x7fc4e405cd10 2020-11-20T12:04:59.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.433+0000 7fc50ccc3700 1 -- 172.21.15.189:0/4012071887 >> [v2:172.21.15.189:6811/11760,v1:172.21.15.189:6814/11760] conn(0x7fc4e405cd10 msgr2=0x7fc4e405f130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.433+0000 7fc50ccc3700 1 --2- 172.21.15.189:0/4012071887 >> [v2:172.21.15.189:6811/11760,v1:172.21.15.189:6814/11760] conn(0x7fc4e405cd10 0x7fc4e405f130 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.433+0000 7fc50ccc3700 1 -- 172.21.15.189:0/4012071887 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc4e40557c0 msgr2=0x7fc4e4057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.433+0000 7fc50ccc3700 1 --2- 172.21.15.189:0/4012071887 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc4e40557c0 0x7fc4e4057c80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fc5080672b0 tx=0x7fc4f0015000).stop 2020-11-20T12:04:59.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.433+0000 7fc50ccc3700 1 -- 172.21.15.189:0/4012071887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5080a8840 msgr2=0x7fc50813e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.433+0000 7fc50ccc3700 1 --2- 172.21.15.189:0/4012071887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5080a8840 0x7fc50813e830 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fc4f800ca90 tx=0x7fc4f8001460).stop 2020-11-20T12:04:59.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.433+0000 7fc507fff700 1 -- 172.21.15.189:0/4012071887 reap_dead start 2020-11-20T12:04:59.436 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.433+0000 7fc50ccc3700 1 -- 172.21.15.189:0/4012071887 shutdown_connections 2020-11-20T12:04:59.436 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.433+0000 7fc50ccc3700 1 -- 172.21.15.189:0/4012071887 >> 172.21.15.189:0/4012071887 conn(0x7fc508060170 msgr2=0x7fc508144320 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:59.436 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.433+0000 7fc50ccc3700 1 -- 172.21.15.189:0/4012071887 shutdown_connections 2020-11-20T12:04:59.436 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.433+0000 7fc50ccc3700 1 -- 172.21.15.189:0/4012071887 wait complete. 2020-11-20T12:04:59.450 INFO:teuthology.orchestra.run.smithi189.stdout:55834574853 2020-11-20T12:04:59.451 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2020-11-20T12:04:59.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.569+0000 7f0ff359e700 1 Processor -- start 2020-11-20T12:04:59.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.569+0000 7f0ff359e700 1 -- start start 2020-11-20T12:04:59.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.569+0000 7f0ff359e700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0ff40aebd0 0x7f0ff40aa620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.569+0000 7f0ff359e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0ff40aefb0 0x7f0ff40a9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.569+0000 7f0ff359e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ff40a9fd0 0x7f0ff413be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.569+0000 7f0ff359e700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff40a9680 con 0x7f0ff40aebd0 2020-11-20T12:04:59.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.569+0000 7f0ff359e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff40aae50 con 0x7f0ff40aefb0 2020-11-20T12:04:59.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.569+0000 7f0ff359e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff413ba10 con 0x7f0ff40a9fd0 2020-11-20T12:04:59.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.569+0000 7f0ff1d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0ff40aefb0 0x7f0ff40a9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.569+0000 7f0ff1d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0ff40aefb0 0x7f0ff40a9ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48454/0 (socket says 172.21.15.189:48454) 2020-11-20T12:04:59.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.569+0000 7f0ff1d9b700 1 -- 172.21.15.189:0/2558631911 learned_addr learned my addr 172.21.15.189:0/2558631911 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:59.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.569+0000 7f0ff2d9d700 1 --2- 172.21.15.189:0/2558631911 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ff40a9fd0 0x7f0ff413be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.569+0000 7f0ff259c700 1 --2- 172.21.15.189:0/2558631911 >> v2:172.21.15.47:3300/0 conn(0x7f0ff40aebd0 0x7f0ff40aa620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff1d9b700 1 -- 172.21.15.189:0/2558631911 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ff40a9fd0 msgr2=0x7f0ff413be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:59.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff1d9b700 1 --2- 172.21.15.189:0/2558631911 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ff40a9fd0 0x7f0ff413be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff1d9b700 1 -- 172.21.15.189:0/2558631911 >> v2:172.21.15.47:3300/0 conn(0x7f0ff40aebd0 msgr2=0x7f0ff40aa620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:59.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff1d9b700 1 --2- 172.21.15.189:0/2558631911 >> v2:172.21.15.47:3300/0 conn(0x7f0ff40aebd0 0x7f0ff40aa620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff1d9b700 1 -- 172.21.15.189:0/2558631911 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff40aab10 con 0x7f0ff40aefb0 2020-11-20T12:04:59.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff259c700 1 --2- 172.21.15.189:0/2558631911 >> v2:172.21.15.47:3300/0 conn(0x7f0ff40aebd0 0x7f0ff40aa620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:04:59.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff1d9b700 1 --2- 172.21.15.189:0/2558631911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ff40aefb0 0x7f0ff40a9ae0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f0ff40a6cd0 tx=0x7f0fe4009290).ready entity=mon.1 client_cookie=6ac889fc7c95ce44 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:59.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff159a700 1 -- 172.21.15.189:0/2558631911 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe400bbc0 con 0x7f0ff40aefb0 2020-11-20T12:04:59.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff159a700 1 -- 172.21.15.189:0/2558631911 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe400bd20 con 0x7f0ff40aefb0 2020-11-20T12:04:59.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff159a700 1 -- 172.21.15.189:0/2558631911 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe4011610 con 0x7f0ff40aefb0 2020-11-20T12:04:59.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff359e700 1 -- 172.21.15.189:0/2558631911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ff40aefb0 msgr2=0x7f0ff40a9ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff359e700 1 --2- 172.21.15.189:0/2558631911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ff40aefb0 0x7f0ff40a9ae0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f0ff40a6cd0 tx=0x7f0fe4009290).stop 2020-11-20T12:04:59.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff359e700 1 -- 172.21.15.189:0/2558631911 shutdown_connections 2020-11-20T12:04:59.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff359e700 1 --2- 172.21.15.189:0/2558631911 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ff40a9fd0 0x7f0ff413be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff359e700 1 --2- 172.21.15.189:0/2558631911 >> v2:172.21.15.47:3300/0 conn(0x7f0ff40aebd0 0x7f0ff40aa620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff359e700 1 --2- 172.21.15.189:0/2558631911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ff40aefb0 0x7f0ff40a9ae0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff359e700 1 -- 172.21.15.189:0/2558631911 >> 172.21.15.189:0/2558631911 conn(0x7f0ff4062080 msgr2=0x7f0ff4062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:59.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff359e700 1 -- 172.21.15.189:0/2558631911 shutdown_connections 2020-11-20T12:04:59.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff359e700 1 -- 172.21.15.189:0/2558631911 wait complete. 2020-11-20T12:04:59.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff359e700 1 Processor -- start 2020-11-20T12:04:59.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff359e700 1 -- start start 2020-11-20T12:04:59.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff359e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ff40a9fd0 0x7f0ff40a2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff359e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ff40aebd0 0x7f0ff41414f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff359e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ff40aefb0 0x7f0ff4063fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff359e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff40a30f0 con 0x7f0ff40aebd0 2020-11-20T12:04:59.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff359e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff40a5260 con 0x7f0ff40aefb0 2020-11-20T12:04:59.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff359e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff40644b0 con 0x7f0ff40a9fd0 2020-11-20T12:04:59.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff2d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ff40aefb0 0x7f0ff4063fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff2d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ff40aefb0 0x7f0ff4063fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48464/0 (socket says 172.21.15.189:48464) 2020-11-20T12:04:59.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff2d9d700 1 -- 172.21.15.189:0/2481565004 learned_addr learned my addr 172.21.15.189:0/2481565004 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:59.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff259c700 1 --2- 172.21.15.189:0/2481565004 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ff40a9fd0 0x7f0ff40a2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff1d9b700 1 --2- 172.21.15.189:0/2481565004 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ff40aebd0 0x7f0ff41414f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff2d9d700 1 -- 172.21.15.189:0/2481565004 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ff40a9fd0 msgr2=0x7f0ff40a2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff2d9d700 1 --2- 172.21.15.189:0/2481565004 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ff40a9fd0 0x7f0ff40a2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff2d9d700 1 -- 172.21.15.189:0/2481565004 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ff40aebd0 msgr2=0x7f0ff41414f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff2d9d700 1 --2- 172.21.15.189:0/2481565004 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ff40aebd0 0x7f0ff41414f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.588 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff2d9d700 1 -- 172.21.15.189:0/2481565004 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff40a5700 con 0x7f0ff40aefb0 2020-11-20T12:04:59.588 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.573+0000 7f0ff2d9d700 1 --2- 172.21.15.189:0/2481565004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ff40aefb0 0x7f0ff4063fc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f0fec00d860 tx=0x7f0fec008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:59.588 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.577+0000 7f0fea7fc700 1 -- 172.21.15.189:0/2481565004 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fec0096d0 con 0x7f0ff40aefb0 2020-11-20T12:04:59.588 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.577+0000 7f0fea7fc700 1 -- 172.21.15.189:0/2481565004 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fec009830 con 0x7f0ff40aefb0 2020-11-20T12:04:59.588 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.577+0000 7f0fea7fc700 1 -- 172.21.15.189:0/2481565004 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fec009bd0 con 0x7f0ff40aefb0 2020-11-20T12:04:59.589 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.577+0000 7f0ff359e700 1 -- 172.21.15.189:0/2481565004 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff40a4bc0 con 0x7f0ff40aefb0 2020-11-20T12:04:59.589 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.577+0000 7f0ff359e700 1 -- 172.21.15.189:0/2481565004 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff41351d0 con 0x7f0ff40aefb0 2020-11-20T12:04:59.589 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.577+0000 7f0ff359e700 1 -- 172.21.15.189:0/2481565004 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0ff40a59b0 con 0x7f0ff40aefb0 2020-11-20T12:04:59.589 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.577+0000 7f0fea7fc700 1 -- 172.21.15.189:0/2481565004 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f0fec013480 con 0x7f0ff40aefb0 2020-11-20T12:04:59.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.577+0000 7f0fea7fc700 1 --2- 172.21.15.189:0/2481565004 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0fd00557c0 0x7f0fd0057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.577+0000 7f0fea7fc700 1 -- 172.21.15.189:0/2481565004 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0fec06f430 con 0x7f0ff40aefb0 2020-11-20T12:04:59.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.577+0000 7f0ff259c700 1 --2- 172.21.15.189:0/2481565004 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0fd00557c0 0x7f0fd0057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.577+0000 7f0fea7fc700 1 --2- 172.21.15.189:0/2481565004 >> [v2:172.21.15.189:6805/11761,v1:172.21.15.189:6807/11761] conn(0x7f0fd005cd10 0x7f0fd005f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.577+0000 7f0fea7fc700 1 -- 172.21.15.189:0/2481565004 --> [v2:172.21.15.189:6805/11761,v1:172.21.15.189:6807/11761] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0fd005f790 con 0x7f0fd005cd10 2020-11-20T12:04:59.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.577+0000 7f0fea7fc700 1 -- 172.21.15.189:0/2481565004 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0fec06f830 con 0x7f0ff40aefb0 2020-11-20T12:04:59.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.577+0000 7f0ff1d9b700 1 --2- 172.21.15.189:0/2481565004 >> [v2:172.21.15.189:6805/11761,v1:172.21.15.189:6807/11761] conn(0x7f0fd005cd10 0x7f0fd005f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.577+0000 7f0ff259c700 1 --2- 172.21.15.189:0/2481565004 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0fd00557c0 0x7f0fd0057c80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f0fdc00a8c0 tx=0x7f0fdc008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:59.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.581+0000 7f0ff1d9b700 1 --2- 172.21.15.189:0/2481565004 >> [v2:172.21.15.189:6805/11761,v1:172.21.15.189:6807/11761] conn(0x7f0fd005cd10 0x7f0fd005f130 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:59.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.581+0000 7f0fea7fc700 1 -- 172.21.15.189:0/2481565004 <== osd.1 v2:172.21.15.189:6805/11761 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f0fd005f790 con 0x7f0fd005cd10 2020-11-20T12:04:59.598 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.597+0000 7f0ff359e700 1 -- 172.21.15.189:0/2481565004 --> [v2:172.21.15.189:6805/11761,v1:172.21.15.189:6807/11761] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0ff40a32f0 con 0x7f0fd005cd10 2020-11-20T12:04:59.599 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.597+0000 7f0fea7fc700 1 -- 172.21.15.189:0/2481565004 <== osd.1 v2:172.21.15.189:6805/11761 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0fe4006d30 con 0x7f0fd005cd10 2020-11-20T12:04:59.599 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.597+0000 7f0ff359e700 1 -- 172.21.15.189:0/2481565004 >> [v2:172.21.15.189:6805/11761,v1:172.21.15.189:6807/11761] conn(0x7f0fd005cd10 msgr2=0x7f0fd005f130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.600 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.597+0000 7f0ff359e700 1 --2- 172.21.15.189:0/2481565004 >> [v2:172.21.15.189:6805/11761,v1:172.21.15.189:6807/11761] conn(0x7f0fd005cd10 0x7f0fd005f130 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.601 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.597+0000 7f0ff359e700 1 -- 172.21.15.189:0/2481565004 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0fd00557c0 msgr2=0x7f0fd0057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.601 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.597+0000 7f0ff359e700 1 --2- 172.21.15.189:0/2481565004 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0fd00557c0 0x7f0fd0057c80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f0fdc00a8c0 tx=0x7f0fdc008040).stop 2020-11-20T12:04:59.601 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.601+0000 7f0ff359e700 1 -- 172.21.15.189:0/2481565004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ff40aefb0 msgr2=0x7f0ff4063fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.601 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.601+0000 7f0ff359e700 1 --2- 172.21.15.189:0/2481565004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ff40aefb0 0x7f0ff4063fc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f0fec00d860 tx=0x7f0fec008040).stop 2020-11-20T12:04:59.602 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.601+0000 7f0ff2d9d700 1 -- 172.21.15.189:0/2481565004 reap_dead start 2020-11-20T12:04:59.602 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.601+0000 7f0ff359e700 1 -- 172.21.15.189:0/2481565004 shutdown_connections 2020-11-20T12:04:59.602 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.601+0000 7f0ff359e700 1 -- 172.21.15.189:0/2481565004 >> 172.21.15.189:0/2481565004 conn(0x7f0ff4062080 msgr2=0x7f0ff40608c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:59.602 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.601+0000 7f0ff359e700 1 -- 172.21.15.189:0/2481565004 shutdown_connections 2020-11-20T12:04:59.602 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.601+0000 7f0ff359e700 1 -- 172.21.15.189:0/2481565004 wait complete. 2020-11-20T12:04:59.613 INFO:teuthology.orchestra.run.smithi189.stdout:55834574853 2020-11-20T12:04:59.613 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2020-11-20T12:04:59.741 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.741+0000 7ff233fff700 1 Processor -- start 2020-11-20T12:04:59.741 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.741+0000 7ff233fff700 1 -- start start 2020-11-20T12:04:59.742 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.741+0000 7ff233fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff2341386b0 0x7ff2340a9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.741+0000 7ff233fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2340a9a60 con 0x7ff234135b60 2020-11-20T12:04:59.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.741+0000 7ff233fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2340a9d00 con 0x7ff2340a50e0 2020-11-20T12:04:59.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.741+0000 7ff233fff700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff234135ff0 con 0x7ff2341386b0 2020-11-20T12:04:59.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.741+0000 7ff232ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff2341386b0 0x7ff2340a9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.741+0000 7ff22a7fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff2340a50e0 0x7ff234135450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50092/0 (socket says 172.21.15.189:50092) 2020-11-20T12:04:59.744 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.741+0000 7ff22a7fc700 1 -- 172.21.15.189:0/467445815 learned_addr learned my addr 172.21.15.189:0/467445815 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:04:59.744 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.741+0000 7ff232ffd700 1 -- 172.21.15.189:0/467445815 >> v1:172.21.15.189:6789/0 conn(0x7ff2340a50e0 legacy=0x7ff234135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.744 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.741+0000 7ff232ffd700 1 -- 172.21.15.189:0/467445815 >> v1:172.21.15.47:6789/0 conn(0x7ff234135b60 legacy=0x7ff234142df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.744 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.741+0000 7ff232ffd700 1 -- 172.21.15.189:0/467445815 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2340a9fa0 con 0x7ff2341386b0 2020-11-20T12:04:59.747 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.741+0000 7ff232ffd700 1 --2- 172.21.15.189:0/467445815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2341386b0 0x7ff2340a9410 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7ff22400b920 tx=0x7ff224006df0).ready entity=mon.1 client_cookie=8c880b483d1b7681 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:59.747 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.741+0000 7ff2327fc700 1 -- 172.21.15.189:0/467445815 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff224009730 con 0x7ff2341386b0 2020-11-20T12:04:59.747 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.741+0000 7ff2327fc700 1 -- 172.21.15.189:0/467445815 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff224009890 con 0x7ff2341386b0 2020-11-20T12:04:59.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.741+0000 7ff2327fc700 1 -- 172.21.15.189:0/467445815 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff224009c10 con 0x7ff2341386b0 2020-11-20T12:04:59.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff233fff700 1 -- 172.21.15.189:0/467445815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2341386b0 msgr2=0x7ff2340a9410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff233fff700 1 --2- 172.21.15.189:0/467445815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2341386b0 0x7ff2340a9410 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7ff22400b920 tx=0x7ff224006df0).stop 2020-11-20T12:04:59.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff233fff700 1 -- 172.21.15.189:0/467445815 shutdown_connections 2020-11-20T12:04:59.749 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff233fff700 1 --2- 172.21.15.189:0/467445815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2341386b0 0x7ff2340a9410 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.749 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff233fff700 1 -- 172.21.15.189:0/467445815 >> 172.21.15.189:0/467445815 conn(0x7ff234060170 msgr2=0x7ff234060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:59.749 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff233fff700 1 -- 172.21.15.189:0/467445815 shutdown_connections 2020-11-20T12:04:59.749 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff233fff700 1 -- 172.21.15.189:0/467445815 wait complete. 2020-11-20T12:04:59.749 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff233fff700 1 Processor -- start 2020-11-20T12:04:59.750 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff233fff700 1 -- start start 2020-11-20T12:04:59.750 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff233fff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff2340a50e0 0x7ff23409dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.750 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff233fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff234135b60 0x7ff23413df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.750 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff233fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2341386b0 0x7ff234066f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.750 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff233fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff234067480 con 0x7ff234135b60 2020-11-20T12:04:59.750 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff233fff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2340676a0 con 0x7ff2341386b0 2020-11-20T12:04:59.751 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff233fff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff234067940 con 0x7ff2340a50e0 2020-11-20T12:04:59.751 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff2337fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2341386b0 0x7ff234066f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.751 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff2337fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2341386b0 0x7ff234066f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48480/0 (socket says 172.21.15.189:48480) 2020-11-20T12:04:59.751 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff2337fe700 1 -- 172.21.15.189:0/301902395 learned_addr learned my addr 172.21.15.189:0/301902395 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:59.752 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff22a7fc700 1 --2- 172.21.15.189:0/301902395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff234135b60 0x7ff23413df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.752 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff232ffd700 1 --2- 172.21.15.189:0/301902395 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff2340a50e0 0x7ff23409dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.752 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff2337fe700 1 -- 172.21.15.189:0/301902395 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff2340a50e0 msgr2=0x7ff23409dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.752 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff2337fe700 1 --2- 172.21.15.189:0/301902395 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff2340a50e0 0x7ff23409dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.752 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff2337fe700 1 -- 172.21.15.189:0/301902395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff234135b60 msgr2=0x7ff23413df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.753 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff2337fe700 1 --2- 172.21.15.189:0/301902395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff234135b60 0x7ff23413df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.753 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff2337fe700 1 -- 172.21.15.189:0/301902395 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2340661f0 con 0x7ff2341386b0 2020-11-20T12:04:59.753 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff22a7fc700 1 --2- 172.21.15.189:0/301902395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff234135b60 0x7ff23413df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:04:59.753 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff2337fe700 1 --2- 172.21.15.189:0/301902395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2341386b0 0x7ff234066f90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7ff22c00cb10 tx=0x7ff22c00ced0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:59.753 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff22b7fe700 1 -- 172.21.15.189:0/301902395 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22c00a770 con 0x7ff2341386b0 2020-11-20T12:04:59.754 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff22b7fe700 1 -- 172.21.15.189:0/301902395 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff22c00a8d0 con 0x7ff2341386b0 2020-11-20T12:04:59.754 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff233fff700 1 -- 172.21.15.189:0/301902395 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff234066440 con 0x7ff2341386b0 2020-11-20T12:04:59.754 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff22b7fe700 1 -- 172.21.15.189:0/301902395 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22c00ac90 con 0x7ff2341386b0 2020-11-20T12:04:59.754 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.745+0000 7ff233fff700 1 -- 172.21.15.189:0/301902395 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2340669e0 con 0x7ff2341386b0 2020-11-20T12:04:59.756 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.749+0000 7ff233fff700 1 -- 172.21.15.189:0/301902395 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff2340446d0 con 0x7ff2341386b0 2020-11-20T12:04:59.756 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.749+0000 7ff22b7fe700 1 -- 172.21.15.189:0/301902395 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7ff22c010070 con 0x7ff2341386b0 2020-11-20T12:04:59.756 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.749+0000 7ff22b7fe700 1 --2- 172.21.15.189:0/301902395 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff2100557c0 0x7ff210057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.756 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.749+0000 7ff22b7fe700 1 -- 172.21.15.189:0/301902395 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff22c06e520 con 0x7ff2341386b0 2020-11-20T12:04:59.756 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.749+0000 7ff232ffd700 1 --2- 172.21.15.189:0/301902395 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff2100557c0 0x7ff210057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.757 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.749+0000 7ff22b7fe700 1 --2- 172.21.15.189:0/301902395 >> [v2:172.21.15.189:6819/11751,v1:172.21.15.189:6822/11751] conn(0x7ff21005cd10 0x7ff21005f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.757 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.749+0000 7ff22b7fe700 1 -- 172.21.15.189:0/301902395 --> [v2:172.21.15.189:6819/11751,v1:172.21.15.189:6822/11751] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff21005f790 con 0x7ff21005cd10 2020-11-20T12:04:59.757 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.749+0000 7ff22b7fe700 1 -- 172.21.15.189:0/301902395 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff22c070b90 con 0x7ff2341386b0 2020-11-20T12:04:59.758 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.749+0000 7ff22a7fc700 1 --2- 172.21.15.189:0/301902395 >> [v2:172.21.15.189:6819/11751,v1:172.21.15.189:6822/11751] conn(0x7ff21005cd10 0x7ff21005f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.758 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.749+0000 7ff232ffd700 1 --2- 172.21.15.189:0/301902395 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff2100557c0 0x7ff210057c80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7ff224006e30 tx=0x7ff224015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:59.758 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.749+0000 7ff22a7fc700 1 --2- 172.21.15.189:0/301902395 >> [v2:172.21.15.189:6819/11751,v1:172.21.15.189:6822/11751] conn(0x7ff21005cd10 0x7ff21005f130 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:59.758 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.749+0000 7ff22b7fe700 1 -- 172.21.15.189:0/301902395 <== osd.2 v2:172.21.15.189:6819/11751 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7ff21005f790 con 0x7ff21005cd10 2020-11-20T12:04:59.769 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.769+0000 7ff233fff700 1 -- 172.21.15.189:0/301902395 --> [v2:172.21.15.189:6819/11751,v1:172.21.15.189:6822/11751] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff234065f50 con 0x7ff21005cd10 2020-11-20T12:04:59.769 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.769+0000 7ff22b7fe700 1 -- 172.21.15.189:0/301902395 <== osd.2 v2:172.21.15.189:6819/11751 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff234065f50 con 0x7ff21005cd10 2020-11-20T12:04:59.771 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.769+0000 7ff233fff700 1 -- 172.21.15.189:0/301902395 >> [v2:172.21.15.189:6819/11751,v1:172.21.15.189:6822/11751] conn(0x7ff21005cd10 msgr2=0x7ff21005f130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.771 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.769+0000 7ff233fff700 1 --2- 172.21.15.189:0/301902395 >> [v2:172.21.15.189:6819/11751,v1:172.21.15.189:6822/11751] conn(0x7ff21005cd10 0x7ff21005f130 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.771 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.769+0000 7ff233fff700 1 -- 172.21.15.189:0/301902395 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff2100557c0 msgr2=0x7ff210057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.772 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.769+0000 7ff233fff700 1 --2- 172.21.15.189:0/301902395 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff2100557c0 0x7ff210057c80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7ff224006e30 tx=0x7ff224015000).stop 2020-11-20T12:04:59.772 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.769+0000 7ff233fff700 1 -- 172.21.15.189:0/301902395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2341386b0 msgr2=0x7ff234066f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.773 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.769+0000 7ff233fff700 1 --2- 172.21.15.189:0/301902395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2341386b0 0x7ff234066f90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7ff22c00cb10 tx=0x7ff22c00ced0).stop 2020-11-20T12:04:59.773 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.769+0000 7ff2337fe700 1 -- 172.21.15.189:0/301902395 reap_dead start 2020-11-20T12:04:59.773 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.769+0000 7ff233fff700 1 -- 172.21.15.189:0/301902395 shutdown_connections 2020-11-20T12:04:59.774 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.769+0000 7ff233fff700 1 -- 172.21.15.189:0/301902395 >> 172.21.15.189:0/301902395 conn(0x7ff234060170 msgr2=0x7ff2340ac990 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.769+0000 7ff233fff700 1 -- 172.21.15.189:0/301902395 shutdown_connections 2020-11-20T12:04:59.775 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.769+0000 7ff233fff700 1 -- 172.21.15.189:0/301902395 wait complete. 2020-11-20T12:04:59.787 INFO:teuthology.orchestra.run.smithi189.stdout:55834574853 2020-11-20T12:04:59.788 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2020-11-20T12:04:59.909 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd1af3700 1 Processor -- start 2020-11-20T12:04:59.909 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd1af3700 1 -- start start 2020-11-20T12:04:59.911 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd1af3700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6bcc0a8840 0x7f6bcc13dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.912 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd1af3700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6bcc13e100 0x7f6bcc1371b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.912 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd1af3700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6bcc1376a0 0x7f6bcc137b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.912 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd1af3700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bcc138040 con 0x7f6bcc1376a0 2020-11-20T12:04:59.913 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd1af3700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bcc066800 con 0x7f6bcc0a8840 2020-11-20T12:04:59.913 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd1af3700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc066aa0 con 0x7f6bcc13e100 2020-11-20T12:04:59.913 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd0af1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6bcc0a8840 0x7f6bcc13dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.914 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd0af1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6bcc0a8840 0x7f6bcc13dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48488/0 (socket says 172.21.15.189:48488) 2020-11-20T12:04:59.914 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd0af1700 1 -- 172.21.15.189:0/640650956 learned_addr learned my addr 172.21.15.189:0/640650956 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:59.914 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd12f2700 1 --2- 172.21.15.189:0/640650956 >> v2:172.21.15.47:3300/0 conn(0x7f6bcc1376a0 0x7f6bcc137b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.914 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bcbfff700 1 --2- 172.21.15.189:0/640650956 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6bcc13e100 0x7f6bcc1371b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.915 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd0af1700 1 -- 172.21.15.189:0/640650956 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6bcc13e100 msgr2=0x7f6bcc1371b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:59.915 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd0af1700 1 --2- 172.21.15.189:0/640650956 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6bcc13e100 0x7f6bcc1371b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.915 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd0af1700 1 -- 172.21.15.189:0/640650956 >> v2:172.21.15.47:3300/0 conn(0x7f6bcc1376a0 msgr2=0x7f6bcc137b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:04:59.915 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd0af1700 1 --2- 172.21.15.189:0/640650956 >> v2:172.21.15.47:3300/0 conn(0x7f6bcc1376a0 0x7f6bcc137b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.916 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd0af1700 1 -- 172.21.15.189:0/640650956 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bcc0672f0 con 0x7f6bcc0a8840 2020-11-20T12:04:59.916 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd0af1700 1 --2- 172.21.15.189:0/640650956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bcc0a8840 0x7f6bcc13dc10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f6bcc068350 tx=0x7f6bbc006eb0).ready entity=mon.1 client_cookie=c05a9f8279d4f75f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:59.916 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bcb7fe700 1 -- 172.21.15.189:0/640650956 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc009720 con 0x7f6bcc0a8840 2020-11-20T12:04:59.917 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bcb7fe700 1 -- 172.21.15.189:0/640650956 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bbc009880 con 0x7f6bcc0a8840 2020-11-20T12:04:59.917 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bcb7fe700 1 -- 172.21.15.189:0/640650956 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc009c00 con 0x7f6bcc0a8840 2020-11-20T12:04:59.917 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd1af3700 1 -- 172.21.15.189:0/640650956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bcc0a8840 msgr2=0x7f6bcc13dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.917 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd1af3700 1 --2- 172.21.15.189:0/640650956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bcc0a8840 0x7f6bcc13dc10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f6bcc068350 tx=0x7f6bbc006eb0).stop 2020-11-20T12:04:59.918 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd1af3700 1 -- 172.21.15.189:0/640650956 shutdown_connections 2020-11-20T12:04:59.918 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd1af3700 1 --2- 172.21.15.189:0/640650956 >> v2:172.21.15.47:3300/0 conn(0x7f6bcc1376a0 0x7f6bcc137b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.919 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd1af3700 1 --2- 172.21.15.189:0/640650956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bcc0a8840 0x7f6bcc13dc10 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.919 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd1af3700 1 --2- 172.21.15.189:0/640650956 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6bcc13e100 0x7f6bcc1371b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.920 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd1af3700 1 -- 172.21.15.189:0/640650956 >> 172.21.15.189:0/640650956 conn(0x7f6bcc060170 msgr2=0x7f6bcc060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:59.920 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd1af3700 1 -- 172.21.15.189:0/640650956 shutdown_connections 2020-11-20T12:04:59.920 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.909+0000 7f6bd1af3700 1 -- 172.21.15.189:0/640650956 wait complete. 2020-11-20T12:04:59.924 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bd1af3700 1 Processor -- start 2020-11-20T12:04:59.924 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bd1af3700 1 -- start start 2020-11-20T12:04:59.925 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bd1af3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bcc0a8840 0x7f6bcc13d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.925 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bd1af3700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6bcc1376a0 0x7f6bcc0ab9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.925 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bd1af3700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6bcc13e100 0x7f6bcc149da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.925 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bd1af3700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc13df40 con 0x7f6bcc1376a0 2020-11-20T12:04:59.926 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bd1af3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc0a3870 con 0x7f6bcc0a8840 2020-11-20T12:04:59.926 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bd1af3700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc13da00 con 0x7f6bcc13e100 2020-11-20T12:04:59.926 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bd0af1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bcc0a8840 0x7f6bcc13d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.926 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bd0af1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bcc0a8840 0x7f6bcc13d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48492/0 (socket says 172.21.15.189:48492) 2020-11-20T12:04:59.926 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bd0af1700 1 -- 172.21.15.189:0/3150106912 learned_addr learned my addr 172.21.15.189:0/3150106912 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:04:59.927 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bd12f2700 1 --2- 172.21.15.189:0/3150106912 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6bcc13e100 0x7f6bcc149da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.927 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bcbfff700 1 --2- 172.21.15.189:0/3150106912 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6bcc1376a0 0x7f6bcc0ab9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.927 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bd0af1700 1 -- 172.21.15.189:0/3150106912 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6bcc13e100 msgr2=0x7f6bcc149da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.927 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bd0af1700 1 --2- 172.21.15.189:0/3150106912 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6bcc13e100 0x7f6bcc149da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.928 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bd0af1700 1 -- 172.21.15.189:0/3150106912 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6bcc1376a0 msgr2=0x7f6bcc0ab9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.928 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bd0af1700 1 --2- 172.21.15.189:0/3150106912 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6bcc1376a0 0x7f6bcc0ab9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.928 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bd0af1700 1 -- 172.21.15.189:0/3150106912 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bcc13dca0 con 0x7f6bcc0a8840 2020-11-20T12:04:59.929 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bcbfff700 1 --2- 172.21.15.189:0/3150106912 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6bcc1376a0 0x7f6bcc0ab9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:04:59.929 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bd12f2700 1 --2- 172.21.15.189:0/3150106912 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6bcc13e100 0x7f6bcc149da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:04:59.929 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bd0af1700 1 --2- 172.21.15.189:0/3150106912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bcc0a8840 0x7f6bcc13d3d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f6bcc066940 tx=0x7f6bbc015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:59.930 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bc8ff9700 1 -- 172.21.15.189:0/3150106912 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc011da0 con 0x7f6bcc0a8840 2020-11-20T12:04:59.930 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bc8ff9700 1 -- 172.21.15.189:0/3150106912 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bbc01e460 con 0x7f6bcc0a8840 2020-11-20T12:04:59.930 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bd1af3700 1 -- 172.21.15.189:0/3150106912 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bcc0a3160 con 0x7f6bcc0a8840 2020-11-20T12:04:59.930 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bc8ff9700 1 -- 172.21.15.189:0/3150106912 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc01e7e0 con 0x7f6bcc0a8840 2020-11-20T12:04:59.930 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bd1af3700 1 -- 172.21.15.189:0/3150106912 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bcc0a3440 con 0x7f6bcc0a8840 2020-11-20T12:04:59.931 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.913+0000 7f6bd1af3700 1 -- 172.21.15.189:0/3150106912 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6bcc149a20 con 0x7f6bcc0a8840 2020-11-20T12:04:59.931 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.917+0000 7f6bc8ff9700 1 -- 172.21.15.189:0/3150106912 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f6bbc02f070 con 0x7f6bcc0a8840 2020-11-20T12:04:59.931 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.917+0000 7f6bc8ff9700 1 --2- 172.21.15.189:0/3150106912 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6ba80557c0 0x7f6ba8057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.931 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.917+0000 7f6bc8ff9700 1 -- 172.21.15.189:0/3150106912 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6bbc0708e0 con 0x7f6bcc0a8840 2020-11-20T12:04:59.931 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.917+0000 7f6bcbfff700 1 --2- 172.21.15.189:0/3150106912 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6ba80557c0 0x7f6ba8057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.932 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.917+0000 7f6bc8ff9700 1 --2- 172.21.15.189:0/3150106912 >> [v2:172.21.15.189:6800/11762,v1:172.21.15.189:6801/11762] conn(0x7f6ba805cd10 0x7f6ba805f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:04:59.932 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.917+0000 7f6bc8ff9700 1 -- 172.21.15.189:0/3150106912 --> [v2:172.21.15.189:6800/11762,v1:172.21.15.189:6801/11762] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ba805f790 con 0x7f6ba805cd10 2020-11-20T12:04:59.932 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.917+0000 7f6bc8ff9700 1 -- 172.21.15.189:0/3150106912 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6bbc070ce0 con 0x7f6bcc0a8840 2020-11-20T12:04:59.932 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.917+0000 7f6bd12f2700 1 --2- 172.21.15.189:0/3150106912 >> [v2:172.21.15.189:6800/11762,v1:172.21.15.189:6801/11762] conn(0x7f6ba805cd10 0x7f6ba805f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:04:59.932 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.917+0000 7f6bcbfff700 1 --2- 172.21.15.189:0/3150106912 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6ba80557c0 0x7f6ba8057c80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f6bc4006f50 tx=0x7f6bc4008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:59.933 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.917+0000 7f6bd12f2700 1 --2- 172.21.15.189:0/3150106912 >> [v2:172.21.15.189:6800/11762,v1:172.21.15.189:6801/11762] conn(0x7f6ba805cd10 0x7f6ba805f130 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:04:59.933 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.921+0000 7f6bc8ff9700 1 -- 172.21.15.189:0/3150106912 <== osd.3 v2:172.21.15.189:6800/11762 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f6ba805f790 con 0x7f6ba805cd10 2020-11-20T12:04:59.938 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.937+0000 7f6bd1af3700 1 -- 172.21.15.189:0/3150106912 --> [v2:172.21.15.189:6800/11762,v1:172.21.15.189:6801/11762] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6bcc149ba0 con 0x7f6ba805cd10 2020-11-20T12:04:59.940 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.937+0000 7f6bc8ff9700 1 -- 172.21.15.189:0/3150106912 <== osd.3 v2:172.21.15.189:6800/11762 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6bcc149ba0 con 0x7f6ba805cd10 2020-11-20T12:04:59.940 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.937+0000 7f6bd1af3700 1 -- 172.21.15.189:0/3150106912 >> [v2:172.21.15.189:6800/11762,v1:172.21.15.189:6801/11762] conn(0x7f6ba805cd10 msgr2=0x7f6ba805f130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.940 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.937+0000 7f6bd1af3700 1 --2- 172.21.15.189:0/3150106912 >> [v2:172.21.15.189:6800/11762,v1:172.21.15.189:6801/11762] conn(0x7f6ba805cd10 0x7f6ba805f130 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:04:59.940 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.937+0000 7f6bd1af3700 1 -- 172.21.15.189:0/3150106912 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6ba80557c0 msgr2=0x7f6ba8057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.941 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.937+0000 7f6bd1af3700 1 --2- 172.21.15.189:0/3150106912 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6ba80557c0 0x7f6ba8057c80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f6bc4006f50 tx=0x7f6bc4008040).stop 2020-11-20T12:04:59.941 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.937+0000 7f6bd1af3700 1 -- 172.21.15.189:0/3150106912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bcc0a8840 msgr2=0x7f6bcc13d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:04:59.941 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.937+0000 7f6bd1af3700 1 --2- 172.21.15.189:0/3150106912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bcc0a8840 0x7f6bcc13d3d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f6bcc066940 tx=0x7f6bbc015000).stop 2020-11-20T12:04:59.941 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.937+0000 7f6bd12f2700 1 -- 172.21.15.189:0/3150106912 reap_dead start 2020-11-20T12:04:59.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.941+0000 7f6bd1af3700 1 -- 172.21.15.189:0/3150106912 shutdown_connections 2020-11-20T12:04:59.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.941+0000 7f6bd1af3700 1 -- 172.21.15.189:0/3150106912 >> 172.21.15.189:0/3150106912 conn(0x7f6bcc060170 msgr2=0x7f6bcc0a70e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:04:59.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.941+0000 7f6bd1af3700 1 -- 172.21.15.189:0/3150106912 shutdown_connections 2020-11-20T12:04:59.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:04:59.941+0000 7f6bd1af3700 1 -- 172.21.15.189:0/3150106912 wait complete. 2020-11-20T12:04:59.953 INFO:teuthology.orchestra.run.smithi189.stdout:55834574853 2020-11-20T12:04:59.953 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2020-11-20T12:05:00.078 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.077+0000 7fd51ffff700 1 Processor -- start 2020-11-20T12:05:00.079 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.077+0000 7fd51ffff700 1 -- start start 2020-11-20T12:05:00.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.077+0000 7fd51ffff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd5200623c0 0x7fd5200677f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.077+0000 7fd51ffff700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd52013aac0 0x7fd52013af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.077+0000 7fd51ffff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd52013b430 con 0x7fd5201340e0 2020-11-20T12:05:00.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.077+0000 7fd51ffff700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5200a8e60 con 0x7fd52013aac0 2020-11-20T12:05:00.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.077+0000 7fd51ffff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5200a9c60 con 0x7fd5200623c0 2020-11-20T12:05:00.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.077+0000 7fd51e7fc700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd52013aac0 0x7fd52013af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.077+0000 7fd51effd700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd5200623c0 0x7fd5200677f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.077+0000 7fd51f7fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fd5201340e0 0x7fd520134590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.189:44792/0 (socket says 172.21.15.189:44792) 2020-11-20T12:05:00.082 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.077+0000 7fd51e7fc700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd52013aac0 0x7fd52013af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.189:54726/0 (socket says 172.21.15.189:54726) 2020-11-20T12:05:00.082 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.077+0000 7fd51effd700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd5200623c0 0x7fd5200677f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.189:41538/0 (socket says 172.21.15.189:41538) 2020-11-20T12:05:00.082 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.077+0000 7fd51f7fe700 1 -- 172.21.15.189:0/586408755 learned_addr learned my addr 172.21.15.189:0/586408755 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:00.084 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.081+0000 7fd51e7fc700 1 -- 172.21.15.189:0/586408755 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd5200623c0 msgr2=0x7fd5200677f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:00.084 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.081+0000 7fd51e7fc700 1 --2- 172.21.15.189:0/586408755 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd5200623c0 0x7fd5200677f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.085 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.081+0000 7fd51e7fc700 1 -- 172.21.15.189:0/586408755 >> v1:172.21.15.47:6789/0 conn(0x7fd5201340e0 legacy=0x7fd520134590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.085 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.081+0000 7fd51e7fc700 1 -- 172.21.15.189:0/586408755 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd520067e90 con 0x7fd52013aac0 2020-11-20T12:05:00.085 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.081+0000 7fd51dffb700 1 -- 172.21.15.189:0/586408755 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2150048514 0 0) 0x7fd52013b430 con 0x7fd5201340e0 2020-11-20T12:05:00.085 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.081+0000 7fd51e7fc700 1 --2- 172.21.15.189:0/586408755 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd52013aac0 0x7fd52013af40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fd51000b9a0 tx=0x7fd510006df0).ready entity=mon.0 client_cookie=7734e5e406efe26d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:00.086 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.081+0000 7fd51dffb700 1 -- 172.21.15.189:0/586408755 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5100097b0 con 0x7fd52013aac0 2020-11-20T12:05:00.086 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.081+0000 7fd51dffb700 1 -- 172.21.15.189:0/586408755 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd510009910 con 0x7fd52013aac0 2020-11-20T12:05:00.086 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.081+0000 7fd51dffb700 1 -- 172.21.15.189:0/586408755 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd510009c90 con 0x7fd52013aac0 2020-11-20T12:05:00.086 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.081+0000 7fd51ffff700 1 -- 172.21.15.189:0/586408755 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd52013aac0 msgr2=0x7fd52013af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.086 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.081+0000 7fd51ffff700 1 --2- 172.21.15.189:0/586408755 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd52013aac0 0x7fd52013af40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fd51000b9a0 tx=0x7fd510006df0).stop 2020-11-20T12:05:00.087 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.081+0000 7fd51ffff700 1 -- 172.21.15.189:0/586408755 shutdown_connections 2020-11-20T12:05:00.087 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.081+0000 7fd51ffff700 1 --2- 172.21.15.189:0/586408755 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd5200623c0 0x7fd5200677f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.087 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.081+0000 7fd51ffff700 1 --2- 172.21.15.189:0/586408755 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd52013aac0 0x7fd52013af40 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.087 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.081+0000 7fd51ffff700 1 -- 172.21.15.189:0/586408755 >> 172.21.15.189:0/586408755 conn(0x7fd520065fe0 msgr2=0x7fd520061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:00.087 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.081+0000 7fd51ffff700 1 -- 172.21.15.189:0/586408755 shutdown_connections 2020-11-20T12:05:00.088 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.081+0000 7fd51ffff700 1 -- 172.21.15.189:0/586408755 wait complete. 2020-11-20T12:05:00.088 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.081+0000 7fd51ffff700 1 Processor -- start 2020-11-20T12:05:00.088 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.081+0000 7fd51ffff700 1 -- start start 2020-11-20T12:05:00.088 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd51ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5200623c0 0x7fd5200a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.088 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd51ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5201340e0 0x7fd5200a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.089 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd51ffff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd52013aac0 0x7fd520130ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.089 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd51ffff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5200a2340 con 0x7fd5200623c0 2020-11-20T12:05:00.089 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd51ffff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5200a2aa0 con 0x7fd5201340e0 2020-11-20T12:05:00.089 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd51ffff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5200a2560 con 0x7fd52013aac0 2020-11-20T12:05:00.089 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd51e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5201340e0 0x7fd5200a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.090 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd51e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5201340e0 0x7fd5200a1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48510/0 (socket says 172.21.15.189:48510) 2020-11-20T12:05:00.090 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd51e7fc700 1 -- 172.21.15.189:0/3706838891 learned_addr learned my addr 172.21.15.189:0/3706838891 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:00.090 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd51effd700 1 --2- 172.21.15.189:0/3706838891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5200623c0 0x7fd5200a0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.092 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd51f7fe700 1 --2- 172.21.15.189:0/3706838891 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd52013aac0 0x7fd520130ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.093 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd51e7fc700 1 -- 172.21.15.189:0/3706838891 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd52013aac0 msgr2=0x7fd520130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.093 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd51e7fc700 1 --2- 172.21.15.189:0/3706838891 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd52013aac0 0x7fd520130ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.093 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd51e7fc700 1 -- 172.21.15.189:0/3706838891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5200623c0 msgr2=0x7fd5200a0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.093 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd51e7fc700 1 --2- 172.21.15.189:0/3706838891 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd5200623c0 0x7fd5200a0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.094 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd51e7fc700 1 -- 172.21.15.189:0/3706838891 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5200a2800 con 0x7fd5201340e0 2020-11-20T12:05:00.095 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd51e7fc700 1 --2- 172.21.15.189:0/3706838891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5201340e0 0x7fd5200a1e50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fd510006e30 tx=0x7fd510015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:00.095 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd506ffd700 1 -- 172.21.15.189:0/3706838891 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd510011c80 con 0x7fd5201340e0 2020-11-20T12:05:00.095 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd506ffd700 1 -- 172.21.15.189:0/3706838891 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd510011de0 con 0x7fd5201340e0 2020-11-20T12:05:00.095 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd506ffd700 1 -- 172.21.15.189:0/3706838891 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51001e670 con 0x7fd5201340e0 2020-11-20T12:05:00.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd51ffff700 1 -- 172.21.15.189:0/3706838891 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5201313d0 con 0x7fd5201340e0 2020-11-20T12:05:00.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd51ffff700 1 -- 172.21.15.189:0/3706838891 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5200a5610 con 0x7fd5201340e0 2020-11-20T12:05:00.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd51ffff700 1 -- 172.21.15.189:0/3706838891 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd5200663c0 con 0x7fd5201340e0 2020-11-20T12:05:00.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd506ffd700 1 -- 172.21.15.189:0/3706838891 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fd510025030 con 0x7fd5201340e0 2020-11-20T12:05:00.097 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd506ffd700 1 --2- 172.21.15.189:0/3706838891 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd4fc0557c0 0x7fd4fc057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.097 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd506ffd700 1 -- 172.21.15.189:0/3706838891 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd51006fcd0 con 0x7fd5201340e0 2020-11-20T12:05:00.097 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.085+0000 7fd51effd700 1 --2- 172.21.15.189:0/3706838891 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd4fc0557c0 0x7fd4fc057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.097 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.089+0000 7fd506ffd700 1 --2- 172.21.15.189:0/3706838891 >> [v2:172.21.15.47:6824/12114,v1:172.21.15.47:6825/12114] conn(0x7fd4fc05cd10 0x7fd4fc05f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.097 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.089+0000 7fd506ffd700 1 -- 172.21.15.189:0/3706838891 --> [v2:172.21.15.47:6824/12114,v1:172.21.15.47:6825/12114] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4fc05f790 con 0x7fd4fc05cd10 2020-11-20T12:05:00.098 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.089+0000 7fd506ffd700 1 -- 172.21.15.189:0/3706838891 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd510072340 con 0x7fd5201340e0 2020-11-20T12:05:00.098 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.089+0000 7fd51effd700 1 --2- 172.21.15.189:0/3706838891 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd4fc0557c0 0x7fd4fc057c80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fd514006fd0 tx=0x7fd514006df0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:00.098 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.089+0000 7fd51f7fe700 1 --2- 172.21.15.189:0/3706838891 >> [v2:172.21.15.47:6824/12114,v1:172.21.15.47:6825/12114] conn(0x7fd4fc05cd10 0x7fd4fc05f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.098 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.089+0000 7fd51f7fe700 1 --2- 172.21.15.189:0/3706838891 >> [v2:172.21.15.47:6824/12114,v1:172.21.15.47:6825/12114] conn(0x7fd4fc05cd10 0x7fd4fc05f130 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:00.098 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.089+0000 7fd506ffd700 1 -- 172.21.15.189:0/3706838891 <== osd.4 v2:172.21.15.47:6824/12114 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7fd4fc05f790 con 0x7fd4fc05cd10 2020-11-20T12:05:00.110 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.109+0000 7fd51ffff700 1 -- 172.21.15.189:0/3706838891 --> [v2:172.21.15.47:6824/12114,v1:172.21.15.47:6825/12114] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd520133370 con 0x7fd4fc05cd10 2020-11-20T12:05:00.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.109+0000 7fd506ffd700 1 -- 172.21.15.189:0/3706838891 <== osd.4 v2:172.21.15.47:6824/12114 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd520133370 con 0x7fd4fc05cd10 2020-11-20T12:05:00.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.109+0000 7fd51ffff700 1 -- 172.21.15.189:0/3706838891 >> [v2:172.21.15.47:6824/12114,v1:172.21.15.47:6825/12114] conn(0x7fd4fc05cd10 msgr2=0x7fd4fc05f130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.109+0000 7fd51ffff700 1 --2- 172.21.15.189:0/3706838891 >> [v2:172.21.15.47:6824/12114,v1:172.21.15.47:6825/12114] conn(0x7fd4fc05cd10 0x7fd4fc05f130 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.113+0000 7fd51ffff700 1 -- 172.21.15.189:0/3706838891 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd4fc0557c0 msgr2=0x7fd4fc057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.113+0000 7fd51ffff700 1 --2- 172.21.15.189:0/3706838891 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd4fc0557c0 0x7fd4fc057c80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fd514006fd0 tx=0x7fd514006df0).stop 2020-11-20T12:05:00.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.113+0000 7fd51ffff700 1 -- 172.21.15.189:0/3706838891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5201340e0 msgr2=0x7fd5200a1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.113+0000 7fd51ffff700 1 --2- 172.21.15.189:0/3706838891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5201340e0 0x7fd5200a1e50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fd510006e30 tx=0x7fd510015000).stop 2020-11-20T12:05:00.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.113+0000 7fd51f7fe700 1 -- 172.21.15.189:0/3706838891 reap_dead start 2020-11-20T12:05:00.114 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.113+0000 7fd51ffff700 1 -- 172.21.15.189:0/3706838891 shutdown_connections 2020-11-20T12:05:00.114 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.113+0000 7fd51ffff700 1 -- 172.21.15.189:0/3706838891 >> 172.21.15.189:0/3706838891 conn(0x7fd520065fe0 msgr2=0x7fd520134590 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:00.114 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.113+0000 7fd51ffff700 1 -- 172.21.15.189:0/3706838891 shutdown_connections 2020-11-20T12:05:00.114 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.113+0000 7fd51ffff700 1 -- 172.21.15.189:0/3706838891 wait complete. 2020-11-20T12:05:00.127 INFO:teuthology.orchestra.run.smithi189.stdout:55834574854 2020-11-20T12:05:00.128 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2020-11-20T12:05:00.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.261+0000 7f9ad6eb1700 1 Processor -- start 2020-11-20T12:05:00.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.261+0000 7f9ad6eb1700 1 -- start start 2020-11-20T12:05:00.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.265+0000 7f9ad6eb1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9ad00a9e20 0x7f9ad00a83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.265+0000 7f9ad6eb1700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f9ad0068020 0x7f9ad01339f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.265+0000 7f9ad6eb1700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad00663e0 con 0x7f9ad00a9290 2020-11-20T12:05:00.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.265+0000 7f9ad6eb1700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ad0067b80 con 0x7f9ad0068020 2020-11-20T12:05:00.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.265+0000 7f9ad6eb1700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ad00a9970 con 0x7f9ad00a9e20 2020-11-20T12:05:00.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.265+0000 7f9ad4c4d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9ad00a9e20 0x7f9ad00a83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.265+0000 7f9ad4c4d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9ad00a9e20 0x7f9ad00a83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48518/0 (socket says 172.21.15.189:48518) 2020-11-20T12:05:00.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.265+0000 7f9ad4c4d700 1 -- 172.21.15.189:0/1759739929 learned_addr learned my addr 172.21.15.189:0/1759739929 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:00.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.265+0000 7f9acffff700 1 --2- 172.21.15.189:0/1759739929 >> v2:172.21.15.47:3300/0 conn(0x7f9ad0068020 0x7f9ad01339f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.265+0000 7f9ad4c4d700 1 -- 172.21.15.189:0/1759739929 >> v2:172.21.15.47:3300/0 conn(0x7f9ad0068020 msgr2=0x7f9ad01339f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:00.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.265+0000 7f9ad4c4d700 1 --2- 172.21.15.189:0/1759739929 >> v2:172.21.15.47:3300/0 conn(0x7f9ad0068020 0x7f9ad01339f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.265+0000 7f9ad4c4d700 1 -- 172.21.15.189:0/1759739929 >> v1:172.21.15.47:6789/0 conn(0x7f9ad00a9290 legacy=0x7f9ad0133ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.265+0000 7f9ad4c4d700 1 -- 172.21.15.189:0/1759739929 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad0066680 con 0x7f9ad00a9e20 2020-11-20T12:05:00.269 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.265+0000 7f9acffff700 1 --2- 172.21.15.189:0/1759739929 >> v2:172.21.15.47:3300/0 conn(0x7f9ad0068020 0x7f9ad01339f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:00.269 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.265+0000 7f9ad4c4d700 1 --2- 172.21.15.189:0/1759739929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad00a9e20 0x7f9ad00a83f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f9ad00ad1b0 tx=0x7f9ac0006eb0).ready entity=mon.1 client_cookie=d573de157e269e27 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:00.269 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.265+0000 7f9acf7fe700 1 -- 172.21.15.189:0/1759739929 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac00098d0 con 0x7f9ad00a9e20 2020-11-20T12:05:00.269 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.265+0000 7f9acf7fe700 1 -- 172.21.15.189:0/1759739929 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ac0009a30 con 0x7f9ad00a9e20 2020-11-20T12:05:00.269 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.265+0000 7f9acf7fe700 1 -- 172.21.15.189:0/1759739929 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac0009db0 con 0x7f9ad00a9e20 2020-11-20T12:05:00.273 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.265+0000 7f9ad6eb1700 1 -- 172.21.15.189:0/1759739929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad00a9e20 msgr2=0x7f9ad00a83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.273 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.265+0000 7f9ad6eb1700 1 --2- 172.21.15.189:0/1759739929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad00a9e20 0x7f9ad00a83f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f9ad00ad1b0 tx=0x7f9ac0006eb0).stop 2020-11-20T12:05:00.273 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.265+0000 7f9ad6eb1700 1 -- 172.21.15.189:0/1759739929 shutdown_connections 2020-11-20T12:05:00.273 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.265+0000 7f9ad6eb1700 1 --2- 172.21.15.189:0/1759739929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad00a9e20 0x7f9ad00a83f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.274 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.265+0000 7f9ad6eb1700 1 --2- 172.21.15.189:0/1759739929 >> v2:172.21.15.47:3300/0 conn(0x7f9ad0068020 0x7f9ad01339f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.274 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.265+0000 7f9ad6eb1700 1 -- 172.21.15.189:0/1759739929 >> 172.21.15.189:0/1759739929 conn(0x7f9ad00627f0 msgr2=0x7f9ad0065870 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:00.274 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.265+0000 7f9ad6eb1700 1 -- 172.21.15.189:0/1759739929 shutdown_connections 2020-11-20T12:05:00.274 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.265+0000 7f9ad6eb1700 1 -- 172.21.15.189:0/1759739929 wait complete. 2020-11-20T12:05:00.274 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.269+0000 7f9ad6eb1700 1 Processor -- start 2020-11-20T12:05:00.275 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.269+0000 7f9ad6eb1700 1 -- start start 2020-11-20T12:05:00.275 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.269+0000 7f9ad6eb1700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9ad0068020 0x7f9ad00a5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.275 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.269+0000 7f9ad6eb1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ad00a9290 0x7f9ad009c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.275 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.269+0000 7f9ad6eb1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad00a9e20 0x7f9ad0149c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.275 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.269+0000 7f9ad6eb1700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad00a0720 con 0x7f9ad00a9290 2020-11-20T12:05:00.276 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.269+0000 7f9ad6eb1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad00a6480 con 0x7f9ad00a9e20 2020-11-20T12:05:00.276 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.269+0000 7f9ad6eb1700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad01379e0 con 0x7f9ad0068020 2020-11-20T12:05:00.276 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.269+0000 7f9ad544e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad00a9e20 0x7f9ad0149c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.276 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.269+0000 7f9ad544e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad00a9e20 0x7f9ad0149c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48526/0 (socket says 172.21.15.189:48526) 2020-11-20T12:05:00.276 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.269+0000 7f9ad544e700 1 -- 172.21.15.189:0/3264817787 learned_addr learned my addr 172.21.15.189:0/3264817787 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:00.276 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.269+0000 7f9ad4c4d700 1 --2- 172.21.15.189:0/3264817787 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9ad0068020 0x7f9ad00a5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.277 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.269+0000 7f9acffff700 1 --2- 172.21.15.189:0/3264817787 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ad00a9290 0x7f9ad009c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.277 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.269+0000 7f9ad544e700 1 -- 172.21.15.189:0/3264817787 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9ad0068020 msgr2=0x7f9ad00a5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.277 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.269+0000 7f9ad544e700 1 --2- 172.21.15.189:0/3264817787 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9ad0068020 0x7f9ad00a5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.277 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.269+0000 7f9ad544e700 1 -- 172.21.15.189:0/3264817787 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ad00a9290 msgr2=0x7f9ad009c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.277 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.269+0000 7f9ad544e700 1 --2- 172.21.15.189:0/3264817787 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9ad00a9290 0x7f9ad009c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.278 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.269+0000 7f9ad544e700 1 -- 172.21.15.189:0/3264817787 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad00a49a0 con 0x7f9ad00a9e20 2020-11-20T12:05:00.278 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.269+0000 7f9ad544e700 1 --2- 172.21.15.189:0/3264817787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad00a9e20 0x7f9ad0149c90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f9ac800ca90 tx=0x7f9ac8007f10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:00.278 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.269+0000 7f9accff9700 1 -- 172.21.15.189:0/3264817787 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac800ce70 con 0x7f9ad00a9e20 2020-11-20T12:05:00.278 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.269+0000 7f9accff9700 1 -- 172.21.15.189:0/3264817787 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ac800a730 con 0x7f9ad00a9e20 2020-11-20T12:05:00.278 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.269+0000 7f9ad6eb1700 1 -- 172.21.15.189:0/3264817787 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad00a4ca0 con 0x7f9ad00a9e20 2020-11-20T12:05:00.279 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.269+0000 7f9accff9700 1 -- 172.21.15.189:0/3264817787 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac800aab0 con 0x7f9ad00a9e20 2020-11-20T12:05:00.279 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.269+0000 7f9ad6eb1700 1 -- 172.21.15.189:0/3264817787 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad009ca40 con 0x7f9ad00a9e20 2020-11-20T12:05:00.279 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.269+0000 7f9ad6eb1700 1 -- 172.21.15.189:0/3264817787 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9ad0044f60 con 0x7f9ad00a9e20 2020-11-20T12:05:00.279 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.273+0000 7f9accff9700 1 -- 172.21.15.189:0/3264817787 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f9ac8010070 con 0x7f9ad00a9e20 2020-11-20T12:05:00.279 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.273+0000 7f9accff9700 1 --2- 172.21.15.189:0/3264817787 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9aa80557c0 0x7f9aa8057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.280 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.273+0000 7f9accff9700 1 -- 172.21.15.189:0/3264817787 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9ac806e1b0 con 0x7f9ad00a9e20 2020-11-20T12:05:00.280 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.273+0000 7f9accff9700 1 --2- 172.21.15.189:0/3264817787 >> [v2:172.21.15.47:6800/12111,v1:172.21.15.47:6801/12111] conn(0x7f9aa805cd10 0x7f9aa805f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.280 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.273+0000 7f9accff9700 1 -- 172.21.15.189:0/3264817787 --> [v2:172.21.15.47:6800/12111,v1:172.21.15.47:6801/12111] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9aa805f790 con 0x7f9aa805cd10 2020-11-20T12:05:00.280 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.273+0000 7f9accff9700 1 -- 172.21.15.189:0/3264817787 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ac8070820 con 0x7f9ad00a9e20 2020-11-20T12:05:00.280 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.273+0000 7f9ad4c4d700 1 --2- 172.21.15.189:0/3264817787 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9aa80557c0 0x7f9aa8057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.281 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.273+0000 7f9acffff700 1 --2- 172.21.15.189:0/3264817787 >> [v2:172.21.15.47:6800/12111,v1:172.21.15.47:6801/12111] conn(0x7f9aa805cd10 0x7f9aa805f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.282 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.273+0000 7f9ad4c4d700 1 --2- 172.21.15.189:0/3264817787 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9aa80557c0 0x7f9aa8057c80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f9ac0006ef0 tx=0x7f9ac0015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:00.282 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.273+0000 7f9acffff700 1 --2- 172.21.15.189:0/3264817787 >> [v2:172.21.15.47:6800/12111,v1:172.21.15.47:6801/12111] conn(0x7f9aa805cd10 0x7f9aa805f130 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:00.282 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.273+0000 7f9accff9700 1 -- 172.21.15.189:0/3264817787 <== osd.5 v2:172.21.15.47:6800/12111 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f9aa805f790 con 0x7f9aa805cd10 2020-11-20T12:05:00.295 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.293+0000 7f9ad6eb1700 1 -- 172.21.15.189:0/3264817787 --> [v2:172.21.15.47:6800/12111,v1:172.21.15.47:6801/12111] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9ad00a4090 con 0x7f9aa805cd10 2020-11-20T12:05:00.296 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.297+0000 7f9accff9700 1 -- 172.21.15.189:0/3264817787 <== osd.5 v2:172.21.15.47:6800/12111 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9ad00a4090 con 0x7f9aa805cd10 2020-11-20T12:05:00.297 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.297+0000 7f9ad6eb1700 1 -- 172.21.15.189:0/3264817787 >> [v2:172.21.15.47:6800/12111,v1:172.21.15.47:6801/12111] conn(0x7f9aa805cd10 msgr2=0x7f9aa805f130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.297 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.297+0000 7f9ad6eb1700 1 --2- 172.21.15.189:0/3264817787 >> [v2:172.21.15.47:6800/12111,v1:172.21.15.47:6801/12111] conn(0x7f9aa805cd10 0x7f9aa805f130 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.297 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.297+0000 7f9ad6eb1700 1 -- 172.21.15.189:0/3264817787 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9aa80557c0 msgr2=0x7f9aa8057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.297 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.297+0000 7f9ad6eb1700 1 --2- 172.21.15.189:0/3264817787 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9aa80557c0 0x7f9aa8057c80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f9ac0006ef0 tx=0x7f9ac0015000).stop 2020-11-20T12:05:00.298 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.297+0000 7f9ad6eb1700 1 -- 172.21.15.189:0/3264817787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad00a9e20 msgr2=0x7f9ad0149c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.298 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.297+0000 7f9ad6eb1700 1 --2- 172.21.15.189:0/3264817787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ad00a9e20 0x7f9ad0149c90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f9ac800ca90 tx=0x7f9ac8007f10).stop 2020-11-20T12:05:00.298 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.297+0000 7f9ad544e700 1 -- 172.21.15.189:0/3264817787 reap_dead start 2020-11-20T12:05:00.298 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.297+0000 7f9ad6eb1700 1 -- 172.21.15.189:0/3264817787 shutdown_connections 2020-11-20T12:05:00.298 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.297+0000 7f9ad6eb1700 1 -- 172.21.15.189:0/3264817787 >> 172.21.15.189:0/3264817787 conn(0x7f9ad00627f0 msgr2=0x7f9ad0133d30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:00.299 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.297+0000 7f9ad6eb1700 1 -- 172.21.15.189:0/3264817787 shutdown_connections 2020-11-20T12:05:00.299 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.297+0000 7f9ad6eb1700 1 -- 172.21.15.189:0/3264817787 wait complete. 2020-11-20T12:05:00.312 INFO:teuthology.orchestra.run.smithi189.stdout:55834574853 2020-11-20T12:05:00.312 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2020-11-20T12:05:00.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.433+0000 7f870c790700 1 Processor -- start 2020-11-20T12:05:00.436 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.433+0000 7f870c790700 1 -- start start 2020-11-20T12:05:00.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.437+0000 7f870c790700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8704067ca0 0x7f870413b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.437+0000 7f870c790700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87040aeb40 con 0x7f87040ae390 2020-11-20T12:05:00.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.437+0000 7f870c790700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87040ae8a0 con 0x7f87040678c0 2020-11-20T12:05:00.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.437+0000 7f870c790700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8704137690 con 0x7f8704067ca0 2020-11-20T12:05:00.439 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.437+0000 7f870a52c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f87040678c0 0x7f87040a8c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50156/0 (socket says 172.21.15.189:50156) 2020-11-20T12:05:00.439 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.437+0000 7f870a52c700 1 -- 172.21.15.189:0/3874180638 learned_addr learned my addr 172.21.15.189:0/3874180638 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:00.439 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.437+0000 7f8709d2b700 1 --2- 172.21.15.189:0/3874180638 >> v2:172.21.15.47:3300/0 conn(0x7f8704067ca0 0x7f870413b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.439 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.437+0000 7f870952a700 1 -- 172.21.15.189:0/3874180638 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 151692729 0 0) 0x7f87040ae8a0 con 0x7f87040678c0 2020-11-20T12:05:00.440 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.437+0000 7f870952a700 1 -- 172.21.15.189:0/3874180638 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86f0003240 con 0x7f87040678c0 2020-11-20T12:05:00.440 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.437+0000 7f870952a700 1 -- 172.21.15.189:0/3874180638 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3514727895 0 0) 0x7f87040aeb40 con 0x7f87040ae390 2020-11-20T12:05:00.440 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.437+0000 7f870952a700 1 -- 172.21.15.189:0/3874180638 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87040ae8a0 con 0x7f87040ae390 2020-11-20T12:05:00.441 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.437+0000 7f8709d2b700 1 -- 172.21.15.189:0/3874180638 >> v1:172.21.15.189:6789/0 conn(0x7f87040678c0 legacy=0x7f87040a8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.441 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.437+0000 7f8709d2b700 1 -- 172.21.15.189:0/3874180638 >> v1:172.21.15.47:6789/0 conn(0x7f87040ae390 legacy=0x7f8704137b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.442 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.437+0000 7f8709d2b700 1 -- 172.21.15.189:0/3874180638 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f870413b820 con 0x7f8704067ca0 2020-11-20T12:05:00.442 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.437+0000 7f8709d2b700 1 --2- 172.21.15.189:0/3874180638 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8704067ca0 0x7f870413b330 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f86f8009c90 tx=0x7f86f80091d0).ready entity=mon.0 client_cookie=c59b2e198775bb0e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:00.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.437+0000 7f870952a700 1 -- 172.21.15.189:0/3874180638 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f800b9c0 con 0x7f8704067ca0 2020-11-20T12:05:00.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.437+0000 7f870952a700 1 -- 172.21.15.189:0/3874180638 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86f800bb20 con 0x7f8704067ca0 2020-11-20T12:05:00.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.437+0000 7f870952a700 1 -- 172.21.15.189:0/3874180638 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f800bea0 con 0x7f8704067ca0 2020-11-20T12:05:00.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.437+0000 7f870c790700 1 -- 172.21.15.189:0/3874180638 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8704067ca0 msgr2=0x7f870413b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.437+0000 7f870c790700 1 --2- 172.21.15.189:0/3874180638 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8704067ca0 0x7f870413b330 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f86f8009c90 tx=0x7f86f80091d0).stop 2020-11-20T12:05:00.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.437+0000 7f870c790700 1 -- 172.21.15.189:0/3874180638 shutdown_connections 2020-11-20T12:05:00.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.437+0000 7f870c790700 1 --2- 172.21.15.189:0/3874180638 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8704067ca0 0x7f870413b330 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.437+0000 7f870c790700 1 -- 172.21.15.189:0/3874180638 >> 172.21.15.189:0/3874180638 conn(0x7f87040625a0 msgr2=0x7f87040adc80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:00.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.437+0000 7f870c790700 1 -- 172.21.15.189:0/3874180638 shutdown_connections 2020-11-20T12:05:00.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.441+0000 7f870c790700 1 -- 172.21.15.189:0/3874180638 wait complete. 2020-11-20T12:05:00.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.441+0000 7f870c790700 1 Processor -- start 2020-11-20T12:05:00.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.441+0000 7f870c790700 1 -- start start 2020-11-20T12:05:00.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.441+0000 7f870c790700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f87040678c0 0x7f87040ace80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.441+0000 7f870c790700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8704067ca0 0x7f8704131f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.441+0000 7f870c790700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87040ae390 0x7f8704132440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.441+0000 7f870c790700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8704132930 con 0x7f8704067ca0 2020-11-20T12:05:00.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.441+0000 7f870c790700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87040a7520 con 0x7f87040ae390 2020-11-20T12:05:00.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.441+0000 7f870c790700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87040a77c0 con 0x7f87040678c0 2020-11-20T12:05:00.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.441+0000 7f870ad2d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87040ae390 0x7f8704132440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.441+0000 7f870ad2d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87040ae390 0x7f8704132440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48544/0 (socket says 172.21.15.189:48544) 2020-11-20T12:05:00.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.441+0000 7f870ad2d700 1 -- 172.21.15.189:0/310684002 learned_addr learned my addr 172.21.15.189:0/310684002 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:00.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.441+0000 7f870a52c700 1 --2- 172.21.15.189:0/310684002 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f87040678c0 0x7f87040ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.441+0000 7f8709d2b700 1 --2- 172.21.15.189:0/310684002 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8704067ca0 0x7f8704131f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.441+0000 7f870ad2d700 1 -- 172.21.15.189:0/310684002 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f87040678c0 msgr2=0x7f87040ace80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.441+0000 7f870ad2d700 1 --2- 172.21.15.189:0/310684002 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f87040678c0 0x7f87040ace80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.441+0000 7f870ad2d700 1 -- 172.21.15.189:0/310684002 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8704067ca0 msgr2=0x7f8704131f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.441+0000 7f870ad2d700 1 --2- 172.21.15.189:0/310684002 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8704067ca0 0x7f8704131f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.441+0000 7f870ad2d700 1 -- 172.21.15.189:0/310684002 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87040a7a60 con 0x7f87040ae390 2020-11-20T12:05:00.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.441+0000 7f870ad2d700 1 --2- 172.21.15.189:0/310684002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87040ae390 0x7f8704132440 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f86fc00c710 tx=0x7f86fc007a50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:00.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.445+0000 7f87027fc700 1 -- 172.21.15.189:0/310684002 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86fc00f470 con 0x7f87040ae390 2020-11-20T12:05:00.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.445+0000 7f87027fc700 1 -- 172.21.15.189:0/310684002 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86fc00f5d0 con 0x7f87040ae390 2020-11-20T12:05:00.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.445+0000 7f870c790700 1 -- 172.21.15.189:0/310684002 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87040a7d00 con 0x7f87040ae390 2020-11-20T12:05:00.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.445+0000 7f87027fc700 1 -- 172.21.15.189:0/310684002 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86fc00f950 con 0x7f87040ae390 2020-11-20T12:05:00.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.445+0000 7f870c790700 1 -- 172.21.15.189:0/310684002 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87040a4c30 con 0x7f87040ae390 2020-11-20T12:05:00.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.445+0000 7f870c790700 1 -- 172.21.15.189:0/310684002 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8704044f60 con 0x7f87040ae390 2020-11-20T12:05:00.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.445+0000 7f87027fc700 1 -- 172.21.15.189:0/310684002 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f86fc019030 con 0x7f87040ae390 2020-11-20T12:05:00.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.445+0000 7f87027fc700 1 --2- 172.21.15.189:0/310684002 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f86dc0557c0 0x7f86dc057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.445+0000 7f870a52c700 1 --2- 172.21.15.189:0/310684002 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f86dc0557c0 0x7f86dc057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.445+0000 7f87027fc700 1 -- 172.21.15.189:0/310684002 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f86fc06d920 con 0x7f87040ae390 2020-11-20T12:05:00.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.445+0000 7f87027fc700 1 --2- 172.21.15.189:0/310684002 >> [v2:172.21.15.47:6816/12113,v1:172.21.15.47:6817/12113] conn(0x7f86dc05cd10 0x7f86dc05f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.445+0000 7f87027fc700 1 -- 172.21.15.189:0/310684002 --> [v2:172.21.15.47:6816/12113,v1:172.21.15.47:6817/12113] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f86dc05f790 con 0x7f86dc05cd10 2020-11-20T12:05:00.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.445+0000 7f87027fc700 1 -- 172.21.15.189:0/310684002 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f86fc072220 con 0x7f87040ae390 2020-11-20T12:05:00.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.449+0000 7f870a52c700 1 --2- 172.21.15.189:0/310684002 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f86dc0557c0 0x7f86dc057c80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f86f40055a0 tx=0x7f86f4009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:00.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.449+0000 7f8709d2b700 1 --2- 172.21.15.189:0/310684002 >> [v2:172.21.15.47:6816/12113,v1:172.21.15.47:6817/12113] conn(0x7f86dc05cd10 0x7f86dc05f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.449+0000 7f8709d2b700 1 --2- 172.21.15.189:0/310684002 >> [v2:172.21.15.47:6816/12113,v1:172.21.15.47:6817/12113] conn(0x7f86dc05cd10 0x7f86dc05f130 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:00.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.449+0000 7f87027fc700 1 -- 172.21.15.189:0/310684002 <== osd.6 v2:172.21.15.47:6816/12113 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f86dc05f790 con 0x7f86dc05cd10 2020-11-20T12:05:00.468 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.469+0000 7f870c790700 1 -- 172.21.15.189:0/310684002 --> [v2:172.21.15.47:6816/12113,v1:172.21.15.47:6817/12113] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f87040a4630 con 0x7f86dc05cd10 2020-11-20T12:05:00.469 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.469+0000 7f87027fc700 1 -- 172.21.15.189:0/310684002 <== osd.6 v2:172.21.15.47:6816/12113 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f87040a4630 con 0x7f86dc05cd10 2020-11-20T12:05:00.470 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.469+0000 7f870c790700 1 -- 172.21.15.189:0/310684002 >> [v2:172.21.15.47:6816/12113,v1:172.21.15.47:6817/12113] conn(0x7f86dc05cd10 msgr2=0x7f86dc05f130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.470 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.469+0000 7f870c790700 1 --2- 172.21.15.189:0/310684002 >> [v2:172.21.15.47:6816/12113,v1:172.21.15.47:6817/12113] conn(0x7f86dc05cd10 0x7f86dc05f130 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.470 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.469+0000 7f870c790700 1 -- 172.21.15.189:0/310684002 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f86dc0557c0 msgr2=0x7f86dc057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.470 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.469+0000 7f870c790700 1 --2- 172.21.15.189:0/310684002 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f86dc0557c0 0x7f86dc057c80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f86f40055a0 tx=0x7f86f4009040).stop 2020-11-20T12:05:00.471 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.469+0000 7f870c790700 1 -- 172.21.15.189:0/310684002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87040ae390 msgr2=0x7f8704132440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.471 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.469+0000 7f870c790700 1 --2- 172.21.15.189:0/310684002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87040ae390 0x7f8704132440 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f86fc00c710 tx=0x7f86fc007a50).stop 2020-11-20T12:05:00.471 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.469+0000 7f870ad2d700 1 -- 172.21.15.189:0/310684002 reap_dead start 2020-11-20T12:05:00.471 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.469+0000 7f870c790700 1 -- 172.21.15.189:0/310684002 shutdown_connections 2020-11-20T12:05:00.471 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.469+0000 7f870c790700 1 -- 172.21.15.189:0/310684002 >> 172.21.15.189:0/310684002 conn(0x7f87040625a0 msgr2=0x7f87040a8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:00.472 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.469+0000 7f870c790700 1 -- 172.21.15.189:0/310684002 shutdown_connections 2020-11-20T12:05:00.472 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.469+0000 7f870c790700 1 -- 172.21.15.189:0/310684002 wait complete. 2020-11-20T12:05:00.483 INFO:teuthology.orchestra.run.smithi189.stdout:55834574853 2020-11-20T12:05:00.484 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2020-11-20T12:05:00.612 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2557ef0700 1 Processor -- start 2020-11-20T12:05:00.612 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2557ef0700 1 -- start start 2020-11-20T12:05:00.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2557ef0700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f255013f480 0x7f255013b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2557ef0700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25500676a0 con 0x7f255013be50 2020-11-20T12:05:00.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2557ef0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2550067c50 con 0x7f25500a9800 2020-11-20T12:05:00.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2557ef0700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2550065ec0 con 0x7f255013f480 2020-11-20T12:05:00.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2555c8c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f25500a9800 0x7f255013ed70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50172/0 (socket says 172.21.15.189:50172) 2020-11-20T12:05:00.615 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2555c8c700 1 -- 172.21.15.189:0/3066743968 learned_addr learned my addr 172.21.15.189:0/3066743968 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:00.615 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f255548b700 1 --2- 172.21.15.189:0/3066743968 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f255013f480 0x7f255013b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.615 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2554c8a700 1 -- 172.21.15.189:0/3066743968 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3391094333 0 0) 0x7f2550067c50 con 0x7f25500a9800 2020-11-20T12:05:00.615 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2554c8a700 1 -- 172.21.15.189:0/3066743968 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f253c003240 con 0x7f25500a9800 2020-11-20T12:05:00.618 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2554c8a700 1 -- 172.21.15.189:0/3066743968 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 67130714 0 0) 0x7f25500676a0 con 0x7f255013be50 2020-11-20T12:05:00.619 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2554c8a700 1 -- 172.21.15.189:0/3066743968 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2550067c50 con 0x7f255013be50 2020-11-20T12:05:00.619 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2554c8a700 1 -- 172.21.15.189:0/3066743968 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2865067318 0 0) 0x7f253c003240 con 0x7f25500a9800 2020-11-20T12:05:00.619 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2554c8a700 1 -- 172.21.15.189:0/3066743968 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f255013f480 msgr2=0x7f255013b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:00.619 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2554c8a700 1 --2- 172.21.15.189:0/3066743968 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f255013f480 0x7f255013b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.620 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2554c8a700 1 -- 172.21.15.189:0/3066743968 >> v1:172.21.15.47:6789/0 conn(0x7f255013be50 legacy=0x7f25501492c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.620 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2554c8a700 1 -- 172.21.15.189:0/3066743968 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2550067160 con 0x7f25500a9800 2020-11-20T12:05:00.620 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2554c8a700 1 -- 172.21.15.189:0/3066743968 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f25400018a0 con 0x7f25500a9800 2020-11-20T12:05:00.620 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2554c8a700 1 -- 172.21.15.189:0/3066743968 >> v1:172.21.15.189:6789/0 conn(0x7f25500a9800 legacy=0x7f255013ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.620 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2554c8a700 1 --2- 172.21.15.189:0/3066743968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f253c0038c0 0x7f253c003cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.621 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2554c8a700 1 --2- 172.21.15.189:0/3066743968 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f255013f480 0x7f253c0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.621 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2554c8a700 1 --2- 172.21.15.189:0/3066743968 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f253c0049d0 0x7f253c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.621 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2554c8a700 1 -- 172.21.15.189:0/3066743968 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253c005340 con 0x7f253c0049d0 2020-11-20T12:05:00.621 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2554c8a700 1 -- 172.21.15.189:0/3066743968 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253c00b730 con 0x7f253c0038c0 2020-11-20T12:05:00.621 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2554c8a700 1 -- 172.21.15.189:0/3066743968 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f253c00b8c0 con 0x7f255013f480 2020-11-20T12:05:00.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2555c8c700 1 --2- 172.21.15.189:0/3066743968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f253c0038c0 0x7f253c003cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f255548b700 1 --2- 172.21.15.189:0/3066743968 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f255013f480 0x7f253c0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f255648d700 1 --2- 172.21.15.189:0/3066743968 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f253c0049d0 0x7f253c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2555c8c700 1 -- 172.21.15.189:0/3066743968 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f255013f480 msgr2=0x7f253c0044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:00.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2555c8c700 1 --2- 172.21.15.189:0/3066743968 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f255013f480 0x7f253c0044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2555c8c700 1 -- 172.21.15.189:0/3066743968 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f253c0049d0 msgr2=0x7f253c004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:00.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2555c8c700 1 --2- 172.21.15.189:0/3066743968 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f253c0049d0 0x7f253c004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2555c8c700 1 -- 172.21.15.189:0/3066743968 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253c003240 con 0x7f253c0038c0 2020-11-20T12:05:00.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f255648d700 1 --2- 172.21.15.189:0/3066743968 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f253c0049d0 0x7f253c004e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:00.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2555c8c700 1 --2- 172.21.15.189:0/3066743968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f253c0038c0 0x7f253c003cc0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f254000c630 tx=0x7f25500672f0).ready entity=mon.1 client_cookie=138f1bd8629e2bcc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:00.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2554c8a700 1 -- 172.21.15.189:0/3066743968 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2540005b80 con 0x7f253c0038c0 2020-11-20T12:05:00.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2554c8a700 1 -- 172.21.15.189:0/3066743968 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2540005ce0 con 0x7f253c0038c0 2020-11-20T12:05:00.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.613+0000 7f2554c8a700 1 -- 172.21.15.189:0/3066743968 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2540007300 con 0x7f253c0038c0 2020-11-20T12:05:00.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.617+0000 7f2557ef0700 1 -- 172.21.15.189:0/3066743968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f253c0038c0 msgr2=0x7f253c003cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.617+0000 7f2557ef0700 1 --2- 172.21.15.189:0/3066743968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f253c0038c0 0x7f253c003cc0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f254000c630 tx=0x7f25500672f0).stop 2020-11-20T12:05:00.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.617+0000 7f255648d700 1 -- 172.21.15.189:0/3066743968 reap_dead start 2020-11-20T12:05:00.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.617+0000 7f2557ef0700 1 -- 172.21.15.189:0/3066743968 shutdown_connections 2020-11-20T12:05:00.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.617+0000 7f2557ef0700 1 -- 172.21.15.189:0/3066743968 >> 172.21.15.189:0/3066743968 conn(0x7f2550061f60 msgr2=0x7f2550062370 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:00.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.617+0000 7f2557ef0700 1 -- 172.21.15.189:0/3066743968 shutdown_connections 2020-11-20T12:05:00.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.617+0000 7f2557ef0700 1 -- 172.21.15.189:0/3066743968 wait complete. 2020-11-20T12:05:00.626 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.617+0000 7f2557ef0700 1 Processor -- start 2020-11-20T12:05:00.626 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.617+0000 7f2557ef0700 1 -- start start 2020-11-20T12:05:00.628 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.621+0000 7f2557ef0700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2550116070 0x7f2550117180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.621+0000 7f2557ef0700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2550117670 0x7f2550114610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.621+0000 7f2557ef0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2550114b00 0x7f2550114f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.621+0000 7f2557ef0700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2550140400 con 0x7f2550117670 2020-11-20T12:05:00.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.621+0000 7f2557ef0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2550115470 con 0x7f2550114b00 2020-11-20T12:05:00.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.621+0000 7f2557ef0700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2550117af0 con 0x7f2550116070 2020-11-20T12:05:00.630 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.621+0000 7f255648d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2550114b00 0x7f2550114f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.630 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.621+0000 7f255648d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2550114b00 0x7f2550114f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48566/0 (socket says 172.21.15.189:48566) 2020-11-20T12:05:00.630 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.621+0000 7f255648d700 1 -- 172.21.15.189:0/3165952356 learned_addr learned my addr 172.21.15.189:0/3165952356 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:00.630 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.621+0000 7f2555c8c700 1 --2- 172.21.15.189:0/3165952356 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2550116070 0x7f2550117180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.631 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.621+0000 7f255548b700 1 --2- 172.21.15.189:0/3165952356 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2550117670 0x7f2550114610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.631 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.621+0000 7f255648d700 1 -- 172.21.15.189:0/3165952356 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2550116070 msgr2=0x7f2550117180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.631 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.621+0000 7f255648d700 1 --2- 172.21.15.189:0/3165952356 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2550116070 0x7f2550117180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.631 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.621+0000 7f255648d700 1 -- 172.21.15.189:0/3165952356 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2550117670 msgr2=0x7f2550114610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.631 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.621+0000 7f255648d700 1 --2- 172.21.15.189:0/3165952356 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2550117670 0x7f2550114610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.632 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.621+0000 7f255648d700 1 -- 172.21.15.189:0/3165952356 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25501405f0 con 0x7f2550114b00 2020-11-20T12:05:00.632 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.621+0000 7f255648d700 1 --2- 172.21.15.189:0/3165952356 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2550114b00 0x7f2550114f80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f2550068850 tx=0x7f2548009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:00.632 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.621+0000 7f254dffb700 1 -- 172.21.15.189:0/3165952356 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2548001530 con 0x7f2550114b00 2020-11-20T12:05:00.632 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.621+0000 7f254dffb700 1 -- 172.21.15.189:0/3165952356 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2548001690 con 0x7f2550114b00 2020-11-20T12:05:00.632 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.621+0000 7f254dffb700 1 -- 172.21.15.189:0/3165952356 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2548001a10 con 0x7f2550114b00 2020-11-20T12:05:00.632 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.621+0000 7f2557ef0700 1 -- 172.21.15.189:0/3165952356 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2550140890 con 0x7f2550114b00 2020-11-20T12:05:00.633 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.621+0000 7f2557ef0700 1 -- 172.21.15.189:0/3165952356 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2550116450 con 0x7f2550114b00 2020-11-20T12:05:00.633 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.621+0000 7f2557ef0700 1 -- 172.21.15.189:0/3165952356 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2550044f60 con 0x7f2550114b00 2020-11-20T12:05:00.633 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.621+0000 7f254dffb700 1 -- 172.21.15.189:0/3165952356 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f2548013420 con 0x7f2550114b00 2020-11-20T12:05:00.633 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.625+0000 7f254dffb700 1 --2- 172.21.15.189:0/3165952356 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f25280557c0 0x7f2528057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.633 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.625+0000 7f2555c8c700 1 --2- 172.21.15.189:0/3165952356 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f25280557c0 0x7f2528057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.634 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.625+0000 7f254dffb700 1 -- 172.21.15.189:0/3165952356 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f254806e310 con 0x7f2550114b00 2020-11-20T12:05:00.634 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.625+0000 7f254dffb700 1 --2- 172.21.15.189:0/3165952356 >> [v2:172.21.15.47:6808/12112,v1:172.21.15.47:6809/12112] conn(0x7f252805cd10 0x7f252805f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.634 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.625+0000 7f254dffb700 1 -- 172.21.15.189:0/3165952356 --> [v2:172.21.15.47:6808/12112,v1:172.21.15.47:6809/12112] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f252805f790 con 0x7f252805cd10 2020-11-20T12:05:00.634 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.625+0000 7f254dffb700 1 -- 172.21.15.189:0/3165952356 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2548070870 con 0x7f2550114b00 2020-11-20T12:05:00.634 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.625+0000 7f2555c8c700 1 --2- 172.21.15.189:0/3165952356 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f25280557c0 0x7f2528057c80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f2540006000 tx=0x7f2540013000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:00.635 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.625+0000 7f255548b700 1 --2- 172.21.15.189:0/3165952356 >> [v2:172.21.15.47:6808/12112,v1:172.21.15.47:6809/12112] conn(0x7f252805cd10 0x7f252805f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.635 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.625+0000 7f255548b700 1 --2- 172.21.15.189:0/3165952356 >> [v2:172.21.15.47:6808/12112,v1:172.21.15.47:6809/12112] conn(0x7f252805cd10 0x7f252805f130 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:00.635 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.629+0000 7f254dffb700 1 -- 172.21.15.189:0/3165952356 <== osd.7 v2:172.21.15.47:6808/12112 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f252805f790 con 0x7f252805cd10 2020-11-20T12:05:00.648 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.649+0000 7f2557ef0700 1 -- 172.21.15.189:0/3165952356 --> [v2:172.21.15.47:6808/12112,v1:172.21.15.47:6809/12112] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2550061380 con 0x7f252805cd10 2020-11-20T12:05:00.649 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.649+0000 7f254dffb700 1 -- 172.21.15.189:0/3165952356 <== osd.7 v2:172.21.15.47:6808/12112 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2550061380 con 0x7f252805cd10 2020-11-20T12:05:00.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.649+0000 7f2557ef0700 1 -- 172.21.15.189:0/3165952356 >> [v2:172.21.15.47:6808/12112,v1:172.21.15.47:6809/12112] conn(0x7f252805cd10 msgr2=0x7f252805f130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.649+0000 7f2557ef0700 1 --2- 172.21.15.189:0/3165952356 >> [v2:172.21.15.47:6808/12112,v1:172.21.15.47:6809/12112] conn(0x7f252805cd10 0x7f252805f130 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.649+0000 7f2557ef0700 1 -- 172.21.15.189:0/3165952356 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f25280557c0 msgr2=0x7f2528057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.649+0000 7f2557ef0700 1 --2- 172.21.15.189:0/3165952356 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f25280557c0 0x7f2528057c80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f2540006000 tx=0x7f2540013000).stop 2020-11-20T12:05:00.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.649+0000 7f2557ef0700 1 -- 172.21.15.189:0/3165952356 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2550114b00 msgr2=0x7f2550114f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.649+0000 7f2557ef0700 1 --2- 172.21.15.189:0/3165952356 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2550114b00 0x7f2550114f80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f2550068850 tx=0x7f2548009040).stop 2020-11-20T12:05:00.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.649+0000 7f255648d700 1 -- 172.21.15.189:0/3165952356 reap_dead start 2020-11-20T12:05:00.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.649+0000 7f2557ef0700 1 -- 172.21.15.189:0/3165952356 shutdown_connections 2020-11-20T12:05:00.652 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.649+0000 7f2557ef0700 1 -- 172.21.15.189:0/3165952356 >> 172.21.15.189:0/3165952356 conn(0x7f2550061f60 msgr2=0x7f25500a8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:00.652 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.649+0000 7f2557ef0700 1 -- 172.21.15.189:0/3165952356 shutdown_connections 2020-11-20T12:05:00.652 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.649+0000 7f2557ef0700 1 -- 172.21.15.189:0/3165952356 wait complete. 2020-11-20T12:05:00.667 INFO:teuthology.orchestra.run.smithi189.stdout:55834574854 2020-11-20T12:05:00.667 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2020-11-20T12:05:00.792 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.789+0000 7f3c6e503700 1 Processor -- start 2020-11-20T12:05:00.792 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.793+0000 7f3c6e503700 1 -- start start 2020-11-20T12:05:00.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.793+0000 7f3c6e503700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3c68066ed0 0x7f3c6813b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.793+0000 7f3c6e503700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c68066800 con 0x7f3c680a8840 2020-11-20T12:05:00.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.793+0000 7f3c6e503700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c68066aa0 con 0x7f3c6813ba60 2020-11-20T12:05:00.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.793+0000 7f3c6e503700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c68067450 con 0x7f3c68066ed0 2020-11-20T12:05:00.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.793+0000 7f3c6dd02700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3c6813ba60 0x7f3c681495e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50194/0 (socket says 172.21.15.189:50194) 2020-11-20T12:05:00.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.793+0000 7f3c6dd02700 1 -- 172.21.15.189:0/3019846663 learned_addr learned my addr 172.21.15.189:0/3019846663 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:00.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.793+0000 7f3c6cd00700 1 --2- 172.21.15.189:0/3019846663 >> v2:172.21.15.189:3300/0 conn(0x7f3c68066ed0 0x7f3c6813b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.793+0000 7f3c6cd00700 1 -- 172.21.15.189:0/3019846663 >> v1:172.21.15.189:6789/0 conn(0x7f3c6813ba60 legacy=0x7f3c681495e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.796 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.793+0000 7f3c6cd00700 1 -- 172.21.15.189:0/3019846663 >> v1:172.21.15.47:6789/0 conn(0x7f3c680a8840 legacy=0x7f3c6813eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.796 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.793+0000 7f3c67fff700 1 -- 172.21.15.189:0/3019846663 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852997343 0 0) 0x7f3c68066aa0 con 0x7f3c6813ba60 2020-11-20T12:05:00.796 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.793+0000 7f3c6cd00700 1 -- 172.21.15.189:0/3019846663 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c680676f0 con 0x7f3c68066ed0 2020-11-20T12:05:00.796 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.793+0000 7f3c6cd00700 1 --2- 172.21.15.189:0/3019846663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c68066ed0 0x7f3c6813b570 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f3c5800b9a0 tx=0x7f3c58006df0).ready entity=mon.1 client_cookie=aabfdf430ab88eb4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:00.796 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.793+0000 7f3c67fff700 1 -- 172.21.15.189:0/3019846663 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c580096b0 con 0x7f3c68066ed0 2020-11-20T12:05:00.797 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.793+0000 7f3c67fff700 1 -- 172.21.15.189:0/3019846663 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c58009810 con 0x7f3c68066ed0 2020-11-20T12:05:00.797 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.793+0000 7f3c67fff700 1 -- 172.21.15.189:0/3019846663 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c58009b90 con 0x7f3c68066ed0 2020-11-20T12:05:00.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.793+0000 7f3c6e503700 1 -- 172.21.15.189:0/3019846663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c68066ed0 msgr2=0x7f3c6813b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.793+0000 7f3c6e503700 1 --2- 172.21.15.189:0/3019846663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c68066ed0 0x7f3c6813b570 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f3c5800b9a0 tx=0x7f3c58006df0).stop 2020-11-20T12:05:00.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.793+0000 7f3c6e503700 1 -- 172.21.15.189:0/3019846663 shutdown_connections 2020-11-20T12:05:00.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.793+0000 7f3c6e503700 1 --2- 172.21.15.189:0/3019846663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c68066ed0 0x7f3c6813b570 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.793+0000 7f3c6e503700 1 -- 172.21.15.189:0/3019846663 >> 172.21.15.189:0/3019846663 conn(0x7f3c68060170 msgr2=0x7f3c68060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:00.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.793+0000 7f3c6e503700 1 -- 172.21.15.189:0/3019846663 shutdown_connections 2020-11-20T12:05:00.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.793+0000 7f3c6e503700 1 -- 172.21.15.189:0/3019846663 wait complete. 2020-11-20T12:05:00.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.797+0000 7f3c6e503700 1 Processor -- start 2020-11-20T12:05:00.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.797+0000 7f3c6e503700 1 -- start start 2020-11-20T12:05:00.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.797+0000 7f3c6e503700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c68066ed0 0x7f3c6813d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.797+0000 7f3c6e503700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c680a8840 0x7f3c68151f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.797+0000 7f3c6e503700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3c6813ba60 0x7f3c680a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.797+0000 7f3c6e503700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c68152400 con 0x7f3c680a8840 2020-11-20T12:05:00.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.797+0000 7f3c6e503700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c68151530 con 0x7f3c68066ed0 2020-11-20T12:05:00.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.797+0000 7f3c6e503700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c681517d0 con 0x7f3c6813ba60 2020-11-20T12:05:00.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.797+0000 7f3c6d501700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c68066ed0 0x7f3c6813d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.797+0000 7f3c6d501700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c68066ed0 0x7f3c6813d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48576/0 (socket says 172.21.15.189:48576) 2020-11-20T12:05:00.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.797+0000 7f3c6d501700 1 -- 172.21.15.189:0/3295781300 learned_addr learned my addr 172.21.15.189:0/3295781300 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:00.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.797+0000 7f3c6cd00700 1 --2- 172.21.15.189:0/3295781300 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c680a8840 0x7f3c68151f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.797+0000 7f3c6dd02700 1 --2- 172.21.15.189:0/3295781300 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3c6813ba60 0x7f3c680a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.801+0000 7f3c6d501700 1 -- 172.21.15.189:0/3295781300 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3c6813ba60 msgr2=0x7f3c680a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.801+0000 7f3c6d501700 1 --2- 172.21.15.189:0/3295781300 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3c6813ba60 0x7f3c680a0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.801+0000 7f3c6d501700 1 -- 172.21.15.189:0/3295781300 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c680a8840 msgr2=0x7f3c68151f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.801+0000 7f3c6d501700 1 --2- 172.21.15.189:0/3295781300 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c680a8840 0x7f3c68151f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.806 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.801+0000 7f3c6d501700 1 -- 172.21.15.189:0/3295781300 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c68151a70 con 0x7f3c68066ed0 2020-11-20T12:05:00.806 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.801+0000 7f3c6d501700 1 --2- 172.21.15.189:0/3295781300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c68066ed0 0x7f3c6813d340 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f3c5c00c740 tx=0x7f3c5c007af0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:00.806 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.801+0000 7f3c657fa700 1 -- 172.21.15.189:0/3295781300 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c00e560 con 0x7f3c68066ed0 2020-11-20T12:05:00.806 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.801+0000 7f3c657fa700 1 -- 172.21.15.189:0/3295781300 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c5c00e6c0 con 0x7f3c68066ed0 2020-11-20T12:05:00.806 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.801+0000 7f3c6e503700 1 -- 172.21.15.189:0/3295781300 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c6813df60 con 0x7f3c68066ed0 2020-11-20T12:05:00.807 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.801+0000 7f3c657fa700 1 -- 172.21.15.189:0/3295781300 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c00ea80 con 0x7f3c68066ed0 2020-11-20T12:05:00.807 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.801+0000 7f3c6e503700 1 -- 172.21.15.189:0/3295781300 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c680a38c0 con 0x7f3c68066ed0 2020-11-20T12:05:00.807 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.801+0000 7f3c6e503700 1 -- 172.21.15.189:0/3295781300 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c6813e0f0 con 0x7f3c68066ed0 2020-11-20T12:05:00.808 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.801+0000 7f3c657fa700 1 -- 172.21.15.189:0/3295781300 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f3c5c019030 con 0x7f3c68066ed0 2020-11-20T12:05:00.808 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.805+0000 7f3c657fa700 1 --2- 172.21.15.189:0/3295781300 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3c400557c0 0x7f3c40057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:00.808 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.805+0000 7f3c657fa700 1 -- 172.21.15.189:0/3295781300 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3c5c06dc90 con 0x7f3c68066ed0 2020-11-20T12:05:00.808 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.805+0000 7f3c6cd00700 1 --2- 172.21.15.189:0/3295781300 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3c400557c0 0x7f3c40057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:00.809 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.805+0000 7f3c6cd00700 1 --2- 172.21.15.189:0/3295781300 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3c400557c0 0x7f3c40057c80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f3c5800b150 tx=0x7f3c58015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:00.811 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.809+0000 7f3c657fa700 1 -- 172.21.15.189:0/3295781300 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3c5c043b30 con 0x7f3c68066ed0 2020-11-20T12:05:00.965 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.965+0000 7f3c6e503700 1 -- 172.21.15.189:0/3295781300 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f3c680a3b90 con 0x7f3c68066ed0 2020-11-20T12:05:00.966 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.965+0000 7f3c657fa700 1 -- 172.21.15.189:0/3295781300 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3c5c047150 con 0x7f3c68066ed0 2020-11-20T12:05:00.967 INFO:teuthology.orchestra.run.smithi189.stdout:55834574852 2020-11-20T12:05:00.968 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.965+0000 7f3c6e503700 1 -- 172.21.15.189:0/3295781300 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3c400557c0 msgr2=0x7f3c40057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.968 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.965+0000 7f3c6e503700 1 --2- 172.21.15.189:0/3295781300 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3c400557c0 0x7f3c40057c80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f3c5800b150 tx=0x7f3c58015000).stop 2020-11-20T12:05:00.968 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.965+0000 7f3c6e503700 1 -- 172.21.15.189:0/3295781300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c68066ed0 msgr2=0x7f3c6813d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:00.968 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.965+0000 7f3c6e503700 1 --2- 172.21.15.189:0/3295781300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c68066ed0 0x7f3c6813d340 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f3c5c00c740 tx=0x7f3c5c007af0).stop 2020-11-20T12:05:00.969 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.965+0000 7f3c6e503700 1 -- 172.21.15.189:0/3295781300 shutdown_connections 2020-11-20T12:05:00.969 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.969+0000 7f3c6e503700 1 --2- 172.21.15.189:0/3295781300 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3c6813ba60 0x7f3c680a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.969 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.969+0000 7f3c6e503700 1 --2- 172.21.15.189:0/3295781300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c68066ed0 0x7f3c6813d340 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.969 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.969+0000 7f3c6e503700 1 --2- 172.21.15.189:0/3295781300 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3c400557c0 0x7f3c40057c80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.970 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.969+0000 7f3c6e503700 1 --2- 172.21.15.189:0/3295781300 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3c680a8840 0x7f3c68151f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:00.970 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.969+0000 7f3c6e503700 1 -- 172.21.15.189:0/3295781300 >> 172.21.15.189:0/3295781300 conn(0x7f3c68060170 msgr2=0x7f3c6813bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:00.970 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.969+0000 7f3c6e503700 1 -- 172.21.15.189:0/3295781300 shutdown_connections 2020-11-20T12:05:00.970 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:00.969+0000 7f3c6e503700 1 -- 172.21.15.189:0/3295781300 wait complete. 2020-11-20T12:05:00.980 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.0 2020-11-20T12:05:01.982 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2020-11-20T12:05:02.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.109+0000 7f0ab2e1c700 1 Processor -- start 2020-11-20T12:05:02.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.109+0000 7f0ab2e1c700 1 -- start start 2020-11-20T12:05:02.115 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.113+0000 7f0ab2e1c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0aac1347f0 0x7f0aac0677f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.115 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.113+0000 7f0ab2e1c700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aac0a9c60 con 0x7f0aac0623c0 2020-11-20T12:05:02.115 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.113+0000 7f0ab2e1c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aac067e90 con 0x7f0aac13b1d0 2020-11-20T12:05:02.116 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.113+0000 7f0ab2e1c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aac0a8e60 con 0x7f0aac1347f0 2020-11-20T12:05:02.116 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.113+0000 7f0ab1619700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0aac13b1d0 0x7f0aac1340e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50206/0 (socket says 172.21.15.189:50206) 2020-11-20T12:05:02.116 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.113+0000 7f0ab261b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0aac1347f0 0x7f0aac0677f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.116 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.113+0000 7f0ab1619700 1 -- 172.21.15.189:0/261117573 learned_addr learned my addr 172.21.15.189:0/261117573 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:02.117 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.113+0000 7f0ab0e18700 1 -- 172.21.15.189:0/261117573 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 251013438 0 0) 0x7f0aac067e90 con 0x7f0aac13b1d0 2020-11-20T12:05:02.117 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.113+0000 7f0ab0e18700 1 -- 172.21.15.189:0/261117573 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a94003240 con 0x7f0aac13b1d0 2020-11-20T12:05:02.118 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.113+0000 7f0ab261b700 1 -- 172.21.15.189:0/261117573 >> v1:172.21.15.189:6789/0 conn(0x7f0aac13b1d0 legacy=0x7f0aac1340e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.118 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.113+0000 7f0ab261b700 1 -- 172.21.15.189:0/261117573 >> v1:172.21.15.47:6789/0 conn(0x7f0aac0623c0 legacy=0x7f0aac13aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.118 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.113+0000 7f0ab261b700 1 -- 172.21.15.189:0/261117573 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aac068130 con 0x7f0aac1347f0 2020-11-20T12:05:02.118 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.113+0000 7f0ab261b700 1 --2- 172.21.15.189:0/261117573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0aac1347f0 0x7f0aac0677f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f0aa4009c90 tx=0x7f0aa40091d0).ready entity=mon.1 client_cookie=e15b43a9d80504ca server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:02.119 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.113+0000 7f0ab0e18700 1 -- 172.21.15.189:0/261117573 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa400ba70 con 0x7f0aac1347f0 2020-11-20T12:05:02.119 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.113+0000 7f0ab0e18700 1 -- 172.21.15.189:0/261117573 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa400bbd0 con 0x7f0aac1347f0 2020-11-20T12:05:02.120 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.113+0000 7f0ab0e18700 1 -- 172.21.15.189:0/261117573 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa4011400 con 0x7f0aac1347f0 2020-11-20T12:05:02.120 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.113+0000 7f0ab2e1c700 1 -- 172.21.15.189:0/261117573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0aac1347f0 msgr2=0x7f0aac0677f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.120 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.113+0000 7f0ab2e1c700 1 --2- 172.21.15.189:0/261117573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0aac1347f0 0x7f0aac0677f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f0aa4009c90 tx=0x7f0aa40091d0).stop 2020-11-20T12:05:02.121 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.113+0000 7f0ab2e1c700 1 -- 172.21.15.189:0/261117573 shutdown_connections 2020-11-20T12:05:02.121 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.113+0000 7f0ab2e1c700 1 --2- 172.21.15.189:0/261117573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0aac1347f0 0x7f0aac0677f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.122 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.113+0000 7f0ab2e1c700 1 -- 172.21.15.189:0/261117573 >> 172.21.15.189:0/261117573 conn(0x7f0aac065fe0 msgr2=0x7f0aac061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:02.122 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.113+0000 7f0ab2e1c700 1 -- 172.21.15.189:0/261117573 shutdown_connections 2020-11-20T12:05:02.122 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.113+0000 7f0ab2e1c700 1 -- 172.21.15.189:0/261117573 wait complete. 2020-11-20T12:05:02.126 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.117+0000 7f0ab2e1c700 1 Processor -- start 2020-11-20T12:05:02.126 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.117+0000 7f0ab2e1c700 1 -- start start 2020-11-20T12:05:02.126 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.117+0000 7f0ab2e1c700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0aac0623c0 0x7f0aac136300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.126 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.117+0000 7f0ab2e1c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0aac1347f0 0x7f0aac132e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.127 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.117+0000 7f0ab2e1c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0aac13b1d0 0x7f0aac138570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.127 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.117+0000 7f0ab2e1c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac1352e0 con 0x7f0aac13b1d0 2020-11-20T12:05:02.127 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.117+0000 7f0ab2e1c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac133340 con 0x7f0aac1347f0 2020-11-20T12:05:02.128 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.117+0000 7f0ab2e1c700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac1335d0 con 0x7f0aac0623c0 2020-11-20T12:05:02.128 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.117+0000 7f0ab1619700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0aac1347f0 0x7f0aac132e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.128 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.117+0000 7f0ab1619700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0aac1347f0 0x7f0aac132e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48592/0 (socket says 172.21.15.189:48592) 2020-11-20T12:05:02.129 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.117+0000 7f0ab1619700 1 -- 172.21.15.189:0/2885827559 learned_addr learned my addr 172.21.15.189:0/2885827559 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:02.129 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.117+0000 7f0ab1e1a700 1 --2- 172.21.15.189:0/2885827559 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0aac0623c0 0x7f0aac136300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.129 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.117+0000 7f0ab261b700 1 --2- 172.21.15.189:0/2885827559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0aac13b1d0 0x7f0aac138570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.129 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.117+0000 7f0ab1619700 1 -- 172.21.15.189:0/2885827559 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0aac0623c0 msgr2=0x7f0aac136300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.130 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.117+0000 7f0ab1619700 1 --2- 172.21.15.189:0/2885827559 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0aac0623c0 0x7f0aac136300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.130 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.117+0000 7f0ab1619700 1 -- 172.21.15.189:0/2885827559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0aac13b1d0 msgr2=0x7f0aac138570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.130 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.117+0000 7f0ab1619700 1 --2- 172.21.15.189:0/2885827559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0aac13b1d0 0x7f0aac138570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.130 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.117+0000 7f0ab1619700 1 -- 172.21.15.189:0/2885827559 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aac127e80 con 0x7f0aac1347f0 2020-11-20T12:05:02.131 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.117+0000 7f0ab1619700 1 --2- 172.21.15.189:0/2885827559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0aac1347f0 0x7f0aac132e50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f0a9c00c740 tx=0x7f0a9c007ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:02.131 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.117+0000 7f0a99ffb700 1 -- 172.21.15.189:0/2885827559 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c0072e0 con 0x7f0aac1347f0 2020-11-20T12:05:02.132 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.117+0000 7f0a99ffb700 1 -- 172.21.15.189:0/2885827559 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a9c007440 con 0x7f0aac1347f0 2020-11-20T12:05:02.132 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.117+0000 7f0a99ffb700 1 -- 172.21.15.189:0/2885827559 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c005800 con 0x7f0aac1347f0 2020-11-20T12:05:02.132 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.117+0000 7f0ab2e1c700 1 -- 172.21.15.189:0/2885827559 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aac136f10 con 0x7f0aac1347f0 2020-11-20T12:05:02.133 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.117+0000 7f0ab2e1c700 1 -- 172.21.15.189:0/2885827559 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aac128320 con 0x7f0aac1347f0 2020-11-20T12:05:02.133 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.117+0000 7f0ab2e1c700 1 -- 172.21.15.189:0/2885827559 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aac128540 con 0x7f0aac1347f0 2020-11-20T12:05:02.134 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.121+0000 7f0a99ffb700 1 -- 172.21.15.189:0/2885827559 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f0a9c010070 con 0x7f0aac1347f0 2020-11-20T12:05:02.134 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.121+0000 7f0a99ffb700 1 --2- 172.21.15.189:0/2885827559 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0a840557c0 0x7f0a84057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.134 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.121+0000 7f0ab1e1a700 1 --2- 172.21.15.189:0/2885827559 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0a840557c0 0x7f0a84057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.135 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.121+0000 7f0a99ffb700 1 -- 172.21.15.189:0/2885827559 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0a9c06dcf0 con 0x7f0aac1347f0 2020-11-20T12:05:02.135 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.121+0000 7f0ab1e1a700 1 --2- 172.21.15.189:0/2885827559 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0a840557c0 0x7f0a84057c80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f0aa8003610 tx=0x7f0aa8000b80).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:02.136 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.125+0000 7f0a99ffb700 1 -- 172.21.15.189:0/2885827559 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0a9c043b90 con 0x7f0aac1347f0 2020-11-20T12:05:02.275 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.273+0000 7f0ab2e1c700 1 -- 172.21.15.189:0/2885827559 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f0aac0663c0 con 0x7f0aac1347f0 2020-11-20T12:05:02.276 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.273+0000 7f0a99ffb700 1 -- 172.21.15.189:0/2885827559 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0a9c0471b0 con 0x7f0aac1347f0 2020-11-20T12:05:02.276 INFO:teuthology.orchestra.run.smithi189.stdout:55834574853 2020-11-20T12:05:02.279 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.277+0000 7f0ab2e1c700 1 -- 172.21.15.189:0/2885827559 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0a840557c0 msgr2=0x7f0a84057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.279 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.277+0000 7f0ab2e1c700 1 --2- 172.21.15.189:0/2885827559 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0a840557c0 0x7f0a84057c80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f0aa8003610 tx=0x7f0aa8000b80).stop 2020-11-20T12:05:02.280 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.277+0000 7f0ab2e1c700 1 -- 172.21.15.189:0/2885827559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0aac1347f0 msgr2=0x7f0aac132e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.280 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.277+0000 7f0ab2e1c700 1 --2- 172.21.15.189:0/2885827559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0aac1347f0 0x7f0aac132e50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f0a9c00c740 tx=0x7f0a9c007ad0).stop 2020-11-20T12:05:02.280 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.277+0000 7f0ab2e1c700 1 -- 172.21.15.189:0/2885827559 shutdown_connections 2020-11-20T12:05:02.280 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.277+0000 7f0ab2e1c700 1 --2- 172.21.15.189:0/2885827559 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0a840557c0 0x7f0a84057c80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.281 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.277+0000 7f0ab2e1c700 1 --2- 172.21.15.189:0/2885827559 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0aac13b1d0 0x7f0aac138570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.281 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.277+0000 7f0ab2e1c700 1 --2- 172.21.15.189:0/2885827559 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0aac0623c0 0x7f0aac136300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.281 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.277+0000 7f0ab2e1c700 1 --2- 172.21.15.189:0/2885827559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0aac1347f0 0x7f0aac132e50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.281 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.277+0000 7f0ab2e1c700 1 -- 172.21.15.189:0/2885827559 >> 172.21.15.189:0/2885827559 conn(0x7f0aac065fe0 msgr2=0x7f0aac05ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:02.281 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.277+0000 7f0ab2e1c700 1 -- 172.21.15.189:0/2885827559 shutdown_connections 2020-11-20T12:05:02.282 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.277+0000 7f0ab2e1c700 1 -- 172.21.15.189:0/2885827559 wait complete. 2020-11-20T12:05:02.297 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.0 2020-11-20T12:05:02.298 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2020-11-20T12:05:02.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b8bf0700 1 Processor -- start 2020-11-20T12:05:02.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b8bf0700 1 -- start start 2020-11-20T12:05:02.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b8bf0700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f55b40aebd0 0x7f55b40aa620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b8bf0700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b40a9680 con 0x7f55b40a74d0 2020-11-20T12:05:02.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b8bf0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b40aae50 con 0x7f55b40aefb0 2020-11-20T12:05:02.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b8bf0700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b40aa1f0 con 0x7f55b40aebd0 2020-11-20T12:05:02.423 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b2ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f55b40aefb0 0x7f55b40a9ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50222/0 (socket says 172.21.15.189:50222) 2020-11-20T12:05:02.423 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b2ffd700 1 -- 172.21.15.189:0/3410322380 learned_addr learned my addr 172.21.15.189:0/3410322380 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:02.424 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b37fe700 1 --2- 172.21.15.189:0/3410322380 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f55b40aebd0 0x7f55b40aa620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.424 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b27fc700 1 -- 172.21.15.189:0/3410322380 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3892122364 0 0) 0x7f55b40aae50 con 0x7f55b40aefb0 2020-11-20T12:05:02.424 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b27fc700 1 -- 172.21.15.189:0/3410322380 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55a0003240 con 0x7f55b40aefb0 2020-11-20T12:05:02.425 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b27fc700 1 -- 172.21.15.189:0/3410322380 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 405050009 0 0) 0x7f55a0003240 con 0x7f55b40aefb0 2020-11-20T12:05:02.425 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b27fc700 1 -- 172.21.15.189:0/3410322380 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f55b40aebd0 msgr2=0x7f55b40aa620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:02.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b27fc700 1 --2- 172.21.15.189:0/3410322380 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f55b40aebd0 0x7f55b40aa620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b27fc700 1 -- 172.21.15.189:0/3410322380 >> v1:172.21.15.47:6789/0 conn(0x7f55b40a74d0 legacy=0x7f55b413be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b27fc700 1 -- 172.21.15.189:0/3410322380 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b413ba10 con 0x7f55b40aefb0 2020-11-20T12:05:02.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b27fc700 1 -- 172.21.15.189:0/3410322380 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f55a80014e0 con 0x7f55b40aefb0 2020-11-20T12:05:02.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b27fc700 1 -- 172.21.15.189:0/3410322380 >> v1:172.21.15.189:6789/0 conn(0x7f55b40aefb0 legacy=0x7f55b40a9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b27fc700 1 --2- 172.21.15.189:0/3410322380 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55a0003730 0x7f55a0003b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b27fc700 1 --2- 172.21.15.189:0/3410322380 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f55b40aebd0 0x7f55a00044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b27fc700 1 --2- 172.21.15.189:0/3410322380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55a00049d0 0x7f55a0004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b27fc700 1 -- 172.21.15.189:0/3410322380 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b40aa1f0 con 0x7f55a0003730 2020-11-20T12:05:02.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b27fc700 1 -- 172.21.15.189:0/3410322380 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a0004140 con 0x7f55a00049d0 2020-11-20T12:05:02.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b27fc700 1 -- 172.21.15.189:0/3410322380 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55a0004300 con 0x7f55b40aebd0 2020-11-20T12:05:02.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b3fff700 1 --2- 172.21.15.189:0/3410322380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55a00049d0 0x7f55a0004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b2ffd700 1 --2- 172.21.15.189:0/3410322380 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f55b40aebd0 0x7f55a00044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b37fe700 1 --2- 172.21.15.189:0/3410322380 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55a0003730 0x7f55a0003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b3fff700 1 -- 172.21.15.189:0/3410322380 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f55b40aebd0 msgr2=0x7f55a00044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:02.430 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b3fff700 1 --2- 172.21.15.189:0/3410322380 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f55b40aebd0 0x7f55a00044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.430 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b3fff700 1 -- 172.21.15.189:0/3410322380 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55a0003730 msgr2=0x7f55a0003b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:02.430 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b3fff700 1 --2- 172.21.15.189:0/3410322380 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55a0003730 0x7f55a0003b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.430 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b3fff700 1 -- 172.21.15.189:0/3410322380 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a0003240 con 0x7f55a00049d0 2020-11-20T12:05:02.431 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b3fff700 1 --2- 172.21.15.189:0/3410322380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55a00049d0 0x7f55a0004e50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f55ac001e40 tx=0x7f55ac007f10).ready entity=mon.1 client_cookie=a0112e3e8fa40af2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:02.431 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b27fc700 1 -- 172.21.15.189:0/3410322380 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac001540 con 0x7f55a00049d0 2020-11-20T12:05:02.431 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b27fc700 1 -- 172.21.15.189:0/3410322380 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ac00cad0 con 0x7f55a00049d0 2020-11-20T12:05:02.431 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.408+0000 7f55b27fc700 1 -- 172.21.15.189:0/3410322380 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac00ce70 con 0x7f55a00049d0 2020-11-20T12:05:02.432 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b8bf0700 1 -- 172.21.15.189:0/3410322380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55a00049d0 msgr2=0x7f55a0004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.432 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b8bf0700 1 --2- 172.21.15.189:0/3410322380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55a00049d0 0x7f55a0004e50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f55ac001e40 tx=0x7f55ac007f10).stop 2020-11-20T12:05:02.432 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b3fff700 1 -- 172.21.15.189:0/3410322380 reap_dead start 2020-11-20T12:05:02.432 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b8bf0700 1 -- 172.21.15.189:0/3410322380 shutdown_connections 2020-11-20T12:05:02.433 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b8bf0700 1 -- 172.21.15.189:0/3410322380 >> 172.21.15.189:0/3410322380 conn(0x7f55b4062080 msgr2=0x7f55b4062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:02.433 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b8bf0700 1 -- 172.21.15.189:0/3410322380 shutdown_connections 2020-11-20T12:05:02.433 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b8bf0700 1 -- 172.21.15.189:0/3410322380 wait complete. 2020-11-20T12:05:02.433 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b8bf0700 1 Processor -- start 2020-11-20T12:05:02.433 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b8bf0700 1 -- start start 2020-11-20T12:05:02.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b8bf0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55a00049d0 0x7f55b40a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b8bf0700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55b40a4330 0x7f55b409c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b8bf0700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f55b414ac00 0x7f55b414b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b8bf0700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b40a2d00 con 0x7f55b40a4330 2020-11-20T12:05:02.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b8bf0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b413af20 con 0x7f55a00049d0 2020-11-20T12:05:02.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b8bf0700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b40a2f30 con 0x7f55b414ac00 2020-11-20T12:05:02.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55a00049d0 0x7f55b40a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55a00049d0 0x7f55b40a3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48610/0 (socket says 172.21.15.189:48610) 2020-11-20T12:05:02.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b37fe700 1 -- 172.21.15.189:0/1420707921 learned_addr learned my addr 172.21.15.189:0/1420707921 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:02.436 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b3fff700 1 --2- 172.21.15.189:0/1420707921 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f55b414ac00 0x7f55b414b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.436 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b2ffd700 1 --2- 172.21.15.189:0/1420707921 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55b40a4330 0x7f55b409c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.436 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b37fe700 1 -- 172.21.15.189:0/1420707921 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f55b414ac00 msgr2=0x7f55b414b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.436 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b37fe700 1 --2- 172.21.15.189:0/1420707921 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f55b414ac00 0x7f55b414b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.436 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b37fe700 1 -- 172.21.15.189:0/1420707921 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55b40a4330 msgr2=0x7f55b409c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.437 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b37fe700 1 --2- 172.21.15.189:0/1420707921 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55b40a4330 0x7f55b409c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.437 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b37fe700 1 -- 172.21.15.189:0/1420707921 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b40a31d0 con 0x7f55a00049d0 2020-11-20T12:05:02.437 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b2ffd700 1 --2- 172.21.15.189:0/1420707921 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55b40a4330 0x7f55b409c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:02.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b3fff700 1 --2- 172.21.15.189:0/1420707921 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f55b414ac00 0x7f55b414b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:02.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f55b37fe700 1 --2- 172.21.15.189:0/1420707921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55a00049d0 0x7f55b40a3670 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f55a4010c50 tx=0x7f55a4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:02.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f559b7fe700 1 -- 172.21.15.189:0/1420707921 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a400ea20 con 0x7f55a00049d0 2020-11-20T12:05:02.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.412+0000 7f559b7fe700 1 -- 172.21.15.189:0/1420707921 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a400eb80 con 0x7f55a00049d0 2020-11-20T12:05:02.439 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.416+0000 7f55b8bf0700 1 -- 172.21.15.189:0/1420707921 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b40a47b0 con 0x7f55a00049d0 2020-11-20T12:05:02.439 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.416+0000 7f559b7fe700 1 -- 172.21.15.189:0/1420707921 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a4014470 con 0x7f55a00049d0 2020-11-20T12:05:02.440 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.416+0000 7f55b8bf0700 1 -- 172.21.15.189:0/1420707921 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b409ca40 con 0x7f55a00049d0 2020-11-20T12:05:02.440 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.416+0000 7f55b8bf0700 1 -- 172.21.15.189:0/1420707921 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55b40a4940 con 0x7f55a00049d0 2020-11-20T12:05:02.440 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.416+0000 7f559b7fe700 1 -- 172.21.15.189:0/1420707921 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f55a4024070 con 0x7f55a00049d0 2020-11-20T12:05:02.440 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.416+0000 7f559b7fe700 1 --2- 172.21.15.189:0/1420707921 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f55880557f0 0x7f5588057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.441 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.416+0000 7f559b7fe700 1 -- 172.21.15.189:0/1420707921 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f55a406efe0 con 0x7f55a00049d0 2020-11-20T12:05:02.441 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.416+0000 7f55b2ffd700 1 --2- 172.21.15.189:0/1420707921 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f55880557f0 0x7f5588057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.441 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.416+0000 7f55b2ffd700 1 --2- 172.21.15.189:0/1420707921 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f55880557f0 0x7f5588057cb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f55a8001010 tx=0x7f55a800a040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:02.442 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.424+0000 7f559b7fe700 1 -- 172.21.15.189:0/1420707921 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f55a4044e80 con 0x7f55a00049d0 2020-11-20T12:05:02.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.576+0000 7f55b8bf0700 1 -- 172.21.15.189:0/1420707921 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f55b40a5d80 con 0x7f55a00049d0 2020-11-20T12:05:02.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.576+0000 7f559b7fe700 1 -- 172.21.15.189:0/1420707921 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f55a4018070 con 0x7f55a00049d0 2020-11-20T12:05:02.580 INFO:teuthology.orchestra.run.smithi189.stdout:55834574853 2020-11-20T12:05:02.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.580+0000 7f55b8bf0700 1 -- 172.21.15.189:0/1420707921 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f55880557f0 msgr2=0x7f5588057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.580+0000 7f55b8bf0700 1 --2- 172.21.15.189:0/1420707921 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f55880557f0 0x7f5588057cb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f55a8001010 tx=0x7f55a800a040).stop 2020-11-20T12:05:02.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.580+0000 7f55b8bf0700 1 -- 172.21.15.189:0/1420707921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55a00049d0 msgr2=0x7f55b40a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.580+0000 7f55b8bf0700 1 --2- 172.21.15.189:0/1420707921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55a00049d0 0x7f55b40a3670 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f55a4010c50 tx=0x7f55a4008040).stop 2020-11-20T12:05:02.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.580+0000 7f55b8bf0700 1 -- 172.21.15.189:0/1420707921 shutdown_connections 2020-11-20T12:05:02.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.580+0000 7f55b8bf0700 1 --2- 172.21.15.189:0/1420707921 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f55b414ac00 0x7f55b414b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.580+0000 7f55b8bf0700 1 --2- 172.21.15.189:0/1420707921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55a00049d0 0x7f55b40a3670 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.580+0000 7f55b8bf0700 1 --2- 172.21.15.189:0/1420707921 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f55880557f0 0x7f5588057cb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.580+0000 7f55b8bf0700 1 --2- 172.21.15.189:0/1420707921 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f55b40a4330 0x7f55b409c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.580+0000 7f55b8bf0700 1 -- 172.21.15.189:0/1420707921 >> 172.21.15.189:0/1420707921 conn(0x7f55b4062080 msgr2=0x7f55b41363b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:02.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.580+0000 7f55b8bf0700 1 -- 172.21.15.189:0/1420707921 shutdown_connections 2020-11-20T12:05:02.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.580+0000 7f55b8bf0700 1 -- 172.21.15.189:0/1420707921 wait complete. 2020-11-20T12:05:02.599 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.1 2020-11-20T12:05:02.600 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2020-11-20T12:05:02.726 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f00a5f53700 1 Processor -- start 2020-11-20T12:05:02.726 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f00a5f53700 1 -- start start 2020-11-20T12:05:02.730 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f00a5f53700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f00a00ad3e0 0x7f00a0066d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.730 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f00a5f53700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f00a013a6e0 0x7f00a013ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.730 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f00a5f53700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00a0067730 con 0x7f00a00ad000 2020-11-20T12:05:02.731 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f00a5f53700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a0066050 con 0x7f00a013a6e0 2020-11-20T12:05:02.731 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f00a5f53700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a0138190 con 0x7f00a00ad3e0 2020-11-20T12:05:02.731 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f00a4f51700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f00a00ad000 0x7f00a013e480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50240/0 (socket says 172.21.15.189:50240) 2020-11-20T12:05:02.731 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f00a4f51700 1 -- 172.21.15.189:0/3234181044 learned_addr learned my addr 172.21.15.189:0/3234181044 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:02.732 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f00a5752700 1 --2- 172.21.15.189:0/3234181044 >> v2:172.21.15.47:3300/0 conn(0x7f00a013a6e0 0x7f00a013ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.732 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f009ffff700 1 --2- 172.21.15.189:0/3234181044 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f00a00ad3e0 0x7f00a0066d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.732 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f009f7fe700 1 -- 172.21.15.189:0/3234181044 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47274940 0 0) 0x7f00a0067730 con 0x7f00a00ad000 2020-11-20T12:05:02.732 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f009f7fe700 1 -- 172.21.15.189:0/3234181044 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0084003240 con 0x7f00a00ad000 2020-11-20T12:05:02.733 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f009f7fe700 1 -- 172.21.15.189:0/3234181044 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1270154903 0 0) 0x7f0084003240 con 0x7f00a00ad000 2020-11-20T12:05:02.733 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f009f7fe700 1 -- 172.21.15.189:0/3234181044 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f00a00ad3e0 msgr2=0x7f00a0066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:02.733 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f009f7fe700 1 --2- 172.21.15.189:0/3234181044 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f00a00ad3e0 0x7f00a0066d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.733 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f009f7fe700 1 -- 172.21.15.189:0/3234181044 >> v2:172.21.15.47:3300/0 conn(0x7f00a013a6e0 msgr2=0x7f00a013ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:02.733 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f009f7fe700 1 --2- 172.21.15.189:0/3234181044 >> v2:172.21.15.47:3300/0 conn(0x7f00a013a6e0 0x7f00a013ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.734 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f009f7fe700 1 -- 172.21.15.189:0/3234181044 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a013dd40 con 0x7f00a00ad000 2020-11-20T12:05:02.734 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f009f7fe700 1 -- 172.21.15.189:0/3234181044 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f00940014e0 con 0x7f00a00ad000 2020-11-20T12:05:02.734 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f009f7fe700 1 -- 172.21.15.189:0/3234181044 >> v1:172.21.15.189:6789/0 conn(0x7f00a00ad000 legacy=0x7f00a013e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.734 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f009f7fe700 1 --2- 172.21.15.189:0/3234181044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0084003730 0x7f0084003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.735 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f009f7fe700 1 --2- 172.21.15.189:0/3234181044 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f00a00ad3e0 0x7f00840044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.735 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f009f7fe700 1 --2- 172.21.15.189:0/3234181044 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00840049d0 0x7f0084004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.735 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f009f7fe700 1 -- 172.21.15.189:0/3234181044 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a0138190 con 0x7f00840049d0 2020-11-20T12:05:02.735 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f009f7fe700 1 -- 172.21.15.189:0/3234181044 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00840041d0 con 0x7f0084003730 2020-11-20T12:05:02.735 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f009f7fe700 1 -- 172.21.15.189:0/3234181044 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f008400b730 con 0x7f00a00ad3e0 2020-11-20T12:05:02.736 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f00a4f51700 1 --2- 172.21.15.189:0/3234181044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0084003730 0x7f0084003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.736 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f00a5752700 1 --2- 172.21.15.189:0/3234181044 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00840049d0 0x7f0084004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.736 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f009ffff700 1 --2- 172.21.15.189:0/3234181044 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f00a00ad3e0 0x7f00840044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.736 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f00a4f51700 1 -- 172.21.15.189:0/3234181044 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f00a00ad3e0 msgr2=0x7f00840044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:02.737 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.724+0000 7f00a4f51700 1 --2- 172.21.15.189:0/3234181044 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f00a00ad3e0 0x7f00840044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.737 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.728+0000 7f00a4f51700 1 -- 172.21.15.189:0/3234181044 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00840049d0 msgr2=0x7f0084004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:02.737 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.728+0000 7f00a4f51700 1 --2- 172.21.15.189:0/3234181044 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00840049d0 0x7f0084004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.737 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.728+0000 7f00a4f51700 1 -- 172.21.15.189:0/3234181044 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0084003240 con 0x7f0084003730 2020-11-20T12:05:02.737 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.728+0000 7f00a5752700 1 --2- 172.21.15.189:0/3234181044 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00840049d0 0x7f0084004e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:02.738 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.728+0000 7f00a4f51700 1 --2- 172.21.15.189:0/3234181044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0084003730 0x7f0084003b50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f009400c440 tx=0x7f00a013ded0).ready entity=mon.1 client_cookie=6689d3a7ab279657 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:02.738 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.728+0000 7f009f7fe700 1 -- 172.21.15.189:0/3234181044 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00940072e0 con 0x7f0084003730 2020-11-20T12:05:02.738 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.728+0000 7f009f7fe700 1 -- 172.21.15.189:0/3234181044 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0094007440 con 0x7f0084003730 2020-11-20T12:05:02.738 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.728+0000 7f009f7fe700 1 -- 172.21.15.189:0/3234181044 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0094005820 con 0x7f0084003730 2020-11-20T12:05:02.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.728+0000 7f00a5f53700 1 -- 172.21.15.189:0/3234181044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0084003730 msgr2=0x7f0084003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.728+0000 7f00a5f53700 1 --2- 172.21.15.189:0/3234181044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0084003730 0x7f0084003b50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f009400c440 tx=0x7f00a013ded0).stop 2020-11-20T12:05:02.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.728+0000 7f00a5752700 1 -- 172.21.15.189:0/3234181044 reap_dead start 2020-11-20T12:05:02.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.728+0000 7f00a5f53700 1 -- 172.21.15.189:0/3234181044 shutdown_connections 2020-11-20T12:05:02.740 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.728+0000 7f00a5f53700 1 -- 172.21.15.189:0/3234181044 >> 172.21.15.189:0/3234181044 conn(0x7f00a0062080 msgr2=0x7f00a0062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:02.740 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.728+0000 7f00a5f53700 1 -- 172.21.15.189:0/3234181044 shutdown_connections 2020-11-20T12:05:02.740 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.728+0000 7f00a5f53700 1 -- 172.21.15.189:0/3234181044 wait complete. 2020-11-20T12:05:02.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.728+0000 7f00a5f53700 1 Processor -- start 2020-11-20T12:05:02.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.728+0000 7f00a5f53700 1 -- start start 2020-11-20T12:05:02.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.728+0000 7f00a5f53700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0084003730 0x7f00a013cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.744 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.728+0000 7f00a5f53700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f00a00aaf70 0x7f00a0149ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.744 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.728+0000 7f00a5f53700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00a0149fd0 0x7f00a0137070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.744 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.728+0000 7f00a5f53700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a014a450 con 0x7f0084003730 2020-11-20T12:05:02.744 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.728+0000 7f00a5f53700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a00a6370 con 0x7f00a0149fd0 2020-11-20T12:05:02.745 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.728+0000 7f00a5f53700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a00a6610 con 0x7f00a00aaf70 2020-11-20T12:05:02.745 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.732+0000 7f00a5752700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00a0149fd0 0x7f00a0137070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.745 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.732+0000 7f00a5752700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00a0149fd0 0x7f00a0137070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48634/0 (socket says 172.21.15.189:48634) 2020-11-20T12:05:02.745 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.732+0000 7f00a5752700 1 -- 172.21.15.189:0/2743363751 learned_addr learned my addr 172.21.15.189:0/2743363751 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:02.746 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.732+0000 7f009ffff700 1 --2- 172.21.15.189:0/2743363751 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f00a00aaf70 0x7f00a0149ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.746 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.732+0000 7f00a4f51700 1 --2- 172.21.15.189:0/2743363751 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0084003730 0x7f00a013cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.746 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.732+0000 7f00a5752700 1 -- 172.21.15.189:0/2743363751 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f00a00aaf70 msgr2=0x7f00a0149ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.746 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.732+0000 7f00a5752700 1 --2- 172.21.15.189:0/2743363751 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f00a00aaf70 0x7f00a0149ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.747 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.732+0000 7f00a5752700 1 -- 172.21.15.189:0/2743363751 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0084003730 msgr2=0x7f00a013cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.747 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.732+0000 7f00a5752700 1 --2- 172.21.15.189:0/2743363751 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0084003730 0x7f00a013cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.747 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.732+0000 7f00a5752700 1 -- 172.21.15.189:0/2743363751 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a00a6060 con 0x7f00a0149fd0 2020-11-20T12:05:02.747 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.732+0000 7f00a5752700 1 --2- 172.21.15.189:0/2743363751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00a0149fd0 0x7f00a0137070 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f00a013dc30 tx=0x7f0098008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:02.747 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.732+0000 7f009cff9700 1 -- 172.21.15.189:0/2743363751 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009800e960 con 0x7f00a0149fd0 2020-11-20T12:05:02.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.732+0000 7f009cff9700 1 -- 172.21.15.189:0/2743363751 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f009800eae0 con 0x7f00a0149fd0 2020-11-20T12:05:02.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.732+0000 7f00a5f53700 1 -- 172.21.15.189:0/2743363751 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a013d540 con 0x7f00a0149fd0 2020-11-20T12:05:02.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.732+0000 7f009cff9700 1 -- 172.21.15.189:0/2743363751 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009800eea0 con 0x7f00a0149fd0 2020-11-20T12:05:02.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.732+0000 7f00a5f53700 1 -- 172.21.15.189:0/2743363751 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00a00a7e80 con 0x7f00a0149fd0 2020-11-20T12:05:02.749 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.732+0000 7f00a5f53700 1 -- 172.21.15.189:0/2743363751 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a00a67d0 con 0x7f00a0149fd0 2020-11-20T12:05:02.749 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.732+0000 7f009cff9700 1 -- 172.21.15.189:0/2743363751 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f009801e030 con 0x7f00a0149fd0 2020-11-20T12:05:02.749 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.732+0000 7f009cff9700 1 --2- 172.21.15.189:0/2743363751 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f00780557c0 0x7f0078057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:02.750 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.732+0000 7f009cff9700 1 -- 172.21.15.189:0/2743363751 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f009806e2e0 con 0x7f00a0149fd0 2020-11-20T12:05:02.750 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.732+0000 7f00a4f51700 1 --2- 172.21.15.189:0/2743363751 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f00780557c0 0x7f0078057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:02.750 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.732+0000 7f00a4f51700 1 --2- 172.21.15.189:0/2743363751 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f00780557c0 0x7f0078057c80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f00940078a0 tx=0x7f0094015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:02.750 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.740+0000 7f009cff9700 1 -- 172.21.15.189:0/2743363751 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0098044180 con 0x7f00a0149fd0 2020-11-20T12:05:02.923 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.912+0000 7f00a5f53700 1 -- 172.21.15.189:0/2743363751 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f00a00b1370 con 0x7f00a0149fd0 2020-11-20T12:05:02.924 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.912+0000 7f009cff9700 1 -- 172.21.15.189:0/2743363751 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f00980477a0 con 0x7f00a0149fd0 2020-11-20T12:05:02.924 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.916+0000 7f00a5f53700 1 -- 172.21.15.189:0/2743363751 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f00780557c0 msgr2=0x7f0078057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.926 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.916+0000 7f00a5f53700 1 --2- 172.21.15.189:0/2743363751 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f00780557c0 0x7f0078057c80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f00940078a0 tx=0x7f0094015000).stop 2020-11-20T12:05:02.927 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.916+0000 7f00a5f53700 1 -- 172.21.15.189:0/2743363751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00a0149fd0 msgr2=0x7f00a0137070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:02.927 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.916+0000 7f00a5f53700 1 --2- 172.21.15.189:0/2743363751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00a0149fd0 0x7f00a0137070 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f00a013dc30 tx=0x7f0098008040).stop 2020-11-20T12:05:02.928 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.916+0000 7f00a5f53700 1 -- 172.21.15.189:0/2743363751 shutdown_connections 2020-11-20T12:05:02.928 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.916+0000 7f00a5f53700 1 --2- 172.21.15.189:0/2743363751 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f00780557c0 0x7f0078057c80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.929 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.916+0000 7f00a5f53700 1 --2- 172.21.15.189:0/2743363751 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0084003730 0x7f00a013cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.929 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.916+0000 7f00a5f53700 1 --2- 172.21.15.189:0/2743363751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00a0149fd0 0x7f00a0137070 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.930 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.916+0000 7f00a5f53700 1 --2- 172.21.15.189:0/2743363751 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f00a00aaf70 0x7f00a0149ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:02.930 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.916+0000 7f00a5f53700 1 -- 172.21.15.189:0/2743363751 >> 172.21.15.189:0/2743363751 conn(0x7f00a0062080 msgr2=0x7f00a0066c50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:02.931 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.916+0000 7f00a5f53700 1 -- 172.21.15.189:0/2743363751 shutdown_connections 2020-11-20T12:05:02.931 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:02.916+0000 7f00a5f53700 1 -- 172.21.15.189:0/2743363751 wait complete. 2020-11-20T12:05:02.932 INFO:teuthology.orchestra.run.smithi189.stdout:55834574853 2020-11-20T12:05:02.934 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.2 2020-11-20T12:05:02.936 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2020-11-20T12:05:03.078 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.076+0000 7f102759e700 1 Processor -- start 2020-11-20T12:05:03.183 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.076+0000 7f102759e700 1 -- start start 2020-11-20T12:05:03.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.076+0000 7f102759e700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f10280a8840 0x7f102813eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:03.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.076+0000 7f102759e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1028142eb0 con 0x7f1028066ed0 2020-11-20T12:05:03.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.076+0000 7f102759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1028066800 con 0x7f102813f030 2020-11-20T12:05:03.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.076+0000 7f102759e700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f102813f4a0 con 0x7f10280a8840 2020-11-20T12:05:03.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.076+0000 7f1026d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f102813f030 0x7f10281427a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50260/0 (socket says 172.21.15.189:50260) 2020-11-20T12:05:03.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.076+0000 7f1026d9d700 1 -- 172.21.15.189:0/2791923865 learned_addr learned my addr 172.21.15.189:0/2791923865 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:03.194 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.076+0000 7f102659c700 1 --2- 172.21.15.189:0/2791923865 >> v2:172.21.15.47:3300/0 conn(0x7f10280a8840 0x7f102813eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:03.194 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.076+0000 7f102559a700 1 -- 172.21.15.189:0/2791923865 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199476334 0 0) 0x7f1028066800 con 0x7f102813f030 2020-11-20T12:05:03.194 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.076+0000 7f102559a700 1 -- 172.21.15.189:0/2791923865 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f100c003240 con 0x7f102813f030 2020-11-20T12:05:03.194 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.076+0000 7f102659c700 1 -- 172.21.15.189:0/2791923865 >> v1:172.21.15.189:6789/0 conn(0x7f102813f030 legacy=0x7f10281427a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.194 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.076+0000 7f102659c700 1 -- 172.21.15.189:0/2791923865 >> v1:172.21.15.47:6789/0 conn(0x7f1028066ed0 legacy=0x7f1028142090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.195 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.076+0000 7f102659c700 1 -- 172.21.15.189:0/2791923865 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1028066aa0 con 0x7f10280a8840 2020-11-20T12:05:03.195 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.076+0000 7f102659c700 1 --2- 172.21.15.189:0/2791923865 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f10280a8840 0x7f102813eb40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f101c00b9a0 tx=0x7f101c006df0).ready entity=mon.0 client_cookie=a60cdd4f94fa5eff server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:03.195 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102559a700 1 -- 172.21.15.189:0/2791923865 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101c0097b0 con 0x7f10280a8840 2020-11-20T12:05:03.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102559a700 1 -- 172.21.15.189:0/2791923865 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f101c009910 con 0x7f10280a8840 2020-11-20T12:05:03.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102559a700 1 -- 172.21.15.189:0/2791923865 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101c009c90 con 0x7f10280a8840 2020-11-20T12:05:03.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102759e700 1 -- 172.21.15.189:0/2791923865 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f10280a8840 msgr2=0x7f102813eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102759e700 1 --2- 172.21.15.189:0/2791923865 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f10280a8840 0x7f102813eb40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f101c00b9a0 tx=0x7f101c006df0).stop 2020-11-20T12:05:03.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102759e700 1 -- 172.21.15.189:0/2791923865 shutdown_connections 2020-11-20T12:05:03.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102759e700 1 --2- 172.21.15.189:0/2791923865 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f10280a8840 0x7f102813eb40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102759e700 1 -- 172.21.15.189:0/2791923865 >> 172.21.15.189:0/2791923865 conn(0x7f1028060170 msgr2=0x7f1028060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:03.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102759e700 1 -- 172.21.15.189:0/2791923865 shutdown_connections 2020-11-20T12:05:03.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102759e700 1 -- 172.21.15.189:0/2791923865 wait complete. 2020-11-20T12:05:03.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102759e700 1 Processor -- start 2020-11-20T12:05:03.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102759e700 1 -- start start 2020-11-20T12:05:03.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102759e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1028066ed0 0x7f10280a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:03.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102759e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f10280a8840 0x7f1028151430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:03.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102759e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f102813f030 0x7f10280a4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:03.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102759e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10280a4f50 con 0x7f10280a8840 2020-11-20T12:05:03.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102759e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10280a5180 con 0x7f1028066ed0 2020-11-20T12:05:03.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102759e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10280a5420 con 0x7f102813f030 2020-11-20T12:05:03.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1028066ed0 0x7f10280a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:03.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1028066ed0 0x7f10280a0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48644/0 (socket says 172.21.15.189:48644) 2020-11-20T12:05:03.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102659c700 1 -- 172.21.15.189:0/3199887185 learned_addr learned my addr 172.21.15.189:0/3199887185 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:03.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f1025d9b700 1 --2- 172.21.15.189:0/3199887185 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f10280a8840 0x7f1028151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:03.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f1026d9d700 1 --2- 172.21.15.189:0/3199887185 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f102813f030 0x7f10280a4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:03.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102659c700 1 -- 172.21.15.189:0/3199887185 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f102813f030 msgr2=0x7f10280a4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102659c700 1 --2- 172.21.15.189:0/3199887185 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f102813f030 0x7f10280a4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102659c700 1 -- 172.21.15.189:0/3199887185 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f10280a8840 msgr2=0x7f1028151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.202 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102659c700 1 --2- 172.21.15.189:0/3199887185 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f10280a8840 0x7f1028151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.202 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102659c700 1 -- 172.21.15.189:0/3199887185 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10280a56c0 con 0x7f1028066ed0 2020-11-20T12:05:03.202 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f1025d9b700 1 --2- 172.21.15.189:0/3199887185 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f10280a8840 0x7f1028151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:03.202 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102659c700 1 --2- 172.21.15.189:0/3199887185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1028066ed0 0x7f10280a0be0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f101c0098e0 tx=0x7f101c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:03.202 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f10167fc700 1 -- 172.21.15.189:0/3199887185 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101c01d3c0 con 0x7f1028066ed0 2020-11-20T12:05:03.203 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f10167fc700 1 -- 172.21.15.189:0/3199887185 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f101c01d520 con 0x7f1028066ed0 2020-11-20T12:05:03.203 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102759e700 1 -- 172.21.15.189:0/3199887185 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1028151920 con 0x7f1028066ed0 2020-11-20T12:05:03.203 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f10167fc700 1 -- 172.21.15.189:0/3199887185 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101c01d8c0 con 0x7f1028066ed0 2020-11-20T12:05:03.203 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.080+0000 7f102759e700 1 -- 172.21.15.189:0/3199887185 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10280a4a60 con 0x7f1028066ed0 2020-11-20T12:05:03.203 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.084+0000 7f102759e700 1 -- 172.21.15.189:0/3199887185 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10280a5d80 con 0x7f1028066ed0 2020-11-20T12:05:03.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.084+0000 7f10167fc700 1 -- 172.21.15.189:0/3199887185 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f101c01a040 con 0x7f1028066ed0 2020-11-20T12:05:03.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.084+0000 7f10167fc700 1 --2- 172.21.15.189:0/3199887185 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0ffc0557f0 0x7f0ffc057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:03.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.084+0000 7f10167fc700 1 -- 172.21.15.189:0/3199887185 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f101c06fc10 con 0x7f1028066ed0 2020-11-20T12:05:03.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.084+0000 7f1025d9b700 1 --2- 172.21.15.189:0/3199887185 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0ffc0557f0 0x7f0ffc057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:03.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.084+0000 7f1025d9b700 1 --2- 172.21.15.189:0/3199887185 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0ffc0557f0 0x7f0ffc057cb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f1018001550 tx=0x7f1018009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:03.205 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.088+0000 7f10167fc700 1 -- 172.21.15.189:0/3199887185 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f10280a5d80 con 0x7f1028066ed0 2020-11-20T12:05:03.242 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.240+0000 7f102759e700 1 -- 172.21.15.189:0/3199887185 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f10280a5d80 con 0x7f1028066ed0 2020-11-20T12:05:03.242 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.240+0000 7f10167fc700 1 -- 172.21.15.189:0/3199887185 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f10280a5d80 con 0x7f1028066ed0 2020-11-20T12:05:03.243 INFO:teuthology.orchestra.run.smithi189.stdout:55834574853 2020-11-20T12:05:03.244 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.240+0000 7f102759e700 1 -- 172.21.15.189:0/3199887185 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0ffc0557f0 msgr2=0x7f0ffc057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.245 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.240+0000 7f102759e700 1 --2- 172.21.15.189:0/3199887185 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0ffc0557f0 0x7f0ffc057cb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f1018001550 tx=0x7f1018009040).stop 2020-11-20T12:05:03.245 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.240+0000 7f102759e700 1 -- 172.21.15.189:0/3199887185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1028066ed0 msgr2=0x7f10280a0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.245 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.240+0000 7f102759e700 1 --2- 172.21.15.189:0/3199887185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1028066ed0 0x7f10280a0be0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f101c0098e0 tx=0x7f101c015000).stop 2020-11-20T12:05:03.245 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.240+0000 7f102759e700 1 -- 172.21.15.189:0/3199887185 shutdown_connections 2020-11-20T12:05:03.245 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.240+0000 7f102759e700 1 --2- 172.21.15.189:0/3199887185 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f102813f030 0x7f10280a4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.246 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.240+0000 7f102759e700 1 --2- 172.21.15.189:0/3199887185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1028066ed0 0x7f10280a0be0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.246 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.240+0000 7f102759e700 1 --2- 172.21.15.189:0/3199887185 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0ffc0557f0 0x7f0ffc057cb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.246 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.240+0000 7f102759e700 1 --2- 172.21.15.189:0/3199887185 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f10280a8840 0x7f1028151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.246 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.240+0000 7f102759e700 1 -- 172.21.15.189:0/3199887185 >> 172.21.15.189:0/3199887185 conn(0x7f1028060170 msgr2=0x7f102805ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:03.246 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.240+0000 7f102759e700 1 -- 172.21.15.189:0/3199887185 shutdown_connections 2020-11-20T12:05:03.247 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.240+0000 7f102759e700 1 -- 172.21.15.189:0/3199887185 wait complete. 2020-11-20T12:05:03.258 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.3 2020-11-20T12:05:03.259 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2020-11-20T12:05:03.381 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.376+0000 7f9731b54700 1 Processor -- start 2020-11-20T12:05:03.381 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f9731b54700 1 -- start start 2020-11-20T12:05:03.384 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f9731b54700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f972c0a8690 0x7f972c13a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:03.384 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f9731b54700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f972c065d90 0x7f972c13add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:03.384 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f9731b54700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f972c060f50 con 0x7f972c134360 2020-11-20T12:05:03.384 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f9731b54700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f972c13b2c0 con 0x7f972c065d90 2020-11-20T12:05:03.385 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f9731b54700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f972c065ae0 con 0x7f972c0a8690 2020-11-20T12:05:03.385 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f9730b52700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f972c0a8690 0x7f972c13a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:03.385 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f9731353700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f972c134360 0x7f972c1347d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50276/0 (socket says 172.21.15.189:50276) 2020-11-20T12:05:03.385 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f9731353700 1 -- 172.21.15.189:0/2541943192 learned_addr learned my addr 172.21.15.189:0/2541943192 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:03.385 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f972bfff700 1 --2- 172.21.15.189:0/2541943192 >> v2:172.21.15.47:3300/0 conn(0x7f972c065d90 0x7f972c13add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:03.386 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f9730b52700 1 -- 172.21.15.189:0/2541943192 >> v2:172.21.15.47:3300/0 conn(0x7f972c065d90 msgr2=0x7f972c13add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:03.386 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f9730b52700 1 --2- 172.21.15.189:0/2541943192 >> v2:172.21.15.47:3300/0 conn(0x7f972c065d90 0x7f972c13add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.386 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f9730b52700 1 -- 172.21.15.189:0/2541943192 >> v1:172.21.15.189:6789/0 conn(0x7f972c134360 legacy=0x7f972c1347d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.386 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f9730b52700 1 -- 172.21.15.189:0/2541943192 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f972c0662a0 con 0x7f972c0a8690 2020-11-20T12:05:03.386 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f972b7fe700 1 -- 172.21.15.189:0/2541943192 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767870917 0 0) 0x7f972c060f50 con 0x7f972c134360 2020-11-20T12:05:03.387 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f9730b52700 1 --2- 172.21.15.189:0/2541943192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f972c0a8690 0x7f972c13a8e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f972c068350 tx=0x7f9718006df0).ready entity=mon.1 client_cookie=ef3b6d84903c3d54 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:03.387 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f972b7fe700 1 -- 172.21.15.189:0/2541943192 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9718009840 con 0x7f972c0a8690 2020-11-20T12:05:03.387 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f972b7fe700 1 -- 172.21.15.189:0/2541943192 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97180099a0 con 0x7f972c0a8690 2020-11-20T12:05:03.387 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f972b7fe700 1 -- 172.21.15.189:0/2541943192 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9718009d20 con 0x7f972c0a8690 2020-11-20T12:05:03.388 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f9731b54700 1 -- 172.21.15.189:0/2541943192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f972c0a8690 msgr2=0x7f972c13a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.388 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f9731b54700 1 --2- 172.21.15.189:0/2541943192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f972c0a8690 0x7f972c13a8e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f972c068350 tx=0x7f9718006df0).stop 2020-11-20T12:05:03.388 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f9731b54700 1 -- 172.21.15.189:0/2541943192 shutdown_connections 2020-11-20T12:05:03.388 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f9731b54700 1 --2- 172.21.15.189:0/2541943192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f972c0a8690 0x7f972c13a8e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.388 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f9731b54700 1 --2- 172.21.15.189:0/2541943192 >> v2:172.21.15.47:3300/0 conn(0x7f972c065d90 0x7f972c13add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.389 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f9731b54700 1 -- 172.21.15.189:0/2541943192 >> 172.21.15.189:0/2541943192 conn(0x7f972c060170 msgr2=0x7f972c060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:03.389 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f9731b54700 1 -- 172.21.15.189:0/2541943192 shutdown_connections 2020-11-20T12:05:03.389 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.380+0000 7f9731b54700 1 -- 172.21.15.189:0/2541943192 wait complete. 2020-11-20T12:05:03.392 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.384+0000 7f9731b54700 1 Processor -- start 2020-11-20T12:05:03.392 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.384+0000 7f9731b54700 1 -- start start 2020-11-20T12:05:03.393 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.384+0000 7f9731b54700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f972c065d90 0x7f972c0ac680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:03.393 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.384+0000 7f9731b54700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f972c0a8690 0x7f972c09ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:03.393 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.384+0000 7f9731b54700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f972c134360 0x7f972c142a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:03.393 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.384+0000 7f9731b54700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972c0abdb0 con 0x7f972c134360 2020-11-20T12:05:03.393 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.384+0000 7f9731b54700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972c0a6540 con 0x7f972c0a8690 2020-11-20T12:05:03.394 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.384+0000 7f9731b54700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f972c0ab3d0 con 0x7f972c065d90 2020-11-20T12:05:03.394 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.384+0000 7f972bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f972c0a8690 0x7f972c09ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:03.394 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.384+0000 7f972bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f972c0a8690 0x7f972c09ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48660/0 (socket says 172.21.15.189:48660) 2020-11-20T12:05:03.394 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.384+0000 7f972bfff700 1 -- 172.21.15.189:0/2226034277 learned_addr learned my addr 172.21.15.189:0/2226034277 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:03.394 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.384+0000 7f9730b52700 1 --2- 172.21.15.189:0/2226034277 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f972c065d90 0x7f972c0ac680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:03.395 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.384+0000 7f9731353700 1 --2- 172.21.15.189:0/2226034277 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f972c134360 0x7f972c142a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:03.395 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.384+0000 7f972bfff700 1 -- 172.21.15.189:0/2226034277 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f972c065d90 msgr2=0x7f972c0ac680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.395 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.384+0000 7f972bfff700 1 --2- 172.21.15.189:0/2226034277 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f972c065d90 0x7f972c0ac680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.395 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.384+0000 7f972bfff700 1 -- 172.21.15.189:0/2226034277 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f972c134360 msgr2=0x7f972c142a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.395 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.384+0000 7f972bfff700 1 --2- 172.21.15.189:0/2226034277 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f972c134360 0x7f972c142a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.396 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.384+0000 7f972bfff700 1 -- 172.21.15.189:0/2226034277 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f972c0ab670 con 0x7f972c0a8690 2020-11-20T12:05:03.396 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.384+0000 7f9730b52700 1 --2- 172.21.15.189:0/2226034277 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f972c065d90 0x7f972c0ac680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:03.396 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.384+0000 7f972bfff700 1 --2- 172.21.15.189:0/2226034277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f972c0a8690 0x7f972c09ca50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f972000ebd0 tx=0x7f9720006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:03.396 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.384+0000 7f9728ff9700 1 -- 172.21.15.189:0/2226034277 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972000b9a0 con 0x7f972c0a8690 2020-11-20T12:05:03.396 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.384+0000 7f9728ff9700 1 -- 172.21.15.189:0/2226034277 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972000bb00 con 0x7f972c0a8690 2020-11-20T12:05:03.397 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.384+0000 7f9731b54700 1 -- 172.21.15.189:0/2226034277 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f972c0ab910 con 0x7f972c0a8690 2020-11-20T12:05:03.397 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.384+0000 7f9728ff9700 1 -- 172.21.15.189:0/2226034277 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972000be80 con 0x7f972c0a8690 2020-11-20T12:05:03.397 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.384+0000 7f9731b54700 1 -- 172.21.15.189:0/2226034277 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f972c09cf40 con 0x7f972c0a8690 2020-11-20T12:05:03.397 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.384+0000 7f9731b54700 1 -- 172.21.15.189:0/2226034277 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f972c0abaa0 con 0x7f972c0a8690 2020-11-20T12:05:03.398 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.388+0000 7f9728ff9700 1 -- 172.21.15.189:0/2226034277 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f9720012600 con 0x7f972c0a8690 2020-11-20T12:05:03.398 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.388+0000 7f9728ff9700 1 --2- 172.21.15.189:0/2226034277 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f97040557c0 0x7f9704057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:03.398 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.388+0000 7f9728ff9700 1 -- 172.21.15.189:0/2226034277 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f972006ea20 con 0x7f972c0a8690 2020-11-20T12:05:03.398 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.388+0000 7f9730b52700 1 --2- 172.21.15.189:0/2226034277 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f97040557c0 0x7f9704057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:03.399 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.388+0000 7f9730b52700 1 --2- 172.21.15.189:0/2226034277 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f97040557c0 0x7f9704057c80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f9718006e30 tx=0x7f9718017000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:03.400 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.392+0000 7f9728ff9700 1 -- 172.21.15.189:0/2226034277 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9720044860 con 0x7f972c0a8690 2020-11-20T12:05:03.548 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.544+0000 7f9731b54700 1 -- 172.21.15.189:0/2226034277 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f972c064a80 con 0x7f972c0a8690 2020-11-20T12:05:03.549 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.544+0000 7f9728ff9700 1 -- 172.21.15.189:0/2226034277 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9720047e80 con 0x7f972c0a8690 2020-11-20T12:05:03.549 INFO:teuthology.orchestra.run.smithi189.stdout:55834574854 2020-11-20T12:05:03.551 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.548+0000 7f9731b54700 1 -- 172.21.15.189:0/2226034277 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f97040557c0 msgr2=0x7f9704057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.551 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.548+0000 7f9731b54700 1 --2- 172.21.15.189:0/2226034277 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f97040557c0 0x7f9704057c80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f9718006e30 tx=0x7f9718017000).stop 2020-11-20T12:05:03.551 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.548+0000 7f9731b54700 1 -- 172.21.15.189:0/2226034277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f972c0a8690 msgr2=0x7f972c09ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.551 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.548+0000 7f9731b54700 1 --2- 172.21.15.189:0/2226034277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f972c0a8690 0x7f972c09ca50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f972000ebd0 tx=0x7f9720006df0).stop 2020-11-20T12:05:03.552 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.548+0000 7f9731b54700 1 -- 172.21.15.189:0/2226034277 shutdown_connections 2020-11-20T12:05:03.552 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.548+0000 7f9731b54700 1 --2- 172.21.15.189:0/2226034277 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f97040557c0 0x7f9704057c80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.552 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.548+0000 7f9731b54700 1 --2- 172.21.15.189:0/2226034277 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f972c134360 0x7f972c142a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.552 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.548+0000 7f9731b54700 1 --2- 172.21.15.189:0/2226034277 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f972c065d90 0x7f972c0ac680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.552 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.548+0000 7f9731b54700 1 --2- 172.21.15.189:0/2226034277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f972c0a8690 0x7f972c09ca50 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.553 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.548+0000 7f9731b54700 1 -- 172.21.15.189:0/2226034277 >> 172.21.15.189:0/2226034277 conn(0x7f972c060170 msgr2=0x7f972c136b20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:03.553 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.548+0000 7f9731b54700 1 -- 172.21.15.189:0/2226034277 shutdown_connections 2020-11-20T12:05:03.553 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.548+0000 7f9731b54700 1 -- 172.21.15.189:0/2226034277 wait complete. 2020-11-20T12:05:03.566 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.4 2020-11-20T12:05:03.566 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2020-11-20T12:05:03.705 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.688+0000 7fd9e8f63700 1 Processor -- start 2020-11-20T12:05:03.706 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.688+0000 7fd9e8f63700 1 -- start start 2020-11-20T12:05:03.708 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.688+0000 7fd9e8f63700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd9e40a7280 0x7fd9e40a0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:03.708 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.688+0000 7fd9e8f63700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd9e40a0a10 0x7fd9e4141cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:03.709 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.688+0000 7fd9e8f63700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e4066e80 con 0x7fd9e4060310 2020-11-20T12:05:03.710 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.688+0000 7fd9e8f63700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9e4067760 con 0x7fd9e40a7280 2020-11-20T12:05:03.711 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.688+0000 7fd9e8f63700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e40a9320 con 0x7fd9e40a0a10 2020-11-20T12:05:03.712 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.688+0000 7fd9e259c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd9e4060310 0x7fd9e40a9cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50292/0 (socket says 172.21.15.189:50292) 2020-11-20T12:05:03.713 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.688+0000 7fd9e259c700 1 -- 172.21.15.189:0/3186325253 learned_addr learned my addr 172.21.15.189:0/3186325253 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:03.713 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.688+0000 7fd9e1d9b700 1 --2- 172.21.15.189:0/3186325253 >> v2:172.21.15.47:3300/0 conn(0x7fd9e40a7280 0x7fd9e40a0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:03.714 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.688+0000 7fd9e2d9d700 1 --2- 172.21.15.189:0/3186325253 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd9e40a0a10 0x7fd9e4141cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:03.715 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.688+0000 7fd9e159a700 1 -- 172.21.15.189:0/3186325253 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338264491 0 0) 0x7fd9e4066e80 con 0x7fd9e4060310 2020-11-20T12:05:03.715 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.688+0000 7fd9e159a700 1 -- 172.21.15.189:0/3186325253 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9c8003240 con 0x7fd9e4060310 2020-11-20T12:05:03.716 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e159a700 1 -- 172.21.15.189:0/3186325253 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1243941039 0 0) 0x7fd9c8003240 con 0x7fd9e4060310 2020-11-20T12:05:03.716 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e1d9b700 1 -- 172.21.15.189:0/3186325253 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd9e40a0a10 msgr2=0x7fd9e4141cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:03.716 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e1d9b700 1 --2- 172.21.15.189:0/3186325253 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd9e40a0a10 0x7fd9e4141cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.716 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e1d9b700 1 -- 172.21.15.189:0/3186325253 >> v1:172.21.15.189:6789/0 conn(0x7fd9e4060310 legacy=0x7fd9e40a9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e1d9b700 1 -- 172.21.15.189:0/3186325253 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e40606f0 con 0x7fd9e40a7280 2020-11-20T12:05:03.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e1d9b700 1 --2- 172.21.15.189:0/3186325253 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9e40a7280 0x7fd9e40a0520 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fd9d400b9a0 tx=0x7fd9d4006df0).ready entity=mon.0 client_cookie=63eda8e70022b685 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:03.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e159a700 1 -- 172.21.15.189:0/3186325253 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d80014e0 con 0x7fd9e40a7280 2020-11-20T12:05:03.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e159a700 1 -- 172.21.15.189:0/3186325253 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d4011400 con 0x7fd9e40a7280 2020-11-20T12:05:03.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e159a700 1 -- 172.21.15.189:0/3186325253 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d4011780 con 0x7fd9e40a7280 2020-11-20T12:05:03.718 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e8f63700 1 -- 172.21.15.189:0/3186325253 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9e40a7280 msgr2=0x7fd9e40a0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.718 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e8f63700 1 --2- 172.21.15.189:0/3186325253 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9e40a7280 0x7fd9e40a0520 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fd9d400b9a0 tx=0x7fd9d4006df0).stop 2020-11-20T12:05:03.718 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e8f63700 1 -- 172.21.15.189:0/3186325253 shutdown_connections 2020-11-20T12:05:03.718 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e8f63700 1 --2- 172.21.15.189:0/3186325253 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd9e40a0a10 0x7fd9e4141cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.718 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e8f63700 1 --2- 172.21.15.189:0/3186325253 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9e40a7280 0x7fd9e40a0520 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.719 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e8f63700 1 -- 172.21.15.189:0/3186325253 >> 172.21.15.189:0/3186325253 conn(0x7fd9e4065910 msgr2=0x7fd9e4060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:03.719 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e8f63700 1 -- 172.21.15.189:0/3186325253 shutdown_connections 2020-11-20T12:05:03.719 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e8f63700 1 -- 172.21.15.189:0/3186325253 wait complete. 2020-11-20T12:05:03.719 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e8f63700 1 Processor -- start 2020-11-20T12:05:03.719 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e8f63700 1 -- start start 2020-11-20T12:05:03.720 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e8f63700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9e4060310 0x7fd9e4149ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:03.720 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e8f63700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd9e40a0a10 0x7fd9e413aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:03.720 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e8f63700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9e40a7280 0x7fd9e41362c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:03.720 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e8f63700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e414a290 con 0x7fd9e4060310 2020-11-20T12:05:03.720 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e8f63700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e414a530 con 0x7fd9e40a7280 2020-11-20T12:05:03.721 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e8f63700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e414a7d0 con 0x7fd9e40a0a10 2020-11-20T12:05:03.721 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e2d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9e40a7280 0x7fd9e41362c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:03.721 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e2d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9e40a7280 0x7fd9e41362c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48676/0 (socket says 172.21.15.189:48676) 2020-11-20T12:05:03.721 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e2d9d700 1 -- 172.21.15.189:0/2457622804 learned_addr learned my addr 172.21.15.189:0/2457622804 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:03.721 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e1d9b700 1 --2- 172.21.15.189:0/2457622804 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd9e40a0a10 0x7fd9e413aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:03.722 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e259c700 1 --2- 172.21.15.189:0/2457622804 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9e4060310 0x7fd9e4149ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:03.722 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e2d9d700 1 -- 172.21.15.189:0/2457622804 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd9e40a0a10 msgr2=0x7fd9e413aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.722 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e2d9d700 1 --2- 172.21.15.189:0/2457622804 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd9e40a0a10 0x7fd9e413aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.722 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e2d9d700 1 -- 172.21.15.189:0/2457622804 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9e4060310 msgr2=0x7fd9e4149ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.722 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e2d9d700 1 --2- 172.21.15.189:0/2457622804 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9e4060310 0x7fd9e4149ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.723 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e2d9d700 1 -- 172.21.15.189:0/2457622804 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e41367b0 con 0x7fd9e40a7280 2020-11-20T12:05:03.723 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e1d9b700 1 --2- 172.21.15.189:0/2457622804 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd9e40a0a10 0x7fd9e413aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:03.723 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.692+0000 7fd9e259c700 1 --2- 172.21.15.189:0/2457622804 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9e4060310 0x7fd9e4149ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:03.723 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.696+0000 7fd9e2d9d700 1 --2- 172.21.15.189:0/2457622804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9e40a7280 0x7fd9e41362c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fd9dc00ef50 tx=0x7fd9dc0091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:03.724 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.696+0000 7fd9d27fc700 1 -- 172.21.15.189:0/2457622804 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc00ccc0 con 0x7fd9e40a7280 2020-11-20T12:05:03.724 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.696+0000 7fd9d27fc700 1 -- 172.21.15.189:0/2457622804 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9dc00ce20 con 0x7fd9e40a7280 2020-11-20T12:05:03.724 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.696+0000 7fd9e8f63700 1 -- 172.21.15.189:0/2457622804 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e4140950 con 0x7fd9e40a7280 2020-11-20T12:05:03.724 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.696+0000 7fd9d27fc700 1 -- 172.21.15.189:0/2457622804 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc012620 con 0x7fd9e40a7280 2020-11-20T12:05:03.724 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.696+0000 7fd9e8f63700 1 -- 172.21.15.189:0/2457622804 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e414d490 con 0x7fd9e40a7280 2020-11-20T12:05:03.725 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.696+0000 7fd9e8f63700 1 -- 172.21.15.189:0/2457622804 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9e4140c50 con 0x7fd9e40a7280 2020-11-20T12:05:03.725 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.696+0000 7fd9d27fc700 1 -- 172.21.15.189:0/2457622804 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fd9dc022070 con 0x7fd9e40a7280 2020-11-20T12:05:03.726 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.700+0000 7fd9d27fc700 1 --2- 172.21.15.189:0/2457622804 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd9bc0557c0 0x7fd9bc057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:03.726 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.700+0000 7fd9d27fc700 1 -- 172.21.15.189:0/2457622804 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd9dc06ef60 con 0x7fd9e40a7280 2020-11-20T12:05:03.726 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.700+0000 7fd9e259c700 1 --2- 172.21.15.189:0/2457622804 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd9bc0557c0 0x7fd9bc057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:03.726 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.700+0000 7fd9e259c700 1 --2- 172.21.15.189:0/2457622804 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd9bc0557c0 0x7fd9bc057c80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fd9e4140e80 tx=0x7fd9d8001e30).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:03.727 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.704+0000 7fd9d27fc700 1 -- 172.21.15.189:0/2457622804 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd9dc044e00 con 0x7fd9e40a7280 2020-11-20T12:05:03.859 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.856+0000 7fd9e8f63700 1 -- 172.21.15.189:0/2457622804 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fd9e4136940 con 0x7fd9e40a7280 2020-11-20T12:05:03.860 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.856+0000 7fd9d27fc700 1 -- 172.21.15.189:0/2457622804 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd9dc016070 con 0x7fd9e40a7280 2020-11-20T12:05:03.860 INFO:teuthology.orchestra.run.smithi189.stdout:55834574853 2020-11-20T12:05:03.861 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.856+0000 7fd9e8f63700 1 -- 172.21.15.189:0/2457622804 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd9bc0557c0 msgr2=0x7fd9bc057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.861 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.856+0000 7fd9e8f63700 1 --2- 172.21.15.189:0/2457622804 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd9bc0557c0 0x7fd9bc057c80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fd9e4140e80 tx=0x7fd9d8001e30).stop 2020-11-20T12:05:03.862 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.856+0000 7fd9e8f63700 1 -- 172.21.15.189:0/2457622804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9e40a7280 msgr2=0x7fd9e41362c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:03.862 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.856+0000 7fd9e8f63700 1 --2- 172.21.15.189:0/2457622804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9e40a7280 0x7fd9e41362c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fd9dc00ef50 tx=0x7fd9dc0091d0).stop 2020-11-20T12:05:03.862 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.860+0000 7fd9e8f63700 1 -- 172.21.15.189:0/2457622804 shutdown_connections 2020-11-20T12:05:03.862 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.860+0000 7fd9e8f63700 1 --2- 172.21.15.189:0/2457622804 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd9bc0557c0 0x7fd9bc057c80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.862 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.860+0000 7fd9e8f63700 1 --2- 172.21.15.189:0/2457622804 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd9e4060310 0x7fd9e4149ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.860+0000 7fd9e8f63700 1 --2- 172.21.15.189:0/2457622804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9e40a7280 0x7fd9e41362c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.860+0000 7fd9e8f63700 1 --2- 172.21.15.189:0/2457622804 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd9e40a0a10 0x7fd9e413aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:03.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.860+0000 7fd9e8f63700 1 -- 172.21.15.189:0/2457622804 >> 172.21.15.189:0/2457622804 conn(0x7fd9e4065910 msgr2=0x7fd9e40ad2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:03.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.860+0000 7fd9e8f63700 1 -- 172.21.15.189:0/2457622804 shutdown_connections 2020-11-20T12:05:03.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:03.860+0000 7fd9e8f63700 1 -- 172.21.15.189:0/2457622804 wait complete. 2020-11-20T12:05:03.875 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.5 2020-11-20T12:05:03.875 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2020-11-20T12:05:04.013 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.008+0000 7f5d5359e700 1 Processor -- start 2020-11-20T12:05:04.014 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.008+0000 7f5d5359e700 1 -- start start 2020-11-20T12:05:04.018 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d5359e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5d540a0a90 0x7f5d540a8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.019 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d5359e700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f5d5413dc20 0x7f5d54061180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.019 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d5359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d54066ee0 con 0x7f5d54137130 2020-11-20T12:05:04.019 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d5359e700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d540a96e0 con 0x7f5d5413dc20 2020-11-20T12:05:04.020 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d5359e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d54067180 con 0x7f5d540a0a90 2020-11-20T12:05:04.020 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d52d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5d54137130 0x7f5d541375b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50304/0 (socket says 172.21.15.189:50304) 2020-11-20T12:05:04.020 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d52d9d700 1 -- 172.21.15.189:0/3431038740 learned_addr learned my addr 172.21.15.189:0/3431038740 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:04.020 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d5259c700 1 --2- 172.21.15.189:0/3431038740 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5d540a0a90 0x7f5d540a8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.020 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d51d9b700 1 --2- 172.21.15.189:0/3431038740 >> v2:172.21.15.47:3300/0 conn(0x7f5d5413dc20 0x7f5d54061180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.021 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d5159a700 1 -- 172.21.15.189:0/3431038740 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 787977379 0 0) 0x7f5d54066ee0 con 0x7f5d54137130 2020-11-20T12:05:04.021 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d5159a700 1 -- 172.21.15.189:0/3431038740 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d34003240 con 0x7f5d54137130 2020-11-20T12:05:04.021 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d5259c700 1 -- 172.21.15.189:0/3431038740 >> v2:172.21.15.47:3300/0 conn(0x7f5d5413dc20 msgr2=0x7f5d54061180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:04.021 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d5259c700 1 --2- 172.21.15.189:0/3431038740 >> v2:172.21.15.47:3300/0 conn(0x7f5d5413dc20 0x7f5d54061180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.022 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d5259c700 1 -- 172.21.15.189:0/3431038740 >> v1:172.21.15.189:6789/0 conn(0x7f5d54137130 legacy=0x7f5d541375b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.022 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d5259c700 1 -- 172.21.15.189:0/3431038740 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d540a0e70 con 0x7f5d540a0a90 2020-11-20T12:05:04.022 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d51d9b700 1 --2- 172.21.15.189:0/3431038740 >> v2:172.21.15.47:3300/0 conn(0x7f5d5413dc20 0x7f5d54061180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:05:04.022 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d5259c700 1 --2- 172.21.15.189:0/3431038740 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5d540a0a90 0x7f5d540a8b80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f5d4c009c90 tx=0x7f5d4c0091d0).ready entity=mon.2 client_cookie=42a8eb8eb55f4daa server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:04.022 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d5159a700 1 -- 172.21.15.189:0/3431038740 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c00b9c0 con 0x7f5d540a0a90 2020-11-20T12:05:04.023 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d5159a700 1 -- 172.21.15.189:0/3431038740 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d4c00bb20 con 0x7f5d540a0a90 2020-11-20T12:05:04.024 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d5159a700 1 -- 172.21.15.189:0/3431038740 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c00bea0 con 0x7f5d540a0a90 2020-11-20T12:05:04.024 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d5359e700 1 -- 172.21.15.189:0/3431038740 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5d540a0a90 msgr2=0x7f5d540a8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.024 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d5359e700 1 --2- 172.21.15.189:0/3431038740 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5d540a0a90 0x7f5d540a8b80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f5d4c009c90 tx=0x7f5d4c0091d0).stop 2020-11-20T12:05:04.024 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d5359e700 1 -- 172.21.15.189:0/3431038740 shutdown_connections 2020-11-20T12:05:04.024 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d5359e700 1 --2- 172.21.15.189:0/3431038740 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5d540a0a90 0x7f5d540a8b80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.025 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d5359e700 1 --2- 172.21.15.189:0/3431038740 >> v2:172.21.15.47:3300/0 conn(0x7f5d5413dc20 0x7f5d54061180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.025 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d5359e700 1 -- 172.21.15.189:0/3431038740 >> 172.21.15.189:0/3431038740 conn(0x7f5d54063210 msgr2=0x7f5d54060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:04.025 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d5359e700 1 -- 172.21.15.189:0/3431038740 shutdown_connections 2020-11-20T12:05:04.025 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.012+0000 7f5d5359e700 1 -- 172.21.15.189:0/3431038740 wait complete. 2020-11-20T12:05:04.028 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.016+0000 7f5d5359e700 1 Processor -- start 2020-11-20T12:05:04.028 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.016+0000 7f5d5359e700 1 -- start start 2020-11-20T12:05:04.028 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.016+0000 7f5d5359e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d540a0a90 0x7f5d54149900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.028 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.016+0000 7f5d5359e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d54137130 0x7f5d5413cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.029 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.016+0000 7f5d5359e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5d5413dc20 0x7f5d54140750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.029 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.016+0000 7f5d5359e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d54139af0 con 0x7f5d540a0a90 2020-11-20T12:05:04.029 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.016+0000 7f5d5359e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d54149ef0 con 0x7f5d54137130 2020-11-20T12:05:04.029 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.016+0000 7f5d5359e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5414a190 con 0x7f5d5413dc20 2020-11-20T12:05:04.029 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.016+0000 7f5d51d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d54137130 0x7f5d5413cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.030 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.016+0000 7f5d51d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d54137130 0x7f5d5413cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48688/0 (socket says 172.21.15.189:48688) 2020-11-20T12:05:04.030 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.016+0000 7f5d51d9b700 1 -- 172.21.15.189:0/2721422300 learned_addr learned my addr 172.21.15.189:0/2721422300 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:04.030 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.016+0000 7f5d5259c700 1 --2- 172.21.15.189:0/2721422300 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d540a0a90 0x7f5d54149900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.030 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.016+0000 7f5d52d9d700 1 --2- 172.21.15.189:0/2721422300 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5d5413dc20 0x7f5d54140750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.030 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.020+0000 7f5d51d9b700 1 -- 172.21.15.189:0/2721422300 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5d5413dc20 msgr2=0x7f5d54140750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.031 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.020+0000 7f5d51d9b700 1 --2- 172.21.15.189:0/2721422300 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5d5413dc20 0x7f5d54140750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.031 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.020+0000 7f5d51d9b700 1 -- 172.21.15.189:0/2721422300 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d540a0a90 msgr2=0x7f5d54149900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.031 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.020+0000 7f5d51d9b700 1 --2- 172.21.15.189:0/2721422300 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d540a0a90 0x7f5d54149900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.031 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.020+0000 7f5d51d9b700 1 -- 172.21.15.189:0/2721422300 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d5414a430 con 0x7f5d54137130 2020-11-20T12:05:04.031 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.020+0000 7f5d5259c700 1 --2- 172.21.15.189:0/2721422300 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d540a0a90 0x7f5d54149900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:04.032 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.020+0000 7f5d51d9b700 1 --2- 172.21.15.189:0/2721422300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d54137130 0x7f5d5413cc90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f5d3c00ed20 tx=0x7f5d3c006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:04.032 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.020+0000 7f5d4a7fc700 1 -- 172.21.15.189:0/2721422300 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c00cb50 con 0x7f5d54137130 2020-11-20T12:05:04.032 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.020+0000 7f5d5359e700 1 -- 172.21.15.189:0/2721422300 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d54140230 con 0x7f5d54137130 2020-11-20T12:05:04.032 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.020+0000 7f5d5359e700 1 -- 172.21.15.189:0/2721422300 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d54140de0 con 0x7f5d54137130 2020-11-20T12:05:04.032 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.020+0000 7f5d4a7fc700 1 -- 172.21.15.189:0/2721422300 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d3c01c460 con 0x7f5d54137130 2020-11-20T12:05:04.033 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.020+0000 7f5d4a7fc700 1 -- 172.21.15.189:0/2721422300 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c01c5d0 con 0x7f5d54137130 2020-11-20T12:05:04.033 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.020+0000 7f5d5359e700 1 -- 172.21.15.189:0/2721422300 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d540a9070 con 0x7f5d54137130 2020-11-20T12:05:04.033 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.020+0000 7f5d4a7fc700 1 -- 172.21.15.189:0/2721422300 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f5d3c01c940 con 0x7f5d54137130 2020-11-20T12:05:04.033 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.024+0000 7f5d4a7fc700 1 --2- 172.21.15.189:0/2721422300 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5d300557f0 0x7f5d30057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.033 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.024+0000 7f5d4a7fc700 1 -- 172.21.15.189:0/2721422300 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5d3c06f910 con 0x7f5d54137130 2020-11-20T12:05:04.034 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.024+0000 7f5d5259c700 1 --2- 172.21.15.189:0/2721422300 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5d300557f0 0x7f5d30057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.034 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.024+0000 7f5d5259c700 1 --2- 172.21.15.189:0/2721422300 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5d300557f0 0x7f5d30057cb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f5d4c006ee0 tx=0x7f5d4c016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:04.034 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.028+0000 7f5d4a7fc700 1 -- 172.21.15.189:0/2721422300 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5d3c0456a0 con 0x7f5d54137130 2020-11-20T12:05:04.195 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.192+0000 7f5d5359e700 1 -- 172.21.15.189:0/2721422300 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f5d541403c0 con 0x7f5d54137130 2020-11-20T12:05:04.195 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.192+0000 7f5d4a7fc700 1 -- 172.21.15.189:0/2721422300 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5d541403c0 con 0x7f5d54137130 2020-11-20T12:05:04.196 INFO:teuthology.orchestra.run.smithi189.stdout:55834574853 2020-11-20T12:05:04.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.192+0000 7f5d5359e700 1 -- 172.21.15.189:0/2721422300 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5d300557f0 msgr2=0x7f5d30057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.192+0000 7f5d5359e700 1 --2- 172.21.15.189:0/2721422300 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5d300557f0 0x7f5d30057cb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f5d4c006ee0 tx=0x7f5d4c016000).stop 2020-11-20T12:05:04.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.192+0000 7f5d5359e700 1 -- 172.21.15.189:0/2721422300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d54137130 msgr2=0x7f5d5413cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.192+0000 7f5d5359e700 1 --2- 172.21.15.189:0/2721422300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d54137130 0x7f5d5413cc90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f5d3c00ed20 tx=0x7f5d3c006df0).stop 2020-11-20T12:05:04.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.192+0000 7f5d5359e700 1 -- 172.21.15.189:0/2721422300 shutdown_connections 2020-11-20T12:05:04.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.192+0000 7f5d5359e700 1 --2- 172.21.15.189:0/2721422300 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5d300557f0 0x7f5d30057cb0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.192+0000 7f5d5359e700 1 --2- 172.21.15.189:0/2721422300 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5d540a0a90 0x7f5d54149900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.192+0000 7f5d5359e700 1 --2- 172.21.15.189:0/2721422300 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5d5413dc20 0x7f5d54140750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.192+0000 7f5d5359e700 1 --2- 172.21.15.189:0/2721422300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d54137130 0x7f5d5413cc90 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.192+0000 7f5d5359e700 1 -- 172.21.15.189:0/2721422300 >> 172.21.15.189:0/2721422300 conn(0x7f5d54063210 msgr2=0x7f5d541375b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:04.202 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.192+0000 7f5d5359e700 1 -- 172.21.15.189:0/2721422300 shutdown_connections 2020-11-20T12:05:04.202 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.192+0000 7f5d5359e700 1 -- 172.21.15.189:0/2721422300 wait complete. 2020-11-20T12:05:04.210 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.6 2020-11-20T12:05:04.211 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2020-11-20T12:05:04.355 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.347+0000 7eff7bfff700 1 Processor -- start 2020-11-20T12:05:04.356 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.347+0000 7eff7bfff700 1 -- start start 2020-11-20T12:05:04.356 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.347+0000 7eff7bfff700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7eff7c060bc0 0x7eff7c061040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.356 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.347+0000 7eff7bfff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff7c061530 con 0x7eff7c134de0 2020-11-20T12:05:04.356 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.347+0000 7eff7bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff7c065ca0 con 0x7eff7c0623c0 2020-11-20T12:05:04.356 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.347+0000 7eff7bfff700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff7c0687e0 con 0x7eff7c060bc0 2020-11-20T12:05:04.357 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.347+0000 7eff7affd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7eff7c0623c0 0x7eff7c138290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50316/0 (socket says 172.21.15.189:50316) 2020-11-20T12:05:04.358 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.347+0000 7eff7affd700 1 -- 172.21.15.189:0/3059666741 learned_addr learned my addr 172.21.15.189:0/3059666741 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:04.359 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.347+0000 7eff73fff700 1 --2- 172.21.15.189:0/3059666741 >> v2:172.21.15.47:3300/0 conn(0x7eff7c060bc0 0x7eff7c061040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.359 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.347+0000 7eff7a7fc700 1 -- 172.21.15.189:0/3059666741 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592037219 0 0) 0x7eff7c065ca0 con 0x7eff7c0623c0 2020-11-20T12:05:04.359 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.347+0000 7eff7a7fc700 1 -- 172.21.15.189:0/3059666741 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff5c003240 con 0x7eff7c0623c0 2020-11-20T12:05:04.359 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.347+0000 7eff7a7fc700 1 -- 172.21.15.189:0/3059666741 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3897295792 0 0) 0x7eff5c003240 con 0x7eff7c0623c0 2020-11-20T12:05:04.359 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.347+0000 7eff7a7fc700 1 -- 172.21.15.189:0/3059666741 >> v2:172.21.15.47:3300/0 conn(0x7eff7c060bc0 msgr2=0x7eff7c061040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:04.360 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.347+0000 7eff7a7fc700 1 --2- 172.21.15.189:0/3059666741 >> v2:172.21.15.47:3300/0 conn(0x7eff7c060bc0 0x7eff7c061040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.360 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.347+0000 7eff7a7fc700 1 -- 172.21.15.189:0/3059666741 >> v1:172.21.15.47:6789/0 conn(0x7eff7c134de0 legacy=0x7eff7c135270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.360 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.347+0000 7eff7a7fc700 1 -- 172.21.15.189:0/3059666741 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff7c138ba0 con 0x7eff7c0623c0 2020-11-20T12:05:04.360 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.347+0000 7eff7a7fc700 1 -- 172.21.15.189:0/3059666741 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7eff6c0014e0 con 0x7eff7c0623c0 2020-11-20T12:05:04.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.347+0000 7eff7a7fc700 1 -- 172.21.15.189:0/3059666741 >> v1:172.21.15.189:6789/0 conn(0x7eff7c0623c0 legacy=0x7eff7c138290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.347+0000 7eff7a7fc700 1 --2- 172.21.15.189:0/3059666741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff5c003730 0x7eff5c003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.347+0000 7eff7a7fc700 1 --2- 172.21.15.189:0/3059666741 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7eff5c0044e0 0x7eff5c004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.347+0000 7eff7a7fc700 1 --2- 172.21.15.189:0/3059666741 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff5c004e50 0x7eff5c00b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.347+0000 7eff7a7fc700 1 -- 172.21.15.189:0/3059666741 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c004260 con 0x7eff5c004e50 2020-11-20T12:05:04.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.347+0000 7eff7a7fc700 1 -- 172.21.15.189:0/3059666741 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c00bc20 con 0x7eff5c003730 2020-11-20T12:05:04.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.347+0000 7eff7a7fc700 1 -- 172.21.15.189:0/3059666741 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c00bd60 con 0x7eff5c0044e0 2020-11-20T12:05:04.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.347+0000 7eff7affd700 1 --2- 172.21.15.189:0/3059666741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff5c003730 0x7eff5c003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff73fff700 1 --2- 172.21.15.189:0/3059666741 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7eff5c0044e0 0x7eff5c004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7b7fe700 1 --2- 172.21.15.189:0/3059666741 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff5c004e50 0x7eff5c00b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7affd700 1 -- 172.21.15.189:0/3059666741 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7eff5c0044e0 msgr2=0x7eff5c004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:04.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7affd700 1 --2- 172.21.15.189:0/3059666741 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7eff5c0044e0 0x7eff5c004960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7affd700 1 -- 172.21.15.189:0/3059666741 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff5c004e50 msgr2=0x7eff5c00b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:04.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7affd700 1 --2- 172.21.15.189:0/3059666741 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff5c004e50 0x7eff5c00b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.364 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7affd700 1 -- 172.21.15.189:0/3059666741 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5c003240 con 0x7eff5c003730 2020-11-20T12:05:04.364 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7b7fe700 1 --2- 172.21.15.189:0/3059666741 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff5c004e50 0x7eff5c00b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:04.364 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7b7fe700 1 -- 172.21.15.189:0/3059666741 reap_dead start 2020-11-20T12:05:04.364 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7affd700 1 --2- 172.21.15.189:0/3059666741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff5c003730 0x7eff5c003b50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7eff6c00c440 tx=0x7eff7c065e30).ready entity=mon.1 client_cookie=ea1dec337c8863ab server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:04.364 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7a7fc700 1 -- 172.21.15.189:0/3059666741 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c0072e0 con 0x7eff5c003730 2020-11-20T12:05:04.365 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7a7fc700 1 -- 172.21.15.189:0/3059666741 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff6c007440 con 0x7eff5c003730 2020-11-20T12:05:04.365 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7a7fc700 1 -- 172.21.15.189:0/3059666741 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c005800 con 0x7eff5c003730 2020-11-20T12:05:04.365 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7bfff700 1 -- 172.21.15.189:0/3059666741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff5c003730 msgr2=0x7eff5c003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.365 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7bfff700 1 --2- 172.21.15.189:0/3059666741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff5c003730 0x7eff5c003b50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7eff6c00c440 tx=0x7eff7c065e30).stop 2020-11-20T12:05:04.366 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7bfff700 1 -- 172.21.15.189:0/3059666741 shutdown_connections 2020-11-20T12:05:04.366 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7bfff700 1 --2- 172.21.15.189:0/3059666741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff5c003730 0x7eff5c003b50 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.366 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7bfff700 1 -- 172.21.15.189:0/3059666741 >> 172.21.15.189:0/3059666741 conn(0x7eff7c065fe0 msgr2=0x7eff7c061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:04.366 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7bfff700 1 -- 172.21.15.189:0/3059666741 shutdown_connections 2020-11-20T12:05:04.366 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7bfff700 1 -- 172.21.15.189:0/3059666741 wait complete. 2020-11-20T12:05:04.367 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7bfff700 1 Processor -- start 2020-11-20T12:05:04.367 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7bfff700 1 -- start start 2020-11-20T12:05:04.370 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff5c003730 0x7eff7c13c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.370 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7bfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff7c14a8a0 0x7eff7c14d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.370 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7bfff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7eff7c0a9ab0 0x7eff7c14d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.370 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7bfff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7c0a6d60 con 0x7eff7c14a8a0 2020-11-20T12:05:04.371 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7bfff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7c149c20 con 0x7eff5c003730 2020-11-20T12:05:04.371 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7bfff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff7c149ec0 con 0x7eff7c0a9ab0 2020-11-20T12:05:04.371 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7affd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff5c003730 0x7eff7c13c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.371 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7affd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff5c003730 0x7eff7c13c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48706/0 (socket says 172.21.15.189:48706) 2020-11-20T12:05:04.372 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7affd700 1 -- 172.21.15.189:0/662826094 learned_addr learned my addr 172.21.15.189:0/662826094 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:04.372 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7b7fe700 1 --2- 172.21.15.189:0/662826094 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7eff7c0a9ab0 0x7eff7c14d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.372 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff73fff700 1 --2- 172.21.15.189:0/662826094 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff7c14a8a0 0x7eff7c14d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.372 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7affd700 1 -- 172.21.15.189:0/662826094 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7eff7c0a9ab0 msgr2=0x7eff7c14d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.372 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7affd700 1 --2- 172.21.15.189:0/662826094 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7eff7c0a9ab0 0x7eff7c14d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.373 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7affd700 1 -- 172.21.15.189:0/662826094 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff7c14a8a0 msgr2=0x7eff7c14d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.373 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7affd700 1 --2- 172.21.15.189:0/662826094 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff7c14a8a0 0x7eff7c14d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.373 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7affd700 1 -- 172.21.15.189:0/662826094 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff7c14a160 con 0x7eff5c003730 2020-11-20T12:05:04.373 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7affd700 1 --2- 172.21.15.189:0/662826094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff5c003730 0x7eff7c13c190 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7eff6c007a80 tx=0x7eff6c013000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:04.373 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff72ffd700 1 -- 172.21.15.189:0/662826094 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c01a070 con 0x7eff5c003730 2020-11-20T12:05:04.374 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff72ffd700 1 -- 172.21.15.189:0/662826094 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff6c005dc0 con 0x7eff5c003730 2020-11-20T12:05:04.374 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7bfff700 1 -- 172.21.15.189:0/662826094 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff7c14a400 con 0x7eff5c003730 2020-11-20T12:05:04.374 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff72ffd700 1 -- 172.21.15.189:0/662826094 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c001b20 con 0x7eff5c003730 2020-11-20T12:05:04.374 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.351+0000 7eff7bfff700 1 -- 172.21.15.189:0/662826094 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff7c0a81a0 con 0x7eff5c003730 2020-11-20T12:05:04.375 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.355+0000 7eff7bfff700 1 -- 172.21.15.189:0/662826094 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff7c0663c0 con 0x7eff5c003730 2020-11-20T12:05:04.375 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.355+0000 7eff72ffd700 1 -- 172.21.15.189:0/662826094 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7eff6c010070 con 0x7eff5c003730 2020-11-20T12:05:04.375 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.355+0000 7eff72ffd700 1 --2- 172.21.15.189:0/662826094 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7eff580557c0 0x7eff58057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.375 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.355+0000 7eff72ffd700 1 -- 172.21.15.189:0/662826094 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7eff6c06eb10 con 0x7eff5c003730 2020-11-20T12:05:04.375 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.355+0000 7eff73fff700 1 --2- 172.21.15.189:0/662826094 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7eff580557c0 0x7eff58057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.376 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.355+0000 7eff73fff700 1 --2- 172.21.15.189:0/662826094 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7eff580557c0 0x7eff58057c80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7eff64006a90 tx=0x7eff64008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:04.376 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.359+0000 7eff72ffd700 1 -- 172.21.15.189:0/662826094 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7eff6c044a30 con 0x7eff5c003730 2020-11-20T12:05:04.529 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.527+0000 7eff7bfff700 1 -- 172.21.15.189:0/662826094 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7eff7c0a81a0 con 0x7eff5c003730 2020-11-20T12:05:04.530 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.527+0000 7eff72ffd700 1 -- 172.21.15.189:0/662826094 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7eff7c0a81a0 con 0x7eff5c003730 2020-11-20T12:05:04.530 INFO:teuthology.orchestra.run.smithi189.stdout:55834574854 2020-11-20T12:05:04.532 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.527+0000 7eff7bfff700 1 -- 172.21.15.189:0/662826094 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7eff580557c0 msgr2=0x7eff58057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.532 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.527+0000 7eff7bfff700 1 --2- 172.21.15.189:0/662826094 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7eff580557c0 0x7eff58057c80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7eff64006a90 tx=0x7eff64008040).stop 2020-11-20T12:05:04.532 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.527+0000 7eff7bfff700 1 -- 172.21.15.189:0/662826094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff5c003730 msgr2=0x7eff7c13c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.532 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.527+0000 7eff7bfff700 1 --2- 172.21.15.189:0/662826094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff5c003730 0x7eff7c13c190 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7eff6c007a80 tx=0x7eff6c013000).stop 2020-11-20T12:05:04.533 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.527+0000 7eff7bfff700 1 -- 172.21.15.189:0/662826094 shutdown_connections 2020-11-20T12:05:04.533 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.527+0000 7eff7bfff700 1 --2- 172.21.15.189:0/662826094 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7eff7c0a9ab0 0x7eff7c14d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.533 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.527+0000 7eff7bfff700 1 --2- 172.21.15.189:0/662826094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff5c003730 0x7eff7c13c190 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.533 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.527+0000 7eff7bfff700 1 --2- 172.21.15.189:0/662826094 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7eff580557c0 0x7eff58057c80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.534 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.527+0000 7eff7bfff700 1 --2- 172.21.15.189:0/662826094 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7eff7c14a8a0 0x7eff7c14d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.534 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.527+0000 7eff7bfff700 1 -- 172.21.15.189:0/662826094 >> 172.21.15.189:0/662826094 conn(0x7eff7c065fe0 msgr2=0x7eff7c132dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:04.534 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.527+0000 7eff7bfff700 1 -- 172.21.15.189:0/662826094 shutdown_connections 2020-11-20T12:05:04.534 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.527+0000 7eff7bfff700 1 -- 172.21.15.189:0/662826094 wait complete. 2020-11-20T12:05:04.545 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.7 2020-11-20T12:05:04.545 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2020-11-20T12:05:04.546 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-20T12:05:04.670 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.663+0000 7fdf3341d700 1 Processor -- start 2020-11-20T12:05:04.670 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.663+0000 7fdf3341d700 1 -- start start 2020-11-20T12:05:04.671 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf3341d700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdf2c062930 0x7fdf2c0669c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.671 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf3341d700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf2c0acaa0 con 0x7fdf2c13b610 2020-11-20T12:05:04.671 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf3341d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf2c0acd40 con 0x7fdf2c066eb0 2020-11-20T12:05:04.671 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf3341d700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c0ad480 con 0x7fdf2c062930 2020-11-20T12:05:04.671 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf309b8700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdf2c066eb0 0x7fdf2c0ab7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50338/0 (socket says 172.21.15.189:50338) 2020-11-20T12:05:04.672 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf309b8700 1 -- 172.21.15.189:0/1696967028 learned_addr learned my addr 172.21.15.189:0/1696967028 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:04.672 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf311b9700 1 --2- 172.21.15.189:0/1696967028 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdf2c062930 0x7fdf2c0669c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.675 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf23fff700 1 -- 172.21.15.189:0/1696967028 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835719229 0 0) 0x7fdf2c0acd40 con 0x7fdf2c066eb0 2020-11-20T12:05:04.675 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf23fff700 1 -- 172.21.15.189:0/1696967028 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf10003240 con 0x7fdf2c066eb0 2020-11-20T12:05:04.675 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf23fff700 1 -- 172.21.15.189:0/1696967028 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 957945229 0 0) 0x7fdf2c0acaa0 con 0x7fdf2c13b610 2020-11-20T12:05:04.675 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf23fff700 1 -- 172.21.15.189:0/1696967028 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf2c0acd40 con 0x7fdf2c13b610 2020-11-20T12:05:04.676 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf311b9700 1 -- 172.21.15.189:0/1696967028 >> v1:172.21.15.189:6789/0 conn(0x7fdf2c066eb0 legacy=0x7fdf2c0ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.676 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf311b9700 1 -- 172.21.15.189:0/1696967028 >> v1:172.21.15.47:6789/0 conn(0x7fdf2c13b610 legacy=0x7fdf2c13baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.676 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf311b9700 1 -- 172.21.15.189:0/1696967028 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2c0acfe0 con 0x7fdf2c062930 2020-11-20T12:05:04.676 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf311b9700 1 --2- 172.21.15.189:0/1696967028 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdf2c062930 0x7fdf2c0669c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fdf2800b9d0 tx=0x7fdf28006df0).ready entity=mon.2 client_cookie=dc59d96a2c3dcacd server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:04.677 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf23fff700 1 -- 172.21.15.189:0/1696967028 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf28009710 con 0x7fdf2c062930 2020-11-20T12:05:04.677 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf23fff700 1 -- 172.21.15.189:0/1696967028 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf28009870 con 0x7fdf2c062930 2020-11-20T12:05:04.677 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf23fff700 1 -- 172.21.15.189:0/1696967028 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf28009bf0 con 0x7fdf2c062930 2020-11-20T12:05:04.677 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf3341d700 1 -- 172.21.15.189:0/1696967028 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdf2c062930 msgr2=0x7fdf2c0669c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.677 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf3341d700 1 --2- 172.21.15.189:0/1696967028 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdf2c062930 0x7fdf2c0669c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fdf2800b9d0 tx=0x7fdf28006df0).stop 2020-11-20T12:05:04.678 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf3341d700 1 -- 172.21.15.189:0/1696967028 shutdown_connections 2020-11-20T12:05:04.678 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf3341d700 1 --2- 172.21.15.189:0/1696967028 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdf2c062930 0x7fdf2c0669c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.678 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf3341d700 1 -- 172.21.15.189:0/1696967028 >> 172.21.15.189:0/1696967028 conn(0x7fdf2c065080 msgr2=0x7fdf2c062080 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:04.678 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf3341d700 1 -- 172.21.15.189:0/1696967028 shutdown_connections 2020-11-20T12:05:04.678 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf3341d700 1 -- 172.21.15.189:0/1696967028 wait complete. 2020-11-20T12:05:04.679 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf3341d700 1 Processor -- start 2020-11-20T12:05:04.679 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.667+0000 7fdf3341d700 1 -- start start 2020-11-20T12:05:04.680 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.671+0000 7fdf3341d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf2c066eb0 0x7fdf2c143230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.680 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.671+0000 7fdf3341d700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdf2c13b610 0x7fdf2c14e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.680 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.671+0000 7fdf3341d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf2c0a9690 0x7fdf2c1388b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.680 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.671+0000 7fdf3341d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c14df50 con 0x7fdf2c066eb0 2020-11-20T12:05:04.680 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.671+0000 7fdf3341d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c0a9fa0 con 0x7fdf2c0a9690 2020-11-20T12:05:04.681 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.671+0000 7fdf3341d700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c143750 con 0x7fdf2c13b610 2020-11-20T12:05:04.681 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.671+0000 7fdf319ba700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf2c0a9690 0x7fdf2c1388b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.681 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.671+0000 7fdf319ba700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf2c0a9690 0x7fdf2c1388b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48720/0 (socket says 172.21.15.189:48720) 2020-11-20T12:05:04.681 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.671+0000 7fdf319ba700 1 -- 172.21.15.189:0/2985506406 learned_addr learned my addr 172.21.15.189:0/2985506406 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:04.682 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.671+0000 7fdf311b9700 1 --2- 172.21.15.189:0/2985506406 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf2c066eb0 0x7fdf2c143230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.682 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.671+0000 7fdf309b8700 1 --2- 172.21.15.189:0/2985506406 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdf2c13b610 0x7fdf2c14e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.682 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.671+0000 7fdf319ba700 1 -- 172.21.15.189:0/2985506406 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdf2c13b610 msgr2=0x7fdf2c14e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.683 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.671+0000 7fdf319ba700 1 --2- 172.21.15.189:0/2985506406 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdf2c13b610 0x7fdf2c14e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.683 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.671+0000 7fdf319ba700 1 -- 172.21.15.189:0/2985506406 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf2c066eb0 msgr2=0x7fdf2c143230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.683 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.671+0000 7fdf319ba700 1 --2- 172.21.15.189:0/2985506406 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf2c066eb0 0x7fdf2c143230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.683 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.671+0000 7fdf319ba700 1 -- 172.21.15.189:0/2985506406 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2c14e3e0 con 0x7fdf2c0a9690 2020-11-20T12:05:04.684 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.671+0000 7fdf319ba700 1 --2- 172.21.15.189:0/2985506406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf2c0a9690 0x7fdf2c1388b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fdf2400cac0 tx=0x7fdf24007d90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:04.686 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.671+0000 7fdf217fa700 1 -- 172.21.15.189:0/2985506406 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2400ce90 con 0x7fdf2c0a9690 2020-11-20T12:05:04.686 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.671+0000 7fdf217fa700 1 -- 172.21.15.189:0/2985506406 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf24002240 con 0x7fdf2c0a9690 2020-11-20T12:05:04.686 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.671+0000 7fdf3341d700 1 -- 172.21.15.189:0/2985506406 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf2c14ed70 con 0x7fdf2c0a9690 2020-11-20T12:05:04.686 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.671+0000 7fdf217fa700 1 -- 172.21.15.189:0/2985506406 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf24001550 con 0x7fdf2c0a9690 2020-11-20T12:05:04.687 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.671+0000 7fdf3341d700 1 -- 172.21.15.189:0/2985506406 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf2c0aec30 con 0x7fdf2c0a9690 2020-11-20T12:05:04.687 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.671+0000 7fdf3341d700 1 -- 172.21.15.189:0/2985506406 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf2c14e570 con 0x7fdf2c0a9690 2020-11-20T12:05:04.687 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.671+0000 7fdf217fa700 1 -- 172.21.15.189:0/2985506406 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fdf24019030 con 0x7fdf2c0a9690 2020-11-20T12:05:04.687 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.675+0000 7fdf217fa700 1 --2- 172.21.15.189:0/2985506406 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fdf040557c0 0x7fdf04057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:04.688 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.675+0000 7fdf217fa700 1 -- 172.21.15.189:0/2985506406 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdf2406dcd0 con 0x7fdf2c0a9690 2020-11-20T12:05:04.688 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.675+0000 7fdf311b9700 1 --2- 172.21.15.189:0/2985506406 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fdf040557c0 0x7fdf04057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:04.688 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.675+0000 7fdf311b9700 1 --2- 172.21.15.189:0/2985506406 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fdf040557c0 0x7fdf04057c80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fdf2c0673d0 tx=0x7fdf28015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:04.688 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.675+0000 7fdf217fa700 1 -- 172.21.15.189:0/2985506406 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fdf24043b10 con 0x7fdf2c0a9690 2020-11-20T12:05:04.869 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.863+0000 7fdf3341d700 1 -- 172.21.15.189:0/2985506406 --> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdf2c067ab0 con 0x7fdf040557c0 2020-11-20T12:05:04.878 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.871+0000 7fdf217fa700 1 -- 172.21.15.189:0/2985506406 <== mgr.4105 v2:172.21.15.189:6832/11671 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+82252 (secure 0 0 0) 0x7fdf2c067ab0 con 0x7fdf040557c0 2020-11-20T12:05:04.878 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.871+0000 7fdf3341d700 1 -- 172.21.15.189:0/2985506406 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fdf040557c0 msgr2=0x7fdf04057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.879 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.871+0000 7fdf3341d700 1 --2- 172.21.15.189:0/2985506406 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fdf040557c0 0x7fdf04057c80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fdf2c0673d0 tx=0x7fdf28015000).stop 2020-11-20T12:05:04.879 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.871+0000 7fdf3341d700 1 -- 172.21.15.189:0/2985506406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf2c0a9690 msgr2=0x7fdf2c1388b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:04.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.871+0000 7fdf3341d700 1 --2- 172.21.15.189:0/2985506406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf2c0a9690 0x7fdf2c1388b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fdf2400cac0 tx=0x7fdf24007d90).stop 2020-11-20T12:05:04.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.871+0000 7fdf3341d700 1 -- 172.21.15.189:0/2985506406 shutdown_connections 2020-11-20T12:05:04.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.871+0000 7fdf3341d700 1 --2- 172.21.15.189:0/2985506406 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fdf040557c0 0x7fdf04057c80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.881 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.871+0000 7fdf3341d700 1 --2- 172.21.15.189:0/2985506406 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf2c066eb0 0x7fdf2c143230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.881 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.871+0000 7fdf3341d700 1 --2- 172.21.15.189:0/2985506406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf2c0a9690 0x7fdf2c1388b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.881 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.871+0000 7fdf3341d700 1 --2- 172.21.15.189:0/2985506406 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdf2c13b610 0x7fdf2c14e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:04.882 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.871+0000 7fdf3341d700 1 -- 172.21.15.189:0/2985506406 >> 172.21.15.189:0/2985506406 conn(0x7fdf2c065080 msgr2=0x7fdf2c062080 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:04.882 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.871+0000 7fdf3341d700 1 -- 172.21.15.189:0/2985506406 shutdown_connections 2020-11-20T12:05:04.883 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:04.871+0000 7fdf3341d700 1 -- 172.21.15.189:0/2985506406 wait complete. 2020-11-20T12:05:04.883 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2020-11-20T12:05:04.883 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:05:04.891 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2020-11-20T12:05:03.338429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2304,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40,"ondisk_log_size":40,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":1887012,"kb_used_data":747708032,"kb_used_omap":599,"kb_used_meta":0,"kb_avail":747708028,"statfs":{"total":767585320960,"available":765653020672,"internally_reserved":4096,"allocated":765653024768,"data_stored":765653024768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":613652,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.321024"},"pg_stats":[{"pgid":"4.2","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.025896+0000","last_change":"2020-11-20T12:04:54.024234+0000","last_active":"2020-11-20T12:04:54.025896+0000","last_peered":"2020-11-20T12:04:54.025896+0000","last_clean":"2020-11-20T12:04:54.025896+0000","last_became_active":"2020-11-20T12:04:54.023862+0000","last_became_peered":"2020-11-20T12:04:54.023862+0000","last_unstale":"2020-11-20T12:04:54.025896+0000","last_undegraded":"2020-11-20T12:04:54.025896+0000","last_fullsized":"2020-11-20T12:04:54.025896+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.025135+0000","last_change":"2020-11-20T12:04:51.104716+0000","last_active":"2020-11-20T12:04:54.025135+0000","last_peered":"2020-11-20T12:04:54.025135+0000","last_clean":"2020-11-20T12:04:54.025135+0000","last_became_active":"2020-11-20T12:04:51.104301+0000","last_became_peered":"2020-11-20T12:04:51.104301+0000","last_unstale":"2020-11-20T12:04:54.025135+0000","last_undegraded":"2020-11-20T12:04:54.025135+0000","last_fullsized":"2020-11-20T12:04:54.025135+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"21'6","reported_seq":"16","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.039919+0000","last_change":"2020-11-20T12:04:53.664474+0000","last_active":"2020-11-20T12:04:54.039919+0000","last_peered":"2020-11-20T12:04:54.039919+0000","last_clean":"2020-11-20T12:04:54.039919+0000","last_became_active":"2020-11-20T12:04:53.664073+0000","last_became_peered":"2020-11-20T12:04:53.664073+0000","last_unstale":"2020-11-20T12:04:54.039919+0000","last_undegraded":"2020-11-20T12:04:54.039919+0000","last_fullsized":"2020-11-20T12:04:54.039919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.037879+0000","last_change":"2020-11-20T12:04:54.033863+0000","last_active":"2020-11-20T12:04:54.037879+0000","last_peered":"2020-11-20T12:04:54.037879+0000","last_clean":"2020-11-20T12:04:54.037879+0000","last_became_active":"2020-11-20T12:04:54.032723+0000","last_became_peered":"2020-11-20T12:04:54.032723+0000","last_unstale":"2020-11-20T12:04:54.037879+0000","last_undegraded":"2020-11-20T12:04:54.037879+0000","last_fullsized":"2020-11-20T12:04:54.037879+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.018806+0000","last_change":"2020-11-20T12:04:51.107483+0000","last_active":"2020-11-20T12:04:54.018806+0000","last_peered":"2020-11-20T12:04:54.018806+0000","last_clean":"2020-11-20T12:04:54.018806+0000","last_became_active":"2020-11-20T12:04:51.106350+0000","last_became_peered":"2020-11-20T12:04:51.106350+0000","last_unstale":"2020-11-20T12:04:54.018806+0000","last_undegraded":"2020-11-20T12:04:54.018806+0000","last_fullsized":"2020-11-20T12:04:54.018806+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"21'5","reported_seq":"15","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.034249+0000","last_change":"2020-11-20T12:04:53.660754+0000","last_active":"2020-11-20T12:04:54.034249+0000","last_peered":"2020-11-20T12:04:54.034249+0000","last_clean":"2020-11-20T12:04:54.034249+0000","last_became_active":"2020-11-20T12:04:53.660354+0000","last_became_peered":"2020-11-20T12:04:53.660354+0000","last_unstale":"2020-11-20T12:04:54.034249+0000","last_undegraded":"2020-11-20T12:04:54.034249+0000","last_fullsized":"2020-11-20T12:04:54.034249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.036255+0000","last_change":"2020-11-20T12:04:54.036030+0000","last_active":"2020-11-20T12:04:54.036255+0000","last_peered":"2020-11-20T12:04:54.036255+0000","last_clean":"2020-11-20T12:04:54.036255+0000","last_became_active":"2020-11-20T12:04:54.035638+0000","last_became_peered":"2020-11-20T12:04:54.035638+0000","last_unstale":"2020-11-20T12:04:54.036255+0000","last_undegraded":"2020-11-20T12:04:54.036255+0000","last_fullsized":"2020-11-20T12:04:54.036255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.028309+0000","last_change":"2020-11-20T12:04:50.499140+0000","last_active":"2020-11-20T12:04:54.028309+0000","last_peered":"2020-11-20T12:04:54.028309+0000","last_clean":"2020-11-20T12:04:54.028309+0000","last_became_active":"2020-11-20T12:04:50.498705+0000","last_became_peered":"2020-11-20T12:04:50.498705+0000","last_unstale":"2020-11-20T12:04:54.028309+0000","last_undegraded":"2020-11-20T12:04:54.028309+0000","last_fullsized":"2020-11-20T12:04:54.028309+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"21'8","reported_seq":"18","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.040833+0000","last_change":"2020-11-20T12:04:53.657720+0000","last_active":"2020-11-20T12:04:54.040833+0000","last_peered":"2020-11-20T12:04:54.040833+0000","last_clean":"2020-11-20T12:04:54.040833+0000","last_became_active":"2020-11-20T12:04:53.657238+0000","last_became_peered":"2020-11-20T12:04:53.657238+0000","last_unstale":"2020-11-20T12:04:54.040833+0000","last_undegraded":"2020-11-20T12:04:54.040833+0000","last_fullsized":"2020-11-20T12:04:54.040833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.345488+0000","last_change":"2020-11-20T12:04:54.345320+0000","last_active":"2020-11-20T12:04:54.345488+0000","last_peered":"2020-11-20T12:04:54.345488+0000","last_clean":"2020-11-20T12:04:54.345488+0000","last_became_active":"2020-11-20T12:04:54.344995+0000","last_became_peered":"2020-11-20T12:04:54.344995+0000","last_unstale":"2020-11-20T12:04:54.345488+0000","last_undegraded":"2020-11-20T12:04:54.345488+0000","last_fullsized":"2020-11-20T12:04:54.345488+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.019834+0000","last_change":"2020-11-20T12:04:51.425413+0000","last_active":"2020-11-20T12:04:54.019834+0000","last_peered":"2020-11-20T12:04:54.019834+0000","last_clean":"2020-11-20T12:04:54.019834+0000","last_became_active":"2020-11-20T12:04:51.424595+0000","last_became_peered":"2020-11-20T12:04:51.424595+0000","last_unstale":"2020-11-20T12:04:54.019834+0000","last_undegraded":"2020-11-20T12:04:54.019834+0000","last_fullsized":"2020-11-20T12:04:54.019834+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"14'2","reported_seq":"18","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.344529+0000","last_change":"2020-11-20T12:04:44.090086+0000","last_active":"2020-11-20T12:04:54.344529+0000","last_peered":"2020-11-20T12:04:54.344529+0000","last_clean":"2020-11-20T12:04:54.344529+0000","last_became_active":"2020-11-20T12:04:44.089120+0000","last_became_peered":"2020-11-20T12:04:44.089120+0000","last_unstale":"2020-11-20T12:04:54.344529+0000","last_undegraded":"2020-11-20T12:04:54.344529+0000","last_fullsized":"2020-11-20T12:04:54.344529+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:42.003758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:42.003758+0000","last_clean_scrub_stamp":"2020-11-20T12:04:42.003758+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"21'1","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.034984+0000","last_change":"2020-11-20T12:04:53.660727+0000","last_active":"2020-11-20T12:04:54.034984+0000","last_peered":"2020-11-20T12:04:54.034984+0000","last_clean":"2020-11-20T12:04:54.034984+0000","last_became_active":"2020-11-20T12:04:53.660424+0000","last_became_peered":"2020-11-20T12:04:53.660424+0000","last_unstale":"2020-11-20T12:04:54.034984+0000","last_undegraded":"2020-11-20T12:04:54.034984+0000","last_fullsized":"2020-11-20T12:04:54.034984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.038696+0000","last_change":"2020-11-20T12:04:54.038479+0000","last_active":"2020-11-20T12:04:54.038696+0000","last_peered":"2020-11-20T12:04:54.038696+0000","last_clean":"2020-11-20T12:04:54.038696+0000","last_became_active":"2020-11-20T12:04:54.037927+0000","last_became_peered":"2020-11-20T12:04:54.037927+0000","last_unstale":"2020-11-20T12:04:54.038696+0000","last_undegraded":"2020-11-20T12:04:54.038696+0000","last_fullsized":"2020-11-20T12:04:54.038696+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.020094+0000","last_change":"2020-11-20T12:04:50.499789+0000","last_active":"2020-11-20T12:04:54.020094+0000","last_peered":"2020-11-20T12:04:54.020094+0000","last_clean":"2020-11-20T12:04:54.020094+0000","last_became_active":"2020-11-20T12:04:50.499414+0000","last_became_peered":"2020-11-20T12:04:50.499414+0000","last_unstale":"2020-11-20T12:04:54.020094+0000","last_undegraded":"2020-11-20T12:04:54.020094+0000","last_fullsized":"2020-11-20T12:04:54.020094+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"10","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.019676+0000","last_change":"2020-11-20T12:04:53.665538+0000","last_active":"2020-11-20T12:04:54.019676+0000","last_peered":"2020-11-20T12:04:54.019676+0000","last_clean":"2020-11-20T12:04:54.019676+0000","last_became_active":"2020-11-20T12:04:53.664812+0000","last_became_peered":"2020-11-20T12:04:53.664812+0000","last_unstale":"2020-11-20T12:04:54.019676+0000","last_undegraded":"2020-11-20T12:04:54.019676+0000","last_fullsized":"2020-11-20T12:04:54.019676+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.039072+0000","last_change":"2020-11-20T12:04:54.038559+0000","last_active":"2020-11-20T12:04:54.039072+0000","last_peered":"2020-11-20T12:04:54.039072+0000","last_clean":"2020-11-20T12:04:54.039072+0000","last_became_active":"2020-11-20T12:04:54.038134+0000","last_became_peered":"2020-11-20T12:04:54.038134+0000","last_unstale":"2020-11-20T12:04:54.039072+0000","last_undegraded":"2020-11-20T12:04:54.039072+0000","last_fullsized":"2020-11-20T12:04:54.039072+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.019201+0000","last_change":"2020-11-20T12:04:50.500191+0000","last_active":"2020-11-20T12:04:54.019201+0000","last_peered":"2020-11-20T12:04:54.019201+0000","last_clean":"2020-11-20T12:04:54.019201+0000","last_became_active":"2020-11-20T12:04:50.499854+0000","last_became_peered":"2020-11-20T12:04:50.499854+0000","last_unstale":"2020-11-20T12:04:54.019201+0000","last_undegraded":"2020-11-20T12:04:54.019201+0000","last_fullsized":"2020-11-20T12:04:54.019201+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"21'8","reported_seq":"18","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.042236+0000","last_change":"2020-11-20T12:04:53.665307+0000","last_active":"2020-11-20T12:04:54.042236+0000","last_peered":"2020-11-20T12:04:54.042236+0000","last_clean":"2020-11-20T12:04:54.042236+0000","last_became_active":"2020-11-20T12:04:53.664857+0000","last_became_peered":"2020-11-20T12:04:53.664857+0000","last_unstale":"2020-11-20T12:04:54.042236+0000","last_undegraded":"2020-11-20T12:04:54.042236+0000","last_fullsized":"2020-11-20T12:04:54.042236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1535,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.346166+0000","last_change":"2020-11-20T12:04:54.345851+0000","last_active":"2020-11-20T12:04:54.346166+0000","last_peered":"2020-11-20T12:04:54.346166+0000","last_clean":"2020-11-20T12:04:54.346166+0000","last_became_active":"2020-11-20T12:04:54.345040+0000","last_became_peered":"2020-11-20T12:04:54.345040+0000","last_unstale":"2020-11-20T12:04:54.346166+0000","last_undegraded":"2020-11-20T12:04:54.346166+0000","last_fullsized":"2020-11-20T12:04:54.346166+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"21'6","reported_seq":"16","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.039735+0000","last_change":"2020-11-20T12:04:53.663035+0000","last_active":"2020-11-20T12:04:54.039735+0000","last_peered":"2020-11-20T12:04:54.039735+0000","last_clean":"2020-11-20T12:04:54.039735+0000","last_became_active":"2020-11-20T12:04:53.662666+0000","last_became_peered":"2020-11-20T12:04:53.662666+0000","last_unstale":"2020-11-20T12:04:54.039735+0000","last_undegraded":"2020-11-20T12:04:54.039735+0000","last_fullsized":"2020-11-20T12:04:54.039735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.025490+0000","last_change":"2020-11-20T12:04:51.447405+0000","last_active":"2020-11-20T12:04:54.025490+0000","last_peered":"2020-11-20T12:04:54.025490+0000","last_clean":"2020-11-20T12:04:54.025490+0000","last_became_active":"2020-11-20T12:04:51.446352+0000","last_became_peered":"2020-11-20T12:04:51.446352+0000","last_unstale":"2020-11-20T12:04:54.025490+0000","last_undegraded":"2020-11-20T12:04:54.025490+0000","last_fullsized":"2020-11-20T12:04:54.025490+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.024608+0000","last_change":"2020-11-20T12:04:54.024326+0000","last_active":"2020-11-20T12:04:54.024608+0000","last_peered":"2020-11-20T12:04:54.024608+0000","last_clean":"2020-11-20T12:04:54.024608+0000","last_became_active":"2020-11-20T12:04:54.023946+0000","last_became_peered":"2020-11-20T12:04:54.023946+0000","last_unstale":"2020-11-20T12:04:54.024608+0000","last_undegraded":"2020-11-20T12:04:54.024608+0000","last_fullsized":"2020-11-20T12:04:54.024608+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.019334+0000","last_change":"2020-11-20T12:04:51.447214+0000","last_active":"2020-11-20T12:04:54.019334+0000","last_peered":"2020-11-20T12:04:54.019334+0000","last_clean":"2020-11-20T12:04:54.019334+0000","last_became_active":"2020-11-20T12:04:51.446767+0000","last_became_peered":"2020-11-20T12:04:51.446767+0000","last_unstale":"2020-11-20T12:04:54.019334+0000","last_undegraded":"2020-11-20T12:04:54.019334+0000","last_fullsized":"2020-11-20T12:04:54.019334+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"22'4","reported_seq":"14","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:55.029034+0000","last_change":"2020-11-20T12:04:53.658055+0000","last_active":"2020-11-20T12:04:55.029034+0000","last_peered":"2020-11-20T12:04:55.029034+0000","last_clean":"2020-11-20T12:04:55.029034+0000","last_became_active":"2020-11-20T12:04:53.657639+0000","last_became_peered":"2020-11-20T12:04:53.657639+0000","last_unstale":"2020-11-20T12:04:55.029034+0000","last_undegraded":"2020-11-20T12:04:55.029034+0000","last_fullsized":"2020-11-20T12:04:55.029034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235868,"kb_used_data":93463512,"kb_used_omap":71,"kb_used_meta":0,"kb_avail":93463512,"statfs":{"total":95948165120,"available":95706636288,"internally_reserved":0,"allocated":95706636288,"data_stored":95706636288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":73586,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2689999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8220000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0870000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6160000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.6509999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.4910000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":10,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235896,"kb_used_data":93463484,"kb_used_omap":97,"kb_used_meta":0,"kb_avail":93463484,"statfs":{"total":95948165120,"available":95706607616,"internally_reserved":0,"allocated":95706607616,"data_stored":95706607616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99537,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.375}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4060000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8640000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.052}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0230000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.3769999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235868,"kb_used_data":93463512,"kb_used_omap":52,"kb_used_meta":0,"kb_avail":93463512,"statfs":{"total":95948165120,"available":95706636288,"internally_reserved":0,"allocated":95706636288,"data_stored":95706636288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53960,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96799999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2320000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7909999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.734}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8940000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5329999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.889}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235872,"kb_used_data":93463508,"kb_used_omap":68,"kb_used_meta":0,"kb_avail":93463508,"statfs":{"total":95948165120,"available":95706632192,"internally_reserved":0,"allocated":95706632192,"data_stored":95706632192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":70475,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4790000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.4529999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2509999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1989999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2909999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.226}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.698}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235880,"kb_used_data":93463500,"kb_used_omap":76,"kb_used_meta":0,"kb_avail":93463500,"statfs":{"total":95948165120,"available":95706624000,"internally_reserved":0,"allocated":95706624000,"data_stored":95706624000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78075,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4870000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5920000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.02}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.8959999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.415}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.8330000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235868,"kb_used_data":93463512,"kb_used_omap":68,"kb_used_meta":0,"kb_avail":93463512,"statfs":{"total":95948165120,"available":95706636288,"internally_reserved":0,"allocated":95706636288,"data_stored":95706636288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":70478,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.379}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3420000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.827}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2229999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.4209999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.3159999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4910000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235880,"kb_used_data":93463504,"kb_used_omap":75,"kb_used_meta":0,"kb_avail":93463500,"statfs":{"total":95948165120,"available":95706624000,"internally_reserved":4096,"allocated":95706628096,"data_stored":95706628096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77160,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3679999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.75}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4580000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1890000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9199999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":7,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235880,"kb_used_data":93463500,"kb_used_omap":88,"kb_used_meta":0,"kb_avail":93463500,"statfs":{"total":95948165120,"available":95706624000,"internally_reserved":0,"allocated":95706624000,"data_stored":95706624000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90381,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.351}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7310000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.306}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4609999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6520000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7450000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1379999999999999}]}]}]}} 2020-11-20T12:05:04.893 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-20T12:05:05.035 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.031+0000 7f2b0eb50700 1 Processor -- start 2020-11-20T12:05:05.035 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.031+0000 7f2b0eb50700 1 -- start start 2020-11-20T12:05:05.037 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.031+0000 7f2b0eb50700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2b08137410 0x7f2b08137870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.037 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.031+0000 7f2b0eb50700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b08066370 con 0x7f2b08137d60 2020-11-20T12:05:05.037 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.031+0000 7f2b0eb50700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b080a9870 con 0x7f2b08060310 2020-11-20T12:05:05.037 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.031+0000 7f2b0eb50700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b08066ee0 con 0x7f2b08137410 2020-11-20T12:05:05.037 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.031+0000 7f2b07fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2b08137410 0x7f2b08137870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.038 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.031+0000 7f2b0c8ec700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2b08060310 0x7f2b080667d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50350/0 (socket says 172.21.15.189:50350) 2020-11-20T12:05:05.038 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.031+0000 7f2b0c8ec700 1 -- 172.21.15.189:0/321507271 learned_addr learned my addr 172.21.15.189:0/321507271 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:05.039 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.031+0000 7f2b077fe700 1 -- 172.21.15.189:0/321507271 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2923232437 0 0) 0x7f2b080a9870 con 0x7f2b08060310 2020-11-20T12:05:05.040 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.031+0000 7f2b077fe700 1 -- 172.21.15.189:0/321507271 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ae8003240 con 0x7f2b08060310 2020-11-20T12:05:05.040 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.031+0000 7f2b07fff700 1 -- 172.21.15.189:0/321507271 >> v1:172.21.15.189:6789/0 conn(0x7f2b08060310 legacy=0x7f2b080667d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.040 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.031+0000 7f2b07fff700 1 -- 172.21.15.189:0/321507271 >> v1:172.21.15.47:6789/0 conn(0x7f2b08137d60 legacy=0x7f2b08141ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.031+0000 7f2b07fff700 1 -- 172.21.15.189:0/321507271 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b080a95d0 con 0x7f2b08137410 2020-11-20T12:05:05.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.035+0000 7f2b07fff700 1 --2- 172.21.15.189:0/321507271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b08137410 0x7f2b08137870 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f2af000b990 tx=0x7f2af0006d90).ready entity=mon.1 client_cookie=4ba0a9919337ccbb server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:05.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.035+0000 7f2b077fe700 1 -- 172.21.15.189:0/321507271 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af0006030 con 0x7f2b08137410 2020-11-20T12:05:05.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.035+0000 7f2b077fe700 1 -- 172.21.15.189:0/321507271 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af0006190 con 0x7f2b08137410 2020-11-20T12:05:05.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.035+0000 7f2b077fe700 1 -- 172.21.15.189:0/321507271 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af0006510 con 0x7f2b08137410 2020-11-20T12:05:05.042 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.035+0000 7f2b0eb50700 1 -- 172.21.15.189:0/321507271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b08137410 msgr2=0x7f2b08137870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.042 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.035+0000 7f2b0eb50700 1 --2- 172.21.15.189:0/321507271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b08137410 0x7f2b08137870 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f2af000b990 tx=0x7f2af0006d90).stop 2020-11-20T12:05:05.042 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.035+0000 7f2b0eb50700 1 -- 172.21.15.189:0/321507271 shutdown_connections 2020-11-20T12:05:05.042 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.035+0000 7f2b0eb50700 1 --2- 172.21.15.189:0/321507271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b08137410 0x7f2b08137870 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.042 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.035+0000 7f2b0eb50700 1 -- 172.21.15.189:0/321507271 >> 172.21.15.189:0/321507271 conn(0x7f2b080627f0 msgr2=0x7f2b08065870 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:05.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.035+0000 7f2b0eb50700 1 -- 172.21.15.189:0/321507271 shutdown_connections 2020-11-20T12:05:05.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.035+0000 7f2b0eb50700 1 -- 172.21.15.189:0/321507271 wait complete. 2020-11-20T12:05:05.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.039+0000 7f2b0eb50700 1 Processor -- start 2020-11-20T12:05:05.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.039+0000 7f2b0eb50700 1 -- start start 2020-11-20T12:05:05.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.039+0000 7f2b0eb50700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2b08060310 0x7f2b0809e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.039+0000 7f2b0eb50700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b08137410 0x7f2b080a3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.039+0000 7f2b0eb50700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b08137d60 0x7f2b0814b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.039+0000 7f2b0eb50700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0809cb60 con 0x7f2b08137d60 2020-11-20T12:05:05.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.039+0000 7f2b0eb50700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b081473d0 con 0x7f2b08137410 2020-11-20T12:05:05.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.039+0000 7f2b0eb50700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0809efd0 con 0x7f2b08060310 2020-11-20T12:05:05.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.039+0000 7f2b07fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b08137410 0x7f2b080a3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.039+0000 7f2b07fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b08137410 0x7f2b080a3b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48736/0 (socket says 172.21.15.189:48736) 2020-11-20T12:05:05.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.039+0000 7f2b07fff700 1 -- 172.21.15.189:0/1401494091 learned_addr learned my addr 172.21.15.189:0/1401494091 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:05.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.039+0000 7f2b0c8ec700 1 --2- 172.21.15.189:0/1401494091 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2b08060310 0x7f2b0809e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.039+0000 7f2b0d0ed700 1 --2- 172.21.15.189:0/1401494091 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b08137d60 0x7f2b0814b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.039+0000 7f2b07fff700 1 -- 172.21.15.189:0/1401494091 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2b08060310 msgr2=0x7f2b0809e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.039+0000 7f2b07fff700 1 --2- 172.21.15.189:0/1401494091 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2b08060310 0x7f2b0809e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.039+0000 7f2b07fff700 1 -- 172.21.15.189:0/1401494091 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b08137d60 msgr2=0x7f2b0814b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.039+0000 7f2b07fff700 1 --2- 172.21.15.189:0/1401494091 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b08137d60 0x7f2b0814b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.039+0000 7f2b07fff700 1 -- 172.21.15.189:0/1401494091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b0813a990 con 0x7f2b08137410 2020-11-20T12:05:05.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.039+0000 7f2b0d0ed700 1 --2- 172.21.15.189:0/1401494091 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b08137d60 0x7f2b0814b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:05.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.039+0000 7f2b07fff700 1 --2- 172.21.15.189:0/1401494091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b08137410 0x7f2b080a3b10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f2af0006fd0 tx=0x7f2af0015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:05.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.039+0000 7f2b04ff9700 1 -- 172.21.15.189:0/1401494091 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af0021070 con 0x7f2b08137410 2020-11-20T12:05:05.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.039+0000 7f2b0eb50700 1 -- 172.21.15.189:0/1401494091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b0813ab80 con 0x7f2b08137410 2020-11-20T12:05:05.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.039+0000 7f2b04ff9700 1 -- 172.21.15.189:0/1401494091 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af0005ab0 con 0x7f2b08137410 2020-11-20T12:05:05.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.039+0000 7f2b04ff9700 1 -- 172.21.15.189:0/1401494091 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af0005e30 con 0x7f2b08137410 2020-11-20T12:05:05.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.039+0000 7f2b0eb50700 1 -- 172.21.15.189:0/1401494091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b0814b690 con 0x7f2b08137410 2020-11-20T12:05:05.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.043+0000 7f2b04ff9700 1 -- 172.21.15.189:0/1401494091 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f2af000f070 con 0x7f2b08137410 2020-11-20T12:05:05.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.043+0000 7f2b0eb50700 1 -- 172.21.15.189:0/1401494091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b0809ee20 con 0x7f2b08137410 2020-11-20T12:05:05.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.043+0000 7f2b04ff9700 1 --2- 172.21.15.189:0/1401494091 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2ae40557c0 0x7f2ae4057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.043+0000 7f2b0c8ec700 1 --2- 172.21.15.189:0/1401494091 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2ae40557c0 0x7f2ae4057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.043+0000 7f2b04ff9700 1 -- 172.21.15.189:0/1401494091 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2af0070270 con 0x7f2b08137410 2020-11-20T12:05:05.052 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.043+0000 7f2b0c8ec700 1 --2- 172.21.15.189:0/1401494091 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2ae40557c0 0x7f2ae4057c80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f2b0813e900 tx=0x7f2af8009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:05.055 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.051+0000 7f2b04ff9700 1 -- 172.21.15.189:0/1401494091 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2af0046110 con 0x7f2b08137410 2020-11-20T12:05:05.255 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.251+0000 7f2b0eb50700 1 -- 172.21.15.189:0/1401494091 --> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2b0814a9a0 con 0x7f2ae40557c0 2020-11-20T12:05:05.261 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.255+0000 7f2b04ff9700 1 -- 172.21.15.189:0/1401494091 <== mgr.4105 v2:172.21.15.189:6832/11671 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+82252 (secure 0 0 0) 0x7f2b0814a9a0 con 0x7f2ae40557c0 2020-11-20T12:05:05.262 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:05:05.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.259+0000 7f2b0eb50700 1 -- 172.21.15.189:0/1401494091 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2ae40557c0 msgr2=0x7f2ae4057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.259+0000 7f2b0eb50700 1 --2- 172.21.15.189:0/1401494091 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2ae40557c0 0x7f2ae4057c80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f2b0813e900 tx=0x7f2af8009040).stop 2020-11-20T12:05:05.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.259+0000 7f2b0eb50700 1 -- 172.21.15.189:0/1401494091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b08137410 msgr2=0x7f2b080a3b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.259+0000 7f2b0eb50700 1 --2- 172.21.15.189:0/1401494091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b08137410 0x7f2b080a3b10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f2af0006fd0 tx=0x7f2af0015000).stop 2020-11-20T12:05:05.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.259+0000 7f2b0eb50700 1 -- 172.21.15.189:0/1401494091 shutdown_connections 2020-11-20T12:05:05.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.259+0000 7f2b0eb50700 1 --2- 172.21.15.189:0/1401494091 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2ae40557c0 0x7f2ae4057c80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.259+0000 7f2b0eb50700 1 --2- 172.21.15.189:0/1401494091 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b08137d60 0x7f2b0814b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.259+0000 7f2b0eb50700 1 --2- 172.21.15.189:0/1401494091 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2b08060310 0x7f2b0809e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.259+0000 7f2b0eb50700 1 --2- 172.21.15.189:0/1401494091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b08137410 0x7f2b080a3b10 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.259+0000 7f2b0eb50700 1 -- 172.21.15.189:0/1401494091 >> 172.21.15.189:0/1401494091 conn(0x7f2b080627f0 msgr2=0x7f2b08065790 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:05.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.259+0000 7f2b0eb50700 1 -- 172.21.15.189:0/1401494091 shutdown_connections 2020-11-20T12:05:05.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.259+0000 7f2b0eb50700 1 -- 172.21.15.189:0/1401494091 wait complete. 2020-11-20T12:05:05.268 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2020-11-20T12:05:05.279 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2020-11-20T12:05:03.338429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2304,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40,"ondisk_log_size":40,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":1887012,"kb_used_data":747708032,"kb_used_omap":599,"kb_used_meta":0,"kb_avail":747708028,"statfs":{"total":767585320960,"available":765653020672,"internally_reserved":4096,"allocated":765653024768,"data_stored":765653024768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":613652,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.321024"},"pg_stats":[{"pgid":"4.2","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.025896+0000","last_change":"2020-11-20T12:04:54.024234+0000","last_active":"2020-11-20T12:04:54.025896+0000","last_peered":"2020-11-20T12:04:54.025896+0000","last_clean":"2020-11-20T12:04:54.025896+0000","last_became_active":"2020-11-20T12:04:54.023862+0000","last_became_peered":"2020-11-20T12:04:54.023862+0000","last_unstale":"2020-11-20T12:04:54.025896+0000","last_undegraded":"2020-11-20T12:04:54.025896+0000","last_fullsized":"2020-11-20T12:04:54.025896+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.025135+0000","last_change":"2020-11-20T12:04:51.104716+0000","last_active":"2020-11-20T12:04:54.025135+0000","last_peered":"2020-11-20T12:04:54.025135+0000","last_clean":"2020-11-20T12:04:54.025135+0000","last_became_active":"2020-11-20T12:04:51.104301+0000","last_became_peered":"2020-11-20T12:04:51.104301+0000","last_unstale":"2020-11-20T12:04:54.025135+0000","last_undegraded":"2020-11-20T12:04:54.025135+0000","last_fullsized":"2020-11-20T12:04:54.025135+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"21'6","reported_seq":"16","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.039919+0000","last_change":"2020-11-20T12:04:53.664474+0000","last_active":"2020-11-20T12:04:54.039919+0000","last_peered":"2020-11-20T12:04:54.039919+0000","last_clean":"2020-11-20T12:04:54.039919+0000","last_became_active":"2020-11-20T12:04:53.664073+0000","last_became_peered":"2020-11-20T12:04:53.664073+0000","last_unstale":"2020-11-20T12:04:54.039919+0000","last_undegraded":"2020-11-20T12:04:54.039919+0000","last_fullsized":"2020-11-20T12:04:54.039919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.037879+0000","last_change":"2020-11-20T12:04:54.033863+0000","last_active":"2020-11-20T12:04:54.037879+0000","last_peered":"2020-11-20T12:04:54.037879+0000","last_clean":"2020-11-20T12:04:54.037879+0000","last_became_active":"2020-11-20T12:04:54.032723+0000","last_became_peered":"2020-11-20T12:04:54.032723+0000","last_unstale":"2020-11-20T12:04:54.037879+0000","last_undegraded":"2020-11-20T12:04:54.037879+0000","last_fullsized":"2020-11-20T12:04:54.037879+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.018806+0000","last_change":"2020-11-20T12:04:51.107483+0000","last_active":"2020-11-20T12:04:54.018806+0000","last_peered":"2020-11-20T12:04:54.018806+0000","last_clean":"2020-11-20T12:04:54.018806+0000","last_became_active":"2020-11-20T12:04:51.106350+0000","last_became_peered":"2020-11-20T12:04:51.106350+0000","last_unstale":"2020-11-20T12:04:54.018806+0000","last_undegraded":"2020-11-20T12:04:54.018806+0000","last_fullsized":"2020-11-20T12:04:54.018806+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"21'5","reported_seq":"15","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.034249+0000","last_change":"2020-11-20T12:04:53.660754+0000","last_active":"2020-11-20T12:04:54.034249+0000","last_peered":"2020-11-20T12:04:54.034249+0000","last_clean":"2020-11-20T12:04:54.034249+0000","last_became_active":"2020-11-20T12:04:53.660354+0000","last_became_peered":"2020-11-20T12:04:53.660354+0000","last_unstale":"2020-11-20T12:04:54.034249+0000","last_undegraded":"2020-11-20T12:04:54.034249+0000","last_fullsized":"2020-11-20T12:04:54.034249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.036255+0000","last_change":"2020-11-20T12:04:54.036030+0000","last_active":"2020-11-20T12:04:54.036255+0000","last_peered":"2020-11-20T12:04:54.036255+0000","last_clean":"2020-11-20T12:04:54.036255+0000","last_became_active":"2020-11-20T12:04:54.035638+0000","last_became_peered":"2020-11-20T12:04:54.035638+0000","last_unstale":"2020-11-20T12:04:54.036255+0000","last_undegraded":"2020-11-20T12:04:54.036255+0000","last_fullsized":"2020-11-20T12:04:54.036255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.028309+0000","last_change":"2020-11-20T12:04:50.499140+0000","last_active":"2020-11-20T12:04:54.028309+0000","last_peered":"2020-11-20T12:04:54.028309+0000","last_clean":"2020-11-20T12:04:54.028309+0000","last_became_active":"2020-11-20T12:04:50.498705+0000","last_became_peered":"2020-11-20T12:04:50.498705+0000","last_unstale":"2020-11-20T12:04:54.028309+0000","last_undegraded":"2020-11-20T12:04:54.028309+0000","last_fullsized":"2020-11-20T12:04:54.028309+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"21'8","reported_seq":"18","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.040833+0000","last_change":"2020-11-20T12:04:53.657720+0000","last_active":"2020-11-20T12:04:54.040833+0000","last_peered":"2020-11-20T12:04:54.040833+0000","last_clean":"2020-11-20T12:04:54.040833+0000","last_became_active":"2020-11-20T12:04:53.657238+0000","last_became_peered":"2020-11-20T12:04:53.657238+0000","last_unstale":"2020-11-20T12:04:54.040833+0000","last_undegraded":"2020-11-20T12:04:54.040833+0000","last_fullsized":"2020-11-20T12:04:54.040833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.345488+0000","last_change":"2020-11-20T12:04:54.345320+0000","last_active":"2020-11-20T12:04:54.345488+0000","last_peered":"2020-11-20T12:04:54.345488+0000","last_clean":"2020-11-20T12:04:54.345488+0000","last_became_active":"2020-11-20T12:04:54.344995+0000","last_became_peered":"2020-11-20T12:04:54.344995+0000","last_unstale":"2020-11-20T12:04:54.345488+0000","last_undegraded":"2020-11-20T12:04:54.345488+0000","last_fullsized":"2020-11-20T12:04:54.345488+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.019834+0000","last_change":"2020-11-20T12:04:51.425413+0000","last_active":"2020-11-20T12:04:54.019834+0000","last_peered":"2020-11-20T12:04:54.019834+0000","last_clean":"2020-11-20T12:04:54.019834+0000","last_became_active":"2020-11-20T12:04:51.424595+0000","last_became_peered":"2020-11-20T12:04:51.424595+0000","last_unstale":"2020-11-20T12:04:54.019834+0000","last_undegraded":"2020-11-20T12:04:54.019834+0000","last_fullsized":"2020-11-20T12:04:54.019834+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"14'2","reported_seq":"18","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.344529+0000","last_change":"2020-11-20T12:04:44.090086+0000","last_active":"2020-11-20T12:04:54.344529+0000","last_peered":"2020-11-20T12:04:54.344529+0000","last_clean":"2020-11-20T12:04:54.344529+0000","last_became_active":"2020-11-20T12:04:44.089120+0000","last_became_peered":"2020-11-20T12:04:44.089120+0000","last_unstale":"2020-11-20T12:04:54.344529+0000","last_undegraded":"2020-11-20T12:04:54.344529+0000","last_fullsized":"2020-11-20T12:04:54.344529+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:42.003758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:42.003758+0000","last_clean_scrub_stamp":"2020-11-20T12:04:42.003758+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"21'1","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.034984+0000","last_change":"2020-11-20T12:04:53.660727+0000","last_active":"2020-11-20T12:04:54.034984+0000","last_peered":"2020-11-20T12:04:54.034984+0000","last_clean":"2020-11-20T12:04:54.034984+0000","last_became_active":"2020-11-20T12:04:53.660424+0000","last_became_peered":"2020-11-20T12:04:53.660424+0000","last_unstale":"2020-11-20T12:04:54.034984+0000","last_undegraded":"2020-11-20T12:04:54.034984+0000","last_fullsized":"2020-11-20T12:04:54.034984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.038696+0000","last_change":"2020-11-20T12:04:54.038479+0000","last_active":"2020-11-20T12:04:54.038696+0000","last_peered":"2020-11-20T12:04:54.038696+0000","last_clean":"2020-11-20T12:04:54.038696+0000","last_became_active":"2020-11-20T12:04:54.037927+0000","last_became_peered":"2020-11-20T12:04:54.037927+0000","last_unstale":"2020-11-20T12:04:54.038696+0000","last_undegraded":"2020-11-20T12:04:54.038696+0000","last_fullsized":"2020-11-20T12:04:54.038696+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.020094+0000","last_change":"2020-11-20T12:04:50.499789+0000","last_active":"2020-11-20T12:04:54.020094+0000","last_peered":"2020-11-20T12:04:54.020094+0000","last_clean":"2020-11-20T12:04:54.020094+0000","last_became_active":"2020-11-20T12:04:50.499414+0000","last_became_peered":"2020-11-20T12:04:50.499414+0000","last_unstale":"2020-11-20T12:04:54.020094+0000","last_undegraded":"2020-11-20T12:04:54.020094+0000","last_fullsized":"2020-11-20T12:04:54.020094+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"10","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.019676+0000","last_change":"2020-11-20T12:04:53.665538+0000","last_active":"2020-11-20T12:04:54.019676+0000","last_peered":"2020-11-20T12:04:54.019676+0000","last_clean":"2020-11-20T12:04:54.019676+0000","last_became_active":"2020-11-20T12:04:53.664812+0000","last_became_peered":"2020-11-20T12:04:53.664812+0000","last_unstale":"2020-11-20T12:04:54.019676+0000","last_undegraded":"2020-11-20T12:04:54.019676+0000","last_fullsized":"2020-11-20T12:04:54.019676+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.039072+0000","last_change":"2020-11-20T12:04:54.038559+0000","last_active":"2020-11-20T12:04:54.039072+0000","last_peered":"2020-11-20T12:04:54.039072+0000","last_clean":"2020-11-20T12:04:54.039072+0000","last_became_active":"2020-11-20T12:04:54.038134+0000","last_became_peered":"2020-11-20T12:04:54.038134+0000","last_unstale":"2020-11-20T12:04:54.039072+0000","last_undegraded":"2020-11-20T12:04:54.039072+0000","last_fullsized":"2020-11-20T12:04:54.039072+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.019201+0000","last_change":"2020-11-20T12:04:50.500191+0000","last_active":"2020-11-20T12:04:54.019201+0000","last_peered":"2020-11-20T12:04:54.019201+0000","last_clean":"2020-11-20T12:04:54.019201+0000","last_became_active":"2020-11-20T12:04:50.499854+0000","last_became_peered":"2020-11-20T12:04:50.499854+0000","last_unstale":"2020-11-20T12:04:54.019201+0000","last_undegraded":"2020-11-20T12:04:54.019201+0000","last_fullsized":"2020-11-20T12:04:54.019201+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"21'8","reported_seq":"18","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.042236+0000","last_change":"2020-11-20T12:04:53.665307+0000","last_active":"2020-11-20T12:04:54.042236+0000","last_peered":"2020-11-20T12:04:54.042236+0000","last_clean":"2020-11-20T12:04:54.042236+0000","last_became_active":"2020-11-20T12:04:53.664857+0000","last_became_peered":"2020-11-20T12:04:53.664857+0000","last_unstale":"2020-11-20T12:04:54.042236+0000","last_undegraded":"2020-11-20T12:04:54.042236+0000","last_fullsized":"2020-11-20T12:04:54.042236+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1535,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.346166+0000","last_change":"2020-11-20T12:04:54.345851+0000","last_active":"2020-11-20T12:04:54.346166+0000","last_peered":"2020-11-20T12:04:54.346166+0000","last_clean":"2020-11-20T12:04:54.346166+0000","last_became_active":"2020-11-20T12:04:54.345040+0000","last_became_peered":"2020-11-20T12:04:54.345040+0000","last_unstale":"2020-11-20T12:04:54.346166+0000","last_undegraded":"2020-11-20T12:04:54.346166+0000","last_fullsized":"2020-11-20T12:04:54.346166+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"21'6","reported_seq":"16","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.039735+0000","last_change":"2020-11-20T12:04:53.663035+0000","last_active":"2020-11-20T12:04:54.039735+0000","last_peered":"2020-11-20T12:04:54.039735+0000","last_clean":"2020-11-20T12:04:54.039735+0000","last_became_active":"2020-11-20T12:04:53.662666+0000","last_became_peered":"2020-11-20T12:04:53.662666+0000","last_unstale":"2020-11-20T12:04:54.039735+0000","last_undegraded":"2020-11-20T12:04:54.039735+0000","last_fullsized":"2020-11-20T12:04:54.039735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.025490+0000","last_change":"2020-11-20T12:04:51.447405+0000","last_active":"2020-11-20T12:04:54.025490+0000","last_peered":"2020-11-20T12:04:54.025490+0000","last_clean":"2020-11-20T12:04:54.025490+0000","last_became_active":"2020-11-20T12:04:51.446352+0000","last_became_peered":"2020-11-20T12:04:51.446352+0000","last_unstale":"2020-11-20T12:04:54.025490+0000","last_undegraded":"2020-11-20T12:04:54.025490+0000","last_fullsized":"2020-11-20T12:04:54.025490+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.024608+0000","last_change":"2020-11-20T12:04:54.024326+0000","last_active":"2020-11-20T12:04:54.024608+0000","last_peered":"2020-11-20T12:04:54.024608+0000","last_clean":"2020-11-20T12:04:54.024608+0000","last_became_active":"2020-11-20T12:04:54.023946+0000","last_became_peered":"2020-11-20T12:04:54.023946+0000","last_unstale":"2020-11-20T12:04:54.024608+0000","last_undegraded":"2020-11-20T12:04:54.024608+0000","last_fullsized":"2020-11-20T12:04:54.024608+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.019334+0000","last_change":"2020-11-20T12:04:51.447214+0000","last_active":"2020-11-20T12:04:54.019334+0000","last_peered":"2020-11-20T12:04:54.019334+0000","last_clean":"2020-11-20T12:04:54.019334+0000","last_became_active":"2020-11-20T12:04:51.446767+0000","last_became_peered":"2020-11-20T12:04:51.446767+0000","last_unstale":"2020-11-20T12:04:54.019334+0000","last_undegraded":"2020-11-20T12:04:54.019334+0000","last_fullsized":"2020-11-20T12:04:54.019334+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"22'4","reported_seq":"14","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:55.029034+0000","last_change":"2020-11-20T12:04:53.658055+0000","last_active":"2020-11-20T12:04:55.029034+0000","last_peered":"2020-11-20T12:04:55.029034+0000","last_clean":"2020-11-20T12:04:55.029034+0000","last_became_active":"2020-11-20T12:04:53.657639+0000","last_became_peered":"2020-11-20T12:04:53.657639+0000","last_unstale":"2020-11-20T12:04:55.029034+0000","last_undegraded":"2020-11-20T12:04:55.029034+0000","last_fullsized":"2020-11-20T12:04:55.029034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235868,"kb_used_data":93463512,"kb_used_omap":71,"kb_used_meta":0,"kb_avail":93463512,"statfs":{"total":95948165120,"available":95706636288,"internally_reserved":0,"allocated":95706636288,"data_stored":95706636288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":73586,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2689999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8220000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0870000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6160000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.6509999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.4910000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":10,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235896,"kb_used_data":93463484,"kb_used_omap":97,"kb_used_meta":0,"kb_avail":93463484,"statfs":{"total":95948165120,"available":95706607616,"internally_reserved":0,"allocated":95706607616,"data_stored":95706607616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":99537,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.375}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4060000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8640000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.052}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0230000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.3769999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235868,"kb_used_data":93463512,"kb_used_omap":52,"kb_used_meta":0,"kb_avail":93463512,"statfs":{"total":95948165120,"available":95706636288,"internally_reserved":0,"allocated":95706636288,"data_stored":95706636288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53960,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96799999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2320000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7909999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.734}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8940000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5329999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.889}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235872,"kb_used_data":93463508,"kb_used_omap":68,"kb_used_meta":0,"kb_avail":93463508,"statfs":{"total":95948165120,"available":95706632192,"internally_reserved":0,"allocated":95706632192,"data_stored":95706632192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":70475,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4790000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.4529999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2509999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1989999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2909999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.226}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.698}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235880,"kb_used_data":93463500,"kb_used_omap":76,"kb_used_meta":0,"kb_avail":93463500,"statfs":{"total":95948165120,"available":95706624000,"internally_reserved":0,"allocated":95706624000,"data_stored":95706624000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78075,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4870000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5920000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.02}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.8959999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.415}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.8330000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235868,"kb_used_data":93463512,"kb_used_omap":68,"kb_used_meta":0,"kb_avail":93463512,"statfs":{"total":95948165120,"available":95706636288,"internally_reserved":0,"allocated":95706636288,"data_stored":95706636288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":70478,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.379}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3420000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.827}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2229999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.4209999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.3159999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4910000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235880,"kb_used_data":93463504,"kb_used_omap":75,"kb_used_meta":0,"kb_avail":93463500,"statfs":{"total":95948165120,"available":95706624000,"internally_reserved":4096,"allocated":95706628096,"data_stored":95706628096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77160,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3679999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.75}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4580000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1890000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9199999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":7,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235880,"kb_used_data":93463500,"kb_used_omap":88,"kb_used_meta":0,"kb_avail":93463500,"statfs":{"total":95948165120,"available":95706624000,"internally_reserved":0,"allocated":95706624000,"data_stored":95706624000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":90381,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.351}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7310000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.306}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4609999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6520000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7450000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1379999999999999}]}]}]}} 2020-11-20T12:05:05.281 INFO:tasks.ceph.ceph_manager.ceph:clean! 2020-11-20T12:05:05.281 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2020-11-20T12:05:05.282 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2020-11-20T12:05:05.282 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2020-11-20T12:05:05.397 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.391+0000 7f4cade34700 1 Processor -- start 2020-11-20T12:05:05.398 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.391+0000 7f4cade34700 1 -- start start 2020-11-20T12:05:05.400 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4cade34700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4ca8063a50 0x7f4ca80682e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.401 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4cade34700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4ca8063e30 0x7f4ca813a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.401 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4cade34700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4ca813aec0 0x7f4ca80a7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.401 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4cade34700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca80687d0 con 0x7f4ca813aec0 2020-11-20T12:05:05.402 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4cade34700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca813b340 con 0x7f4ca8063a50 2020-11-20T12:05:05.402 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4cade34700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca80a8550 con 0x7f4ca8063e30 2020-11-20T12:05:05.402 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4ca77fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4ca8063a50 0x7f4ca80682e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.402 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4ca77fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4ca8063a50 0x7f4ca80682e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48742/0 (socket says 172.21.15.189:48742) 2020-11-20T12:05:05.403 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4ca77fe700 1 -- 172.21.15.189:0/645136452 learned_addr learned my addr 172.21.15.189:0/645136452 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:05.403 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4ca6ffd700 1 --2- 172.21.15.189:0/645136452 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4ca8063e30 0x7f4ca813a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.403 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4ca7fff700 1 --2- 172.21.15.189:0/645136452 >> v2:172.21.15.47:3300/0 conn(0x7f4ca813aec0 0x7f4ca80a7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.403 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4ca77fe700 1 -- 172.21.15.189:0/645136452 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4ca8063e30 msgr2=0x7f4ca813a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:05.404 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4ca77fe700 1 --2- 172.21.15.189:0/645136452 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4ca8063e30 0x7f4ca813a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.404 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4ca77fe700 1 -- 172.21.15.189:0/645136452 >> v2:172.21.15.47:3300/0 conn(0x7f4ca813aec0 msgr2=0x7f4ca80a7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:05.404 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4ca77fe700 1 --2- 172.21.15.189:0/645136452 >> v2:172.21.15.47:3300/0 conn(0x7f4ca813aec0 0x7f4ca80a7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.404 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4ca77fe700 1 -- 172.21.15.189:0/645136452 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca80a87f0 con 0x7f4ca8063a50 2020-11-20T12:05:05.405 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4ca77fe700 1 --2- 172.21.15.189:0/645136452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ca8063a50 0x7f4ca80682e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f4ca80606b0 tx=0x7f4c98006eb0).ready entity=mon.1 client_cookie=4c0b0dd8a3491eae server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:05.405 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4ca67fc700 1 -- 172.21.15.189:0/645136452 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c980098a0 con 0x7f4ca8063a50 2020-11-20T12:05:05.405 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4ca67fc700 1 -- 172.21.15.189:0/645136452 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c98009a00 con 0x7f4ca8063a50 2020-11-20T12:05:05.405 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4ca67fc700 1 -- 172.21.15.189:0/645136452 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c98009d80 con 0x7f4ca8063a50 2020-11-20T12:05:05.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4cade34700 1 -- 172.21.15.189:0/645136452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ca8063a50 msgr2=0x7f4ca80682e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4cade34700 1 --2- 172.21.15.189:0/645136452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ca8063a50 0x7f4ca80682e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f4ca80606b0 tx=0x7f4c98006eb0).stop 2020-11-20T12:05:05.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4cade34700 1 -- 172.21.15.189:0/645136452 shutdown_connections 2020-11-20T12:05:05.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4cade34700 1 --2- 172.21.15.189:0/645136452 >> v2:172.21.15.47:3300/0 conn(0x7f4ca813aec0 0x7f4ca80a7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4cade34700 1 --2- 172.21.15.189:0/645136452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ca8063a50 0x7f4ca80682e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4cade34700 1 --2- 172.21.15.189:0/645136452 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4ca8063e30 0x7f4ca813a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4cade34700 1 -- 172.21.15.189:0/645136452 >> 172.21.15.189:0/645136452 conn(0x7f4ca80ad8e0 msgr2=0x7f4ca80adcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:05.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4cade34700 1 -- 172.21.15.189:0/645136452 shutdown_connections 2020-11-20T12:05:05.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4cade34700 1 -- 172.21.15.189:0/645136452 wait complete. 2020-11-20T12:05:05.410 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4cade34700 1 Processor -- start 2020-11-20T12:05:05.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4cade34700 1 -- start start 2020-11-20T12:05:05.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.395+0000 7f4cade34700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ca8063a50 0x7f4ca809e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4cade34700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ca8063e30 0x7f4ca8148aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.412 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4cade34700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4ca813aec0 0x7f4ca8148f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.412 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4cade34700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca81380d0 con 0x7f4ca8063e30 2020-11-20T12:05:05.413 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4cade34700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca80ad620 con 0x7f4ca8063a50 2020-11-20T12:05:05.413 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4cade34700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca81376f0 con 0x7f4ca813aec0 2020-11-20T12:05:05.413 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4ca77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ca8063a50 0x7f4ca809e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.413 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4ca77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ca8063a50 0x7f4ca809e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48748/0 (socket says 172.21.15.189:48748) 2020-11-20T12:05:05.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4ca77fe700 1 -- 172.21.15.189:0/1922414429 learned_addr learned my addr 172.21.15.189:0/1922414429 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:05.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4ca6ffd700 1 --2- 172.21.15.189:0/1922414429 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ca8063e30 0x7f4ca8148aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4ca7fff700 1 --2- 172.21.15.189:0/1922414429 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4ca813aec0 0x7f4ca8148f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4ca77fe700 1 -- 172.21.15.189:0/1922414429 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4ca813aec0 msgr2=0x7f4ca8148f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4ca77fe700 1 --2- 172.21.15.189:0/1922414429 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4ca813aec0 0x7f4ca8148f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4ca77fe700 1 -- 172.21.15.189:0/1922414429 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ca8063e30 msgr2=0x7f4ca8148aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4ca77fe700 1 --2- 172.21.15.189:0/1922414429 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ca8063e30 0x7f4ca8148aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4ca77fe700 1 -- 172.21.15.189:0/1922414429 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca8137990 con 0x7f4ca8063a50 2020-11-20T12:05:05.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4ca6ffd700 1 --2- 172.21.15.189:0/1922414429 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ca8063e30 0x7f4ca8148aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:05.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4ca77fe700 1 --2- 172.21.15.189:0/1922414429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ca8063a50 0x7f4ca809e6d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f4c98009fd0 tx=0x7f4c98015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:05.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4c877fe700 1 -- 172.21.15.189:0/1922414429 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9801d5d0 con 0x7f4ca8063a50 2020-11-20T12:05:05.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4c877fe700 1 -- 172.21.15.189:0/1922414429 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9801d730 con 0x7f4ca8063a50 2020-11-20T12:05:05.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4cade34700 1 -- 172.21.15.189:0/1922414429 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca8137c30 con 0x7f4ca8063a50 2020-11-20T12:05:05.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4c877fe700 1 -- 172.21.15.189:0/1922414429 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9801daf0 con 0x7f4ca8063a50 2020-11-20T12:05:05.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4cade34700 1 -- 172.21.15.189:0/1922414429 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca809c940 con 0x7f4ca8063a50 2020-11-20T12:05:05.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4cade34700 1 -- 172.21.15.189:0/1922414429 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca8137dc0 con 0x7f4ca8063a50 2020-11-20T12:05:05.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4c877fe700 1 -- 172.21.15.189:0/1922414429 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f4c98025030 con 0x7f4ca8063a50 2020-11-20T12:05:05.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4c877fe700 1 --2- 172.21.15.189:0/1922414429 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4c800557c0 0x7f4c80057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4c877fe700 1 -- 172.21.15.189:0/1922414429 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4c9806feb0 con 0x7f4ca8063a50 2020-11-20T12:05:05.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4ca6ffd700 1 --2- 172.21.15.189:0/1922414429 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4c800557c0 0x7f4c80057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.399+0000 7f4ca6ffd700 1 --2- 172.21.15.189:0/1922414429 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4c800557c0 0x7f4c80057c80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f4c9c006bd0 tx=0x7f4c9c008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:05.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.407+0000 7f4c877fe700 1 -- 172.21.15.189:0/1922414429 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4c98045d50 con 0x7f4ca8063a50 2020-11-20T12:05:05.618 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.615+0000 7f4cade34700 1 -- 172.21.15.189:0/1922414429 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4ca80a4a20 con 0x7f4ca8063a50 2020-11-20T12:05:05.619 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.615+0000 7f4c877fe700 1 -- 172.21.15.189:0/1922414429 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+45 (secure 0 0 0) 0x7f4c98023070 con 0x7f4ca8063a50 2020-11-20T12:05:05.619 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:05:05.621 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.615+0000 7f4cade34700 1 -- 172.21.15.189:0/1922414429 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4c800557c0 msgr2=0x7f4c80057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.621 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.615+0000 7f4cade34700 1 --2- 172.21.15.189:0/1922414429 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4c800557c0 0x7f4c80057c80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f4c9c006bd0 tx=0x7f4c9c008040).stop 2020-11-20T12:05:05.621 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.615+0000 7f4cade34700 1 -- 172.21.15.189:0/1922414429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ca8063a50 msgr2=0x7f4ca809e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.621 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.615+0000 7f4cade34700 1 --2- 172.21.15.189:0/1922414429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ca8063a50 0x7f4ca809e6d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f4c98009fd0 tx=0x7f4c98015000).stop 2020-11-20T12:05:05.621 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.615+0000 7f4cade34700 1 -- 172.21.15.189:0/1922414429 shutdown_connections 2020-11-20T12:05:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.615+0000 7f4cade34700 1 --2- 172.21.15.189:0/1922414429 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4ca813aec0 0x7f4ca8148f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.615+0000 7f4cade34700 1 --2- 172.21.15.189:0/1922414429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ca8063a50 0x7f4ca809e6d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.615+0000 7f4cade34700 1 --2- 172.21.15.189:0/1922414429 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4c800557c0 0x7f4c80057c80 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.615+0000 7f4cade34700 1 --2- 172.21.15.189:0/1922414429 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4ca8063e30 0x7f4ca8148aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.615+0000 7f4cade34700 1 -- 172.21.15.189:0/1922414429 >> 172.21.15.189:0/1922414429 conn(0x7f4ca80ad8e0 msgr2=0x7f4ca80adcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:05.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.615+0000 7f4cade34700 1 -- 172.21.15.189:0/1922414429 shutdown_connections 2020-11-20T12:05:05.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.615+0000 7f4cade34700 1 -- 172.21.15.189:0/1922414429 wait complete. 2020-11-20T12:05:05.634 INFO:teuthology.orchestra.run.smithi189.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-11-20T12:05:05.635 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2020-11-20T12:05:05.635 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:05:05.774 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.767+0000 7ff58f59e700 1 Processor -- start 2020-11-20T12:05:05.774 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.767+0000 7ff58f59e700 1 -- start start 2020-11-20T12:05:05.775 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.771+0000 7ff58f59e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff5900aeb80 0x7ff59013a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.776 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.771+0000 7ff58f59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59013dcb0 con 0x7ff5900aef60 2020-11-20T12:05:05.776 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.771+0000 7ff58f59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59013df10 con 0x7ff59013e650 2020-11-20T12:05:05.776 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.771+0000 7ff58f59e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff59013eab0 con 0x7ff5900aeb80 2020-11-20T12:05:05.776 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.771+0000 7ff58e59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff5900aeb80 0x7ff59013a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.776 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.771+0000 7ff58ed9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff59013e650 0x7ff5900a1960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50382/0 (socket says 172.21.15.189:50382) 2020-11-20T12:05:05.777 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.771+0000 7ff58ed9d700 1 -- 172.21.15.189:0/3219457090 learned_addr learned my addr 172.21.15.189:0/3219457090 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:05.778 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.771+0000 7ff58e59c700 1 -- 172.21.15.189:0/3219457090 >> v1:172.21.15.189:6789/0 conn(0x7ff59013e650 legacy=0x7ff5900a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.778 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.771+0000 7ff58e59c700 1 -- 172.21.15.189:0/3219457090 >> v1:172.21.15.47:6789/0 conn(0x7ff5900aef60 legacy=0x7ff59013ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.779 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.771+0000 7ff58d59a700 1 -- 172.21.15.189:0/3219457090 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474978499 0 0) 0x7ff59013df10 con 0x7ff59013e650 2020-11-20T12:05:05.779 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.771+0000 7ff58e59c700 1 -- 172.21.15.189:0/3219457090 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff59013b4f0 con 0x7ff5900aeb80 2020-11-20T12:05:05.779 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.771+0000 7ff58e59c700 1 --2- 172.21.15.189:0/3219457090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5900aeb80 0x7ff59013a640 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7ff58400b9a0 tx=0x7ff584006df0).ready entity=mon.1 client_cookie=bc051dbc77b4dc0a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:05.779 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.771+0000 7ff58d59a700 1 -- 172.21.15.189:0/3219457090 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5840097b0 con 0x7ff5900aeb80 2020-11-20T12:05:05.779 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.771+0000 7ff58d59a700 1 -- 172.21.15.189:0/3219457090 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff584009910 con 0x7ff5900aeb80 2020-11-20T12:05:05.780 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.771+0000 7ff58d59a700 1 -- 172.21.15.189:0/3219457090 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff584009c90 con 0x7ff5900aeb80 2020-11-20T12:05:05.780 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.771+0000 7ff58f59e700 1 -- 172.21.15.189:0/3219457090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5900aeb80 msgr2=0x7ff59013a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.780 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.771+0000 7ff58f59e700 1 --2- 172.21.15.189:0/3219457090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5900aeb80 0x7ff59013a640 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7ff58400b9a0 tx=0x7ff584006df0).stop 2020-11-20T12:05:05.780 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.771+0000 7ff58f59e700 1 -- 172.21.15.189:0/3219457090 shutdown_connections 2020-11-20T12:05:05.780 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.771+0000 7ff58f59e700 1 --2- 172.21.15.189:0/3219457090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5900aeb80 0x7ff59013a640 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.781 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.771+0000 7ff58f59e700 1 -- 172.21.15.189:0/3219457090 >> 172.21.15.189:0/3219457090 conn(0x7ff590062080 msgr2=0x7ff590062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:05.781 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.771+0000 7ff58f59e700 1 -- 172.21.15.189:0/3219457090 shutdown_connections 2020-11-20T12:05:05.781 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.771+0000 7ff58f59e700 1 -- 172.21.15.189:0/3219457090 wait complete. 2020-11-20T12:05:05.783 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.775+0000 7ff58f59e700 1 Processor -- start 2020-11-20T12:05:05.783 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.775+0000 7ff58f59e700 1 -- start start 2020-11-20T12:05:05.783 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.775+0000 7ff58f59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5900aef60 0x7ff590064030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.775+0000 7ff58f59e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff59013e650 0x7ff5900a7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.775+0000 7ff58f59e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff59014e0d0 0x7ff590064e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.775+0000 7ff58f59e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5900a7920 con 0x7ff59014e0d0 2020-11-20T12:05:05.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.775+0000 7ff58f59e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff590063b90 con 0x7ff5900aef60 2020-11-20T12:05:05.785 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.775+0000 7ff58f59e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5901382b0 con 0x7ff59013e650 2020-11-20T12:05:05.785 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.775+0000 7ff58e59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5900aef60 0x7ff590064030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.785 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.775+0000 7ff58e59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5900aef60 0x7ff590064030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48766/0 (socket says 172.21.15.189:48766) 2020-11-20T12:05:05.785 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.775+0000 7ff58e59c700 1 -- 172.21.15.189:0/3486666562 learned_addr learned my addr 172.21.15.189:0/3486666562 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:05.785 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.775+0000 7ff58ed9d700 1 --2- 172.21.15.189:0/3486666562 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff59014e0d0 0x7ff590064e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.775+0000 7ff58dd9b700 1 --2- 172.21.15.189:0/3486666562 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff59013e650 0x7ff5900a7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.775+0000 7ff58e59c700 1 -- 172.21.15.189:0/3486666562 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff59013e650 msgr2=0x7ff5900a7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.775+0000 7ff58e59c700 1 --2- 172.21.15.189:0/3486666562 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff59013e650 0x7ff5900a7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.775+0000 7ff58e59c700 1 -- 172.21.15.189:0/3486666562 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff59014e0d0 msgr2=0x7ff590064e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.775+0000 7ff58e59c700 1 --2- 172.21.15.189:0/3486666562 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff59014e0d0 0x7ff590064e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.775+0000 7ff58e59c700 1 -- 172.21.15.189:0/3486666562 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff59014e550 con 0x7ff5900aef60 2020-11-20T12:05:05.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.775+0000 7ff58e59c700 1 --2- 172.21.15.189:0/3486666562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5900aef60 0x7ff590064030 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7ff5900ac720 tx=0x7ff584015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:05.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.775+0000 7ff57e7fc700 1 -- 172.21.15.189:0/3486666562 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff584011e00 con 0x7ff5900aef60 2020-11-20T12:05:05.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.779+0000 7ff57e7fc700 1 -- 172.21.15.189:0/3486666562 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58401e460 con 0x7ff5900aef60 2020-11-20T12:05:05.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.779+0000 7ff58f59e700 1 -- 172.21.15.189:0/3486666562 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff590065360 con 0x7ff5900aef60 2020-11-20T12:05:05.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.779+0000 7ff57e7fc700 1 -- 172.21.15.189:0/3486666562 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58401e7e0 con 0x7ff5900aef60 2020-11-20T12:05:05.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.779+0000 7ff58f59e700 1 -- 172.21.15.189:0/3486666562 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff590135200 con 0x7ff5900aef60 2020-11-20T12:05:05.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.779+0000 7ff57e7fc700 1 -- 172.21.15.189:0/3486666562 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7ff584025030 con 0x7ff5900aef60 2020-11-20T12:05:05.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.779+0000 7ff58f59e700 1 -- 172.21.15.189:0/3486666562 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5900a7770 con 0x7ff5900aef60 2020-11-20T12:05:05.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.779+0000 7ff57e7fc700 1 --2- 172.21.15.189:0/3486666562 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff5640557f0 0x7ff564057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:05.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.779+0000 7ff58dd9b700 1 --2- 172.21.15.189:0/3486666562 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff5640557f0 0x7ff564057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:05.790 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.779+0000 7ff57e7fc700 1 -- 172.21.15.189:0/3486666562 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff58406ff20 con 0x7ff5900aef60 2020-11-20T12:05:05.790 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.779+0000 7ff58dd9b700 1 --2- 172.21.15.189:0/3486666562 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff5640557f0 0x7ff564057cb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7ff59013d320 tx=0x7ff580009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:05.790 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.783+0000 7ff57e7fc700 1 -- 172.21.15.189:0/3486666562 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff584045dc0 con 0x7ff5900aef60 2020-11-20T12:05:05.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.939+0000 7ff58f59e700 1 -- 172.21.15.189:0/3486666562 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5900a7a90 con 0x7ff5900aef60 2020-11-20T12:05:05.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.939+0000 7ff57e7fc700 1 -- 172.21.15.189:0/3486666562 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16826 (secure 0 0 0) 0x7ff5840493e0 con 0x7ff5900aef60 2020-11-20T12:05:05.946 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:05:05.946 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":22,"fsid":"cda18d0e-64b3-41bb-964a-084069c3e648","created":"2020-11-20T12:04:37.538706+0000","modified":"2020-11-20T12:04:55.015818+0000","last_up_change":"2020-11-20T12:04:43.006631+0000","last_in_change":"2020-11-20T12:04:43.006631+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:41.395395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:48.550454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:52.101837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:53.018196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c81214e1-4f84-4aaa-8ceb-99c398b2a89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6811","nonce":11760},{"type":"v1","addr":"172.21.15.189:6814","nonce":11760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":11760},{"type":"v1","addr":"172.21.15.189:6818","nonce":11760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":11760},{"type":"v1","addr":"172.21.15.189:6828","nonce":11760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6821","nonce":11760},{"type":"v1","addr":"172.21.15.189:6824","nonce":11760}]},"public_addr":"172.21.15.189:6814/11760","cluster_addr":"172.21.15.189:6818/11760","heartbeat_back_addr":"172.21.15.189:6828/11760","heartbeat_front_addr":"172.21.15.189:6824/11760","state":["exists","up"]},{"osd":1,"uuid":"d2f3ec7d-d403-484f-b5ed-b2e6b079d63d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6805","nonce":11761},{"type":"v1","addr":"172.21.15.189:6807","nonce":11761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":11761},{"type":"v1","addr":"172.21.15.189:6812","nonce":11761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6817","nonce":11761},{"type":"v1","addr":"172.21.15.189:6820","nonce":11761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6813","nonce":11761},{"type":"v1","addr":"172.21.15.189:6815","nonce":11761}]},"public_addr":"172.21.15.189:6807/11761","cluster_addr":"172.21.15.189:6812/11761","heartbeat_back_addr":"172.21.15.189:6820/11761","heartbeat_front_addr":"172.21.15.189:6815/11761","state":["exists","up"]},{"osd":2,"uuid":"e6cf4790-000c-43eb-8844-175f04c60503","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6819","nonce":11751},{"type":"v1","addr":"172.21.15.189:6822","nonce":11751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":11751},{"type":"v1","addr":"172.21.15.189:6825","nonce":11751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":11751},{"type":"v1","addr":"172.21.15.189:6831","nonce":11751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6827","nonce":11751},{"type":"v1","addr":"172.21.15.189:6829","nonce":11751}]},"public_addr":"172.21.15.189:6822/11751","cluster_addr":"172.21.15.189:6825/11751","heartbeat_back_addr":"172.21.15.189:6831/11751","heartbeat_front_addr":"172.21.15.189:6829/11751","state":["exists","up"]},{"osd":3,"uuid":"ba49dfac-f2b9-476e-853d-42f05e21a77a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":11762},{"type":"v1","addr":"172.21.15.189:6801","nonce":11762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":11762},{"type":"v1","addr":"172.21.15.189:6803","nonce":11762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":11762},{"type":"v1","addr":"172.21.15.189:6809","nonce":11762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":11762},{"type":"v1","addr":"172.21.15.189:6806","nonce":11762}]},"public_addr":"172.21.15.189:6801/11762","cluster_addr":"172.21.15.189:6803/11762","heartbeat_back_addr":"172.21.15.189:6809/11762","heartbeat_front_addr":"172.21.15.189:6806/11762","state":["exists","up"]},{"osd":4,"uuid":"3ca7b438-38b2-4689-8b18-fb5a8ddf6aee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":12114},{"type":"v1","addr":"172.21.15.47:6825","nonce":12114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":12114},{"type":"v1","addr":"172.21.15.47:6827","nonce":12114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":12114},{"type":"v1","addr":"172.21.15.47:6831","nonce":12114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":12114},{"type":"v1","addr":"172.21.15.47:6829","nonce":12114}]},"public_addr":"172.21.15.47:6825/12114","cluster_addr":"172.21.15.47:6827/12114","heartbeat_back_addr":"172.21.15.47:6831/12114","heartbeat_front_addr":"172.21.15.47:6829/12114","state":["exists","up"]},{"osd":5,"uuid":"6a45fca7-83d3-44fd-a1d4-b01e4201a101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":12111},{"type":"v1","addr":"172.21.15.47:6801","nonce":12111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":12111},{"type":"v1","addr":"172.21.15.47:6803","nonce":12111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":12111},{"type":"v1","addr":"172.21.15.47:6807","nonce":12111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":12111},{"type":"v1","addr":"172.21.15.47:6805","nonce":12111}]},"public_addr":"172.21.15.47:6801/12111","cluster_addr":"172.21.15.47:6803/12111","heartbeat_back_addr":"172.21.15.47:6807/12111","heartbeat_front_addr":"172.21.15.47:6805/12111","state":["exists","up"]},{"osd":6,"uuid":"c4b1b9ed-61f8-4677-9755-dff59a30e447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":12113},{"type":"v1","addr":"172.21.15.47:6817","nonce":12113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":12113},{"type":"v1","addr":"172.21.15.47:6819","nonce":12113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":12113},{"type":"v1","addr":"172.21.15.47:6823","nonce":12113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":12113},{"type":"v1","addr":"172.21.15.47:6821","nonce":12113}]},"public_addr":"172.21.15.47:6817/12113","cluster_addr":"172.21.15.47:6819/12113","heartbeat_back_addr":"172.21.15.47:6823/12113","heartbeat_front_addr":"172.21.15.47:6821/12113","state":["exists","up"]},{"osd":7,"uuid":"4608e4ed-cc40-4c5c-ba8b-3779bbc62f65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":12112},{"type":"v1","addr":"172.21.15.47:6809","nonce":12112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":12112},{"type":"v1","addr":"172.21.15.47:6811","nonce":12112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":12112},{"type":"v1","addr":"172.21.15.47:6815","nonce":12112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":12112},{"type":"v1","addr":"172.21.15.47:6813","nonce":12112}]},"public_addr":"172.21.15.47:6809/12112","cluster_addr":"172.21.15.47:6811/12112","heartbeat_back_addr":"172.21.15.47:6815/12112","heartbeat_front_addr":"172.21.15.47:6813/12112","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.435265+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.472776+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.388370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.417366+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.102392+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.116529+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.167430+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.148139+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:05:05.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.943+0000 7ff58f59e700 1 -- 172.21.15.189:0/3486666562 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff5640557f0 msgr2=0x7ff564057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.943+0000 7ff58f59e700 1 --2- 172.21.15.189:0/3486666562 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff5640557f0 0x7ff564057cb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7ff59013d320 tx=0x7ff580009040).stop 2020-11-20T12:05:05.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.943+0000 7ff58f59e700 1 -- 172.21.15.189:0/3486666562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5900aef60 msgr2=0x7ff590064030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:05.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.943+0000 7ff58f59e700 1 --2- 172.21.15.189:0/3486666562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5900aef60 0x7ff590064030 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7ff5900ac720 tx=0x7ff584015000).stop 2020-11-20T12:05:05.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.943+0000 7ff58f59e700 1 -- 172.21.15.189:0/3486666562 shutdown_connections 2020-11-20T12:05:05.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.943+0000 7ff58f59e700 1 --2- 172.21.15.189:0/3486666562 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff5640557f0 0x7ff564057cb0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.943+0000 7ff58f59e700 1 --2- 172.21.15.189:0/3486666562 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff59014e0d0 0x7ff590064e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.943+0000 7ff58f59e700 1 --2- 172.21.15.189:0/3486666562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5900aef60 0x7ff590064030 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.943+0000 7ff58f59e700 1 --2- 172.21.15.189:0/3486666562 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff59013e650 0x7ff5900a7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:05.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.943+0000 7ff58f59e700 1 -- 172.21.15.189:0/3486666562 >> 172.21.15.189:0/3486666562 conn(0x7ff590062080 msgr2=0x7ff59013ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:05.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.943+0000 7ff58f59e700 1 -- 172.21.15.189:0/3486666562 shutdown_connections 2020-11-20T12:05:05.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:05.943+0000 7ff58f59e700 1 -- 172.21.15.189:0/3486666562 wait complete. 2020-11-20T12:05:05.960 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:41.395395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:48.550454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:04:52.101837+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:04:53.018196+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:05:05.961 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:05:06.090 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.083+0000 7fe68d7d4700 1 Processor -- start 2020-11-20T12:05:06.090 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.083+0000 7fe68d7d4700 1 -- start start 2020-11-20T12:05:06.092 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe68d7d4700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe68813a090 0x7fe68813a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.093 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe68d7d4700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe6881339a0 0x7fe688133da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.093 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe68d7d4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6880a0e40 con 0x7fe6880612b0 2020-11-20T12:05:06.093 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe68d7d4700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe688067b80 con 0x7fe6881339a0 2020-11-20T12:05:06.093 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe68d7d4700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe688067e20 con 0x7fe68813a090 2020-11-20T12:05:06.094 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe6877fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe68813a090 0x7fe68813a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.094 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe687fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe6880612b0 0x7fe6880a0730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50392/0 (socket says 172.21.15.189:50392) 2020-11-20T12:05:06.094 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe687fff700 1 -- 172.21.15.189:0/3491769706 learned_addr learned my addr 172.21.15.189:0/3491769706 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:06.094 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe68cfd3700 1 --2- 172.21.15.189:0/3491769706 >> v2:172.21.15.47:3300/0 conn(0x7fe6881339a0 0x7fe688133da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.095 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe6877fe700 1 -- 172.21.15.189:0/3491769706 >> v2:172.21.15.47:3300/0 conn(0x7fe6881339a0 msgr2=0x7fe688133da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:06.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe6877fe700 1 --2- 172.21.15.189:0/3491769706 >> v2:172.21.15.47:3300/0 conn(0x7fe6881339a0 0x7fe688133da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe6877fe700 1 -- 172.21.15.189:0/3491769706 >> v1:172.21.15.189:6789/0 conn(0x7fe6880612b0 legacy=0x7fe6880a0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe6877fe700 1 -- 172.21.15.189:0/3491769706 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6880680c0 con 0x7fe68813a090 2020-11-20T12:05:06.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe686ffd700 1 -- 172.21.15.189:0/3491769706 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927902993 0 0) 0x7fe6880a0e40 con 0x7fe6880612b0 2020-11-20T12:05:06.097 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe6877fe700 1 --2- 172.21.15.189:0/3491769706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe68813a090 0x7fe68813a4f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fe6880662b0 tx=0x7fe670006df0).ready entity=mon.1 client_cookie=78a5a11ca1542d62 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:06.097 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe686ffd700 1 -- 172.21.15.189:0/3491769706 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6700097b0 con 0x7fe68813a090 2020-11-20T12:05:06.097 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe686ffd700 1 -- 172.21.15.189:0/3491769706 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe670009910 con 0x7fe68813a090 2020-11-20T12:05:06.097 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe686ffd700 1 -- 172.21.15.189:0/3491769706 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe670009c90 con 0x7fe68813a090 2020-11-20T12:05:06.098 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe68d7d4700 1 -- 172.21.15.189:0/3491769706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe68813a090 msgr2=0x7fe68813a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.098 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe68d7d4700 1 --2- 172.21.15.189:0/3491769706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe68813a090 0x7fe68813a4f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fe6880662b0 tx=0x7fe670006df0).stop 2020-11-20T12:05:06.098 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe68d7d4700 1 -- 172.21.15.189:0/3491769706 shutdown_connections 2020-11-20T12:05:06.098 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe68d7d4700 1 --2- 172.21.15.189:0/3491769706 >> v2:172.21.15.47:3300/0 conn(0x7fe6881339a0 0x7fe688133da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.099 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe68d7d4700 1 --2- 172.21.15.189:0/3491769706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe68813a090 0x7fe68813a4f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.099 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe68d7d4700 1 -- 172.21.15.189:0/3491769706 >> 172.21.15.189:0/3491769706 conn(0x7fe688065910 msgr2=0x7fe688060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:06.099 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe68d7d4700 1 -- 172.21.15.189:0/3491769706 shutdown_connections 2020-11-20T12:05:06.099 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe68d7d4700 1 -- 172.21.15.189:0/3491769706 wait complete. 2020-11-20T12:05:06.103 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe68d7d4700 1 Processor -- start 2020-11-20T12:05:06.104 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.087+0000 7fe68d7d4700 1 -- start start 2020-11-20T12:05:06.104 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe68d7d4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6880612b0 0x7fe688138060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.104 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe68d7d4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6881339a0 0x7fe688134c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.104 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe68d7d4700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe68813a090 0x7fe6881285a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.104 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe68d7d4700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6880a6860 con 0x7fe6880612b0 2020-11-20T12:05:06.105 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe68d7d4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6880a6a90 con 0x7fe6881339a0 2020-11-20T12:05:06.105 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe68d7d4700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6880a6d30 con 0x7fe68813a090 2020-11-20T12:05:06.105 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe6877fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6881339a0 0x7fe688134c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.105 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe6877fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6881339a0 0x7fe688134c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48778/0 (socket says 172.21.15.189:48778) 2020-11-20T12:05:06.105 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe6877fe700 1 -- 172.21.15.189:0/3895649952 learned_addr learned my addr 172.21.15.189:0/3895649952 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:06.106 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe687fff700 1 --2- 172.21.15.189:0/3895649952 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6880612b0 0x7fe688138060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.106 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe68cfd3700 1 --2- 172.21.15.189:0/3895649952 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe68813a090 0x7fe6881285a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.106 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe6877fe700 1 -- 172.21.15.189:0/3895649952 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe68813a090 msgr2=0x7fe6881285a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.106 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe6877fe700 1 --2- 172.21.15.189:0/3895649952 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe68813a090 0x7fe6881285a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.107 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe6877fe700 1 -- 172.21.15.189:0/3895649952 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6880612b0 msgr2=0x7fe688138060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.107 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe6877fe700 1 --2- 172.21.15.189:0/3895649952 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6880612b0 0x7fe688138060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.107 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe6877fe700 1 -- 172.21.15.189:0/3895649952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe688138c70 con 0x7fe6881339a0 2020-11-20T12:05:06.107 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe6877fe700 1 --2- 172.21.15.189:0/3895649952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6881339a0 0x7fe688134c60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fe670002fc0 tx=0x7fe670015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:06.107 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe67ffff700 1 -- 172.21.15.189:0/3895649952 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe670011d50 con 0x7fe6881339a0 2020-11-20T12:05:06.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe67ffff700 1 -- 172.21.15.189:0/3895649952 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe67001e460 con 0x7fe6881339a0 2020-11-20T12:05:06.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe68d7d4700 1 -- 172.21.15.189:0/3895649952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe688135150 con 0x7fe6881339a0 2020-11-20T12:05:06.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe67ffff700 1 -- 172.21.15.189:0/3895649952 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67001e800 con 0x7fe6881339a0 2020-11-20T12:05:06.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe68d7d4700 1 -- 172.21.15.189:0/3895649952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe688128c30 con 0x7fe6881339a0 2020-11-20T12:05:06.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe68d7d4700 1 -- 172.21.15.189:0/3895649952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6880a6ef0 con 0x7fe6881339a0 2020-11-20T12:05:06.109 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe67ffff700 1 -- 172.21.15.189:0/3895649952 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fe670025030 con 0x7fe6881339a0 2020-11-20T12:05:06.109 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe67ffff700 1 --2- 172.21.15.189:0/3895649952 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe6580557c0 0x7fe658057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.109 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe67ffff700 1 -- 172.21.15.189:0/3895649952 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe67006fe40 con 0x7fe6881339a0 2020-11-20T12:05:06.109 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe687fff700 1 --2- 172.21.15.189:0/3895649952 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe6580557c0 0x7fe658057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.110 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.091+0000 7fe687fff700 1 --2- 172.21.15.189:0/3895649952 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe6580557c0 0x7fe658057c80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fe680001520 tx=0x7fe680007d80).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:06.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.099+0000 7fe67ffff700 1 -- 172.21.15.189:0/3895649952 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe670045ce0 con 0x7fe6881339a0 2020-11-20T12:05:06.254 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.251+0000 7fe68d7d4700 1 -- 172.21.15.189:0/3895649952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fe688128e20 con 0x7fe6881339a0 2020-11-20T12:05:06.255 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.251+0000 7fe67ffff700 1 -- 172.21.15.189:0/3895649952 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v22) v1 ==== 110+0+10 (secure 0 0 0) 0x7fe670023070 con 0x7fe6881339a0 2020-11-20T12:05:06.255 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2020-11-20T12:05:06.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.251+0000 7fe68d7d4700 1 -- 172.21.15.189:0/3895649952 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe6580557c0 msgr2=0x7fe658057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.251+0000 7fe68d7d4700 1 --2- 172.21.15.189:0/3895649952 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe6580557c0 0x7fe658057c80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fe680001520 tx=0x7fe680007d80).stop 2020-11-20T12:05:06.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.251+0000 7fe68d7d4700 1 -- 172.21.15.189:0/3895649952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6881339a0 msgr2=0x7fe688134c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.251+0000 7fe68d7d4700 1 --2- 172.21.15.189:0/3895649952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6881339a0 0x7fe688134c60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fe670002fc0 tx=0x7fe670015000).stop 2020-11-20T12:05:06.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.251+0000 7fe68d7d4700 1 -- 172.21.15.189:0/3895649952 shutdown_connections 2020-11-20T12:05:06.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.251+0000 7fe68d7d4700 1 --2- 172.21.15.189:0/3895649952 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe6580557c0 0x7fe658057c80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.251+0000 7fe68d7d4700 1 --2- 172.21.15.189:0/3895649952 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe6880612b0 0x7fe688138060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.251+0000 7fe68d7d4700 1 --2- 172.21.15.189:0/3895649952 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe68813a090 0x7fe6881285a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.251+0000 7fe68d7d4700 1 --2- 172.21.15.189:0/3895649952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6881339a0 0x7fe688134c60 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.260 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.251+0000 7fe68d7d4700 1 -- 172.21.15.189:0/3895649952 >> 172.21.15.189:0/3895649952 conn(0x7fe688065910 msgr2=0x7fe6880aa5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:06.260 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.251+0000 7fe68d7d4700 1 -- 172.21.15.189:0/3895649952 shutdown_connections 2020-11-20T12:05:06.260 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.251+0000 7fe68d7d4700 1 -- 172.21.15.189:0/3895649952 wait complete. 2020-11-20T12:05:06.275 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:05:06.398 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.390+0000 7f55a0972700 1 Processor -- start 2020-11-20T12:05:06.398 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.390+0000 7f55a0972700 1 -- start start 2020-11-20T12:05:06.402 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f55a0972700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f559c060ba0 0x7f559c0a8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.402 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f55a0972700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559c0a9410 con 0x7f559c137b00 2020-11-20T12:05:06.403 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f55a0972700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559c0678f0 con 0x7f559c067d90 2020-11-20T12:05:06.403 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f55a0972700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f559c0684c0 con 0x7f559c060ba0 2020-11-20T12:05:06.403 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559affd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f559c067d90 0x7f559c1373f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50408/0 (socket says 172.21.15.189:50408) 2020-11-20T12:05:06.403 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559affd700 1 -- 172.21.15.189:0/3328554630 learned_addr learned my addr 172.21.15.189:0/3328554630 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:06.404 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559b7fe700 1 --2- 172.21.15.189:0/3328554630 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f559c060ba0 0x7f559c0a8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.404 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559a7fc700 1 -- 172.21.15.189:0/3328554630 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3955916489 0 0) 0x7f559c0678f0 con 0x7f559c067d90 2020-11-20T12:05:06.404 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559a7fc700 1 -- 172.21.15.189:0/3328554630 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f557c003240 con 0x7f559c067d90 2020-11-20T12:05:06.404 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559a7fc700 1 -- 172.21.15.189:0/3328554630 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1525165928 0 0) 0x7f557c003240 con 0x7f559c067d90 2020-11-20T12:05:06.404 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559a7fc700 1 -- 172.21.15.189:0/3328554630 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f559c060ba0 msgr2=0x7f559c0a8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:06.405 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559a7fc700 1 --2- 172.21.15.189:0/3328554630 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f559c060ba0 0x7f559c0a8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.405 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559a7fc700 1 -- 172.21.15.189:0/3328554630 >> v1:172.21.15.47:6789/0 conn(0x7f559c137b00 legacy=0x7f559c144cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.405 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559a7fc700 1 -- 172.21.15.189:0/3328554630 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559c060f80 con 0x7f559c067d90 2020-11-20T12:05:06.405 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559a7fc700 1 -- 172.21.15.189:0/3328554630 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f55840014e0 con 0x7f559c067d90 2020-11-20T12:05:06.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559a7fc700 1 -- 172.21.15.189:0/3328554630 >> v1:172.21.15.189:6789/0 conn(0x7f559c067d90 legacy=0x7f559c1373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559a7fc700 1 --2- 172.21.15.189:0/3328554630 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f557c003730 0x7f557c003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559a7fc700 1 --2- 172.21.15.189:0/3328554630 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f559c060ba0 0x7f557c0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559a7fc700 1 --2- 172.21.15.189:0/3328554630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f557c0049d0 0x7f557c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559a7fc700 1 -- 172.21.15.189:0/3328554630 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559c0684c0 con 0x7f557c003730 2020-11-20T12:05:06.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559a7fc700 1 -- 172.21.15.189:0/3328554630 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557c0041d0 con 0x7f557c0049d0 2020-11-20T12:05:06.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559a7fc700 1 -- 172.21.15.189:0/3328554630 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f557c00b730 con 0x7f559c060ba0 2020-11-20T12:05:06.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559bfff700 1 --2- 172.21.15.189:0/3328554630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f557c0049d0 0x7f557c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559affd700 1 --2- 172.21.15.189:0/3328554630 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f559c060ba0 0x7f557c0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559b7fe700 1 --2- 172.21.15.189:0/3328554630 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f557c003730 0x7f557c003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559bfff700 1 -- 172.21.15.189:0/3328554630 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f559c060ba0 msgr2=0x7f557c0044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:06.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559bfff700 1 --2- 172.21.15.189:0/3328554630 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f559c060ba0 0x7f557c0044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559bfff700 1 -- 172.21.15.189:0/3328554630 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f557c003730 msgr2=0x7f557c003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:06.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559bfff700 1 --2- 172.21.15.189:0/3328554630 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f557c003730 0x7f557c003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.409 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559bfff700 1 -- 172.21.15.189:0/3328554630 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f557c003240 con 0x7f557c0049d0 2020-11-20T12:05:06.409 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559bfff700 1 --2- 172.21.15.189:0/3328554630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f557c0049d0 0x7f557c004e50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f558c0018f0 tx=0x7f558c007b90).ready entity=mon.1 client_cookie=6d11d85028db2f3f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:06.409 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559a7fc700 1 -- 172.21.15.189:0/3328554630 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558c007020 con 0x7f557c0049d0 2020-11-20T12:05:06.409 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559a7fc700 1 -- 172.21.15.189:0/3328554630 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f558c007180 con 0x7f557c0049d0 2020-11-20T12:05:06.409 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559a7fc700 1 -- 172.21.15.189:0/3328554630 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558c00c7f0 con 0x7f557c0049d0 2020-11-20T12:05:06.410 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f55a0972700 1 -- 172.21.15.189:0/3328554630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f557c0049d0 msgr2=0x7f557c004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.410 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f55a0972700 1 --2- 172.21.15.189:0/3328554630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f557c0049d0 0x7f557c004e50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f558c0018f0 tx=0x7f558c007b90).stop 2020-11-20T12:05:06.410 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f559bfff700 1 -- 172.21.15.189:0/3328554630 reap_dead start 2020-11-20T12:05:06.410 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f55a0972700 1 -- 172.21.15.189:0/3328554630 shutdown_connections 2020-11-20T12:05:06.410 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f55a0972700 1 -- 172.21.15.189:0/3328554630 >> 172.21.15.189:0/3328554630 conn(0x7f559c0ad8e0 msgr2=0x7f559c0adcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:06.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f55a0972700 1 -- 172.21.15.189:0/3328554630 shutdown_connections 2020-11-20T12:05:06.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f55a0972700 1 -- 172.21.15.189:0/3328554630 wait complete. 2020-11-20T12:05:06.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.394+0000 7f55a0972700 1 Processor -- start 2020-11-20T12:05:06.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f55a0972700 1 -- start start 2020-11-20T12:05:06.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f55a0972700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f559c13a040 0x7f559c0a2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f55a0972700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f559c14cf70 0x7f559c1499a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f55a0972700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f559c149e90 0x7f559c0a7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f55a0972700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559c14a2f0 con 0x7f559c149e90 2020-11-20T12:05:06.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f55a0972700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559c13af10 con 0x7f559c13a040 2020-11-20T12:05:06.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f55a0972700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f559c14d4f0 con 0x7f559c14cf70 2020-11-20T12:05:06.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f559b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f559c13a040 0x7f559c0a2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f559b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f559c13a040 0x7f559c0a2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48798/0 (socket says 172.21.15.189:48798) 2020-11-20T12:05:06.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f559b7fe700 1 -- 172.21.15.189:0/238988470 learned_addr learned my addr 172.21.15.189:0/238988470 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:06.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f559affd700 1 --2- 172.21.15.189:0/238988470 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f559c14cf70 0x7f559c1499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f559bfff700 1 --2- 172.21.15.189:0/238988470 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f559c149e90 0x7f559c0a7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f559b7fe700 1 -- 172.21.15.189:0/238988470 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f559c14cf70 msgr2=0x7f559c1499a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f559b7fe700 1 --2- 172.21.15.189:0/238988470 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f559c14cf70 0x7f559c1499a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f559b7fe700 1 -- 172.21.15.189:0/238988470 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f559c149e90 msgr2=0x7f559c0a7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f559b7fe700 1 --2- 172.21.15.189:0/238988470 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f559c149e90 0x7f559c0a7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f559b7fe700 1 -- 172.21.15.189:0/238988470 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559c14d790 con 0x7f559c13a040 2020-11-20T12:05:06.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f559b7fe700 1 --2- 172.21.15.189:0/238988470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f559c13a040 0x7f559c0a2130 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f5594009fd0 tx=0x7f5594008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:06.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f55937fe700 1 -- 172.21.15.189:0/238988470 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559400ecf0 con 0x7f559c13a040 2020-11-20T12:05:06.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f55937fe700 1 -- 172.21.15.189:0/238988470 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f559400ee50 con 0x7f559c13a040 2020-11-20T12:05:06.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f55a0972700 1 -- 172.21.15.189:0/238988470 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f559c14da30 con 0x7f559c13a040 2020-11-20T12:05:06.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f55937fe700 1 -- 172.21.15.189:0/238988470 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5594014640 con 0x7f559c13a040 2020-11-20T12:05:06.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f55a0972700 1 -- 172.21.15.189:0/238988470 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f559c0a7640 con 0x7f559c13a040 2020-11-20T12:05:06.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f55a0972700 1 -- 172.21.15.189:0/238988470 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f559c0a8d70 con 0x7f559c13a040 2020-11-20T12:05:06.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f55937fe700 1 -- 172.21.15.189:0/238988470 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f5594024070 con 0x7f559c13a040 2020-11-20T12:05:06.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f55937fe700 1 --2- 172.21.15.189:0/238988470 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f55780557c0 0x7f5578057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f55937fe700 1 -- 172.21.15.189:0/238988470 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f559406efe0 con 0x7f559c13a040 2020-11-20T12:05:06.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f559affd700 1 --2- 172.21.15.189:0/238988470 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f55780557c0 0x7f5578057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.398+0000 7f559affd700 1 --2- 172.21.15.189:0/238988470 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f55780557c0 0x7f5578057c80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f559c0aa2c0 tx=0x7f5584009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:06.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.406+0000 7f55937fe700 1 -- 172.21.15.189:0/238988470 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5594044e20 con 0x7f559c13a040 2020-11-20T12:05:06.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.578+0000 7f55a0972700 1 -- 172.21.15.189:0/238988470 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f559c0446d0 con 0x7f559c13a040 2020-11-20T12:05:06.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.582+0000 7f55937fe700 1 -- 172.21.15.189:0/238988470 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v22) v1 ==== 92+0+10 (secure 0 0 0) 0x7f55940220a0 con 0x7f559c13a040 2020-11-20T12:05:06.587 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 8 2020-11-20T12:05:06.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.582+0000 7f55a0972700 1 -- 172.21.15.189:0/238988470 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f55780557c0 msgr2=0x7f5578057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.582+0000 7f55a0972700 1 --2- 172.21.15.189:0/238988470 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f55780557c0 0x7f5578057c80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f559c0aa2c0 tx=0x7f5584009040).stop 2020-11-20T12:05:06.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.582+0000 7f55a0972700 1 -- 172.21.15.189:0/238988470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f559c13a040 msgr2=0x7f559c0a2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.582+0000 7f55a0972700 1 --2- 172.21.15.189:0/238988470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f559c13a040 0x7f559c0a2130 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f5594009fd0 tx=0x7f5594008040).stop 2020-11-20T12:05:06.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.582+0000 7f55a0972700 1 -- 172.21.15.189:0/238988470 shutdown_connections 2020-11-20T12:05:06.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.582+0000 7f55a0972700 1 --2- 172.21.15.189:0/238988470 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f55780557c0 0x7f5578057c80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.582+0000 7f55a0972700 1 --2- 172.21.15.189:0/238988470 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f559c149e90 0x7f559c0a7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.592 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.582+0000 7f55a0972700 1 --2- 172.21.15.189:0/238988470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f559c13a040 0x7f559c0a2130 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.592 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.582+0000 7f55a0972700 1 --2- 172.21.15.189:0/238988470 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f559c14cf70 0x7f559c1499a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.592 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.582+0000 7f55a0972700 1 -- 172.21.15.189:0/238988470 >> 172.21.15.189:0/238988470 conn(0x7f559c0ad8e0 msgr2=0x7f559c066970 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:06.593 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.582+0000 7f55a0972700 1 -- 172.21.15.189:0/238988470 shutdown_connections 2020-11-20T12:05:06.593 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.582+0000 7f55a0972700 1 -- 172.21.15.189:0/238988470 wait complete. 2020-11-20T12:05:06.608 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:05:06.740 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.734+0000 7f5db5ca1700 1 Processor -- start 2020-11-20T12:05:06.740 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.734+0000 7f5db5ca1700 1 -- start start 2020-11-20T12:05:06.742 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.734+0000 7f5db5ca1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5db0060310 0x7f5db013a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.742 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.734+0000 7f5db5ca1700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db00a8fc0 con 0x7f5db00a0c30 2020-11-20T12:05:06.742 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.734+0000 7f5db5ca1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db00a9260 con 0x7f5db0066ba0 2020-11-20T12:05:06.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.734+0000 7f5db5ca1700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5db00674b0 con 0x7f5db0060310 2020-11-20T12:05:06.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.734+0000 7f5daffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5db0066ba0 0x7f5db00a0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50426/0 (socket says 172.21.15.189:50426) 2020-11-20T12:05:06.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.734+0000 7f5daffff700 1 -- 172.21.15.189:0/961362032 learned_addr learned my addr 172.21.15.189:0/961362032 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:06.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.734+0000 7f5db4c9f700 1 --2- 172.21.15.189:0/961362032 >> v2:172.21.15.189:3300/0 conn(0x7f5db0060310 0x7f5db013a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.734+0000 7f5daf7fe700 1 -- 172.21.15.189:0/961362032 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810002381 0 0) 0x7f5db00a9260 con 0x7f5db0066ba0 2020-11-20T12:05:06.744 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.734+0000 7f5daf7fe700 1 -- 172.21.15.189:0/961362032 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d9c003240 con 0x7f5db0066ba0 2020-11-20T12:05:06.744 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.734+0000 7f5db4c9f700 1 -- 172.21.15.189:0/961362032 >> v1:172.21.15.189:6789/0 conn(0x7f5db0066ba0 legacy=0x7f5db00a0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.744 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.734+0000 7f5db4c9f700 1 -- 172.21.15.189:0/961362032 >> v1:172.21.15.47:6789/0 conn(0x7f5db00a0c30 legacy=0x7f5db0141c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.744 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.734+0000 7f5db4c9f700 1 -- 172.21.15.189:0/961362032 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db0068460 con 0x7f5db0060310 2020-11-20T12:05:06.744 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.734+0000 7f5db4c9f700 1 --2- 172.21.15.189:0/961362032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5db0060310 0x7f5db013a8f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f5da400b9a0 tx=0x7f5da4006df0).ready entity=mon.1 client_cookie=1dd6c645878c478 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:06.745 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.734+0000 7f5daf7fe700 1 -- 172.21.15.189:0/961362032 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da40097b0 con 0x7f5db0060310 2020-11-20T12:05:06.745 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.734+0000 7f5daf7fe700 1 -- 172.21.15.189:0/961362032 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5da4009910 con 0x7f5db0060310 2020-11-20T12:05:06.745 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.734+0000 7f5daf7fe700 1 -- 172.21.15.189:0/961362032 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da4009c90 con 0x7f5db0060310 2020-11-20T12:05:06.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db5ca1700 1 -- 172.21.15.189:0/961362032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5db0060310 msgr2=0x7f5db013a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db5ca1700 1 --2- 172.21.15.189:0/961362032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5db0060310 0x7f5db013a8f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f5da400b9a0 tx=0x7f5da4006df0).stop 2020-11-20T12:05:06.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db5ca1700 1 -- 172.21.15.189:0/961362032 shutdown_connections 2020-11-20T12:05:06.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db5ca1700 1 --2- 172.21.15.189:0/961362032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5db0060310 0x7f5db013a8f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db5ca1700 1 -- 172.21.15.189:0/961362032 >> 172.21.15.189:0/961362032 conn(0x7f5db0065910 msgr2=0x7f5db0060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:06.749 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db5ca1700 1 -- 172.21.15.189:0/961362032 shutdown_connections 2020-11-20T12:05:06.749 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db5ca1700 1 -- 172.21.15.189:0/961362032 wait complete. 2020-11-20T12:05:06.749 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db5ca1700 1 Processor -- start 2020-11-20T12:05:06.749 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db5ca1700 1 -- start start 2020-11-20T12:05:06.749 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db5ca1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5db0060310 0x7f5db00ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.750 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db5ca1700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5db0066ba0 0x7f5db013acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.750 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db5ca1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5db00a0c30 0x7f5db0145af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.750 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db5ca1700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db01351a0 con 0x7f5db00a0c30 2020-11-20T12:05:06.750 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db5ca1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db00a3960 con 0x7f5db0060310 2020-11-20T12:05:06.750 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db5ca1700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5db00a3c00 con 0x7f5db0066ba0 2020-11-20T12:05:06.751 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db4c9f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5db0060310 0x7f5db00ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.751 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db4c9f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5db0060310 0x7f5db00ad2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48808/0 (socket says 172.21.15.189:48808) 2020-11-20T12:05:06.751 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db4c9f700 1 -- 172.21.15.189:0/3010342395 learned_addr learned my addr 172.21.15.189:0/3010342395 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:06.751 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5daffff700 1 --2- 172.21.15.189:0/3010342395 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5db0066ba0 0x7f5db013acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.751 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db54a0700 1 --2- 172.21.15.189:0/3010342395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5db00a0c30 0x7f5db0145af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.752 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db4c9f700 1 -- 172.21.15.189:0/3010342395 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5db0066ba0 msgr2=0x7f5db013acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.752 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db4c9f700 1 --2- 172.21.15.189:0/3010342395 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5db0066ba0 0x7f5db013acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.752 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db4c9f700 1 -- 172.21.15.189:0/3010342395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5db00a0c30 msgr2=0x7f5db0145af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.752 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db4c9f700 1 --2- 172.21.15.189:0/3010342395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5db00a0c30 0x7f5db0145af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.752 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db4c9f700 1 -- 172.21.15.189:0/3010342395 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db00a3ea0 con 0x7f5db0060310 2020-11-20T12:05:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db4c9f700 1 --2- 172.21.15.189:0/3010342395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5db0060310 0x7f5db00ad2a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f5db00609b0 tx=0x7f5da4017000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5dacff9700 1 -- 172.21.15.189:0/3010342395 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da4020070 con 0x7f5db0060310 2020-11-20T12:05:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5dacff9700 1 -- 172.21.15.189:0/3010342395 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5da401a480 con 0x7f5db0060310 2020-11-20T12:05:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db5ca1700 1 -- 172.21.15.189:0/3010342395 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5db00a3560 con 0x7f5db0060310 2020-11-20T12:05:06.753 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5dacff9700 1 -- 172.21.15.189:0/3010342395 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da401a800 con 0x7f5db0060310 2020-11-20T12:05:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.738+0000 7f5db5ca1700 1 -- 172.21.15.189:0/3010342395 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5db00ad990 con 0x7f5db0060310 2020-11-20T12:05:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.742+0000 7f5db5ca1700 1 -- 172.21.15.189:0/3010342395 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5db00a4030 con 0x7f5db0060310 2020-11-20T12:05:06.754 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.742+0000 7f5dacff9700 1 -- 172.21.15.189:0/3010342395 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f5da401abb0 con 0x7f5db0060310 2020-11-20T12:05:06.755 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.742+0000 7f5dacff9700 1 --2- 172.21.15.189:0/3010342395 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5d8c0557c0 0x7f5d8c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:06.755 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.742+0000 7f5daffff700 1 --2- 172.21.15.189:0/3010342395 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5d8c0557c0 0x7f5d8c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:06.755 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.742+0000 7f5dacff9700 1 -- 172.21.15.189:0/3010342395 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5da4071c90 con 0x7f5db0060310 2020-11-20T12:05:06.756 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.742+0000 7f5daffff700 1 --2- 172.21.15.189:0/3010342395 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5d8c0557c0 0x7f5d8c057c80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f5db0136a80 tx=0x7f5da0009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:06.756 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.746+0000 7f5dacff9700 1 -- 172.21.15.189:0/3010342395 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5da4047b30 con 0x7f5db0060310 2020-11-20T12:05:06.910 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.902+0000 7f5db5ca1700 1 -- 172.21.15.189:0/3010342395 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5db00a4420 con 0x7f5db0060310 2020-11-20T12:05:06.910 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.906+0000 7f5dacff9700 1 -- 172.21.15.189:0/3010342395 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+10 (secure 0 0 0) 0x7f5da404b150 con 0x7f5db0060310 2020-11-20T12:05:06.911 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 8 2020-11-20T12:05:06.912 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.906+0000 7f5db5ca1700 1 -- 172.21.15.189:0/3010342395 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5d8c0557c0 msgr2=0x7f5d8c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.913 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.906+0000 7f5db5ca1700 1 --2- 172.21.15.189:0/3010342395 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5d8c0557c0 0x7f5d8c057c80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f5db0136a80 tx=0x7f5da0009040).stop 2020-11-20T12:05:06.913 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.906+0000 7f5db5ca1700 1 -- 172.21.15.189:0/3010342395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5db0060310 msgr2=0x7f5db00ad2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:06.913 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.906+0000 7f5db5ca1700 1 --2- 172.21.15.189:0/3010342395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5db0060310 0x7f5db00ad2a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f5db00609b0 tx=0x7f5da4017000).stop 2020-11-20T12:05:06.913 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.906+0000 7f5db5ca1700 1 -- 172.21.15.189:0/3010342395 shutdown_connections 2020-11-20T12:05:06.914 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.906+0000 7f5db5ca1700 1 --2- 172.21.15.189:0/3010342395 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5d8c0557c0 0x7f5d8c057c80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.914 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.906+0000 7f5db5ca1700 1 --2- 172.21.15.189:0/3010342395 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5db00a0c30 0x7f5db0145af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.914 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.906+0000 7f5db5ca1700 1 --2- 172.21.15.189:0/3010342395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5db0060310 0x7f5db00ad2a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.914 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.906+0000 7f5db5ca1700 1 --2- 172.21.15.189:0/3010342395 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5db0066ba0 0x7f5db013acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:06.915 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.906+0000 7f5db5ca1700 1 -- 172.21.15.189:0/3010342395 >> 172.21.15.189:0/3010342395 conn(0x7f5db0065910 msgr2=0x7f5db0136f50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:06.915 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.906+0000 7f5db5ca1700 1 -- 172.21.15.189:0/3010342395 shutdown_connections 2020-11-20T12:05:06.915 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:06.906+0000 7f5db5ca1700 1 -- 172.21.15.189:0/3010342395 wait complete. 2020-11-20T12:05:06.926 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:05:07.069 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.062+0000 7f9f4df8f700 1 Processor -- start 2020-11-20T12:05:07.069 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.062+0000 7f9f4df8f700 1 -- start start 2020-11-20T12:05:07.073 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.062+0000 7f9f4df8f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9f48066ba0 0x7f9f480a0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.073 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.062+0000 7f9f4df8f700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9f480a0a10 0x7f9f48141c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.074 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.062+0000 7f9f4df8f700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f480a8fc0 con 0x7f9f48060310 2020-11-20T12:05:07.074 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.062+0000 7f9f4df8f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f480674b0 con 0x7f9f48066ba0 2020-11-20T12:05:07.074 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.062+0000 7f9f4df8f700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f480a9260 con 0x7f9f480a0a10 2020-11-20T12:05:07.074 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.062+0000 7f9f3ffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9f48066ba0 0x7f9f480a0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.075 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.062+0000 7f9f3ffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9f48066ba0 0x7f9f480a0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48820/0 (socket says 172.21.15.189:48820) 2020-11-20T12:05:07.075 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.062+0000 7f9f4cf8d700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f9f48060310 0x7f9f4813a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.189:45102/0 (socket says 172.21.15.189:45102) 2020-11-20T12:05:07.075 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.062+0000 7f9f3ffff700 1 -- 172.21.15.189:0/1059989923 learned_addr learned my addr 172.21.15.189:0/1059989923 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:07.075 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.062+0000 7f9f4d78e700 1 --2- 172.21.15.189:0/1059989923 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9f480a0a10 0x7f9f48141c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.075 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f3f7fe700 1 -- 172.21.15.189:0/1059989923 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3078239254 0 0) 0x7f9f480a8fc0 con 0x7f9f48060310 2020-11-20T12:05:07.076 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f3ffff700 1 -- 172.21.15.189:0/1059989923 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9f480a0a10 msgr2=0x7f9f48141c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:07.076 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f3ffff700 1 --2- 172.21.15.189:0/1059989923 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9f480a0a10 0x7f9f48141c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.076 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f3ffff700 1 -- 172.21.15.189:0/1059989923 >> v1:172.21.15.47:6789/0 conn(0x7f9f48060310 legacy=0x7f9f4813a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.076 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f3ffff700 1 -- 172.21.15.189:0/1059989923 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f48068460 con 0x7f9f48066ba0 2020-11-20T12:05:07.077 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f3ffff700 1 --2- 172.21.15.189:0/1059989923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f48066ba0 0x7f9f480a0520 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f9f38006fd0 tx=0x7f9f38006eb0).ready entity=mon.1 client_cookie=fd235510c0c21fb5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:07.077 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f3f7fe700 1 -- 172.21.15.189:0/1059989923 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f380098a0 con 0x7f9f48066ba0 2020-11-20T12:05:07.077 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f3f7fe700 1 -- 172.21.15.189:0/1059989923 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f38009a00 con 0x7f9f48066ba0 2020-11-20T12:05:07.077 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f3f7fe700 1 -- 172.21.15.189:0/1059989923 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f38009d80 con 0x7f9f48066ba0 2020-11-20T12:05:07.077 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f4df8f700 1 -- 172.21.15.189:0/1059989923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f48066ba0 msgr2=0x7f9f480a0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.078 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f4df8f700 1 --2- 172.21.15.189:0/1059989923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f48066ba0 0x7f9f480a0520 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f9f38006fd0 tx=0x7f9f38006eb0).stop 2020-11-20T12:05:07.078 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f4df8f700 1 -- 172.21.15.189:0/1059989923 shutdown_connections 2020-11-20T12:05:07.078 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f4df8f700 1 --2- 172.21.15.189:0/1059989923 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9f480a0a10 0x7f9f48141c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.078 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f4df8f700 1 --2- 172.21.15.189:0/1059989923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f48066ba0 0x7f9f480a0520 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.079 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f4df8f700 1 -- 172.21.15.189:0/1059989923 >> 172.21.15.189:0/1059989923 conn(0x7f9f48065910 msgr2=0x7f9f48060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:07.079 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f4df8f700 1 -- 172.21.15.189:0/1059989923 shutdown_connections 2020-11-20T12:05:07.079 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f4df8f700 1 -- 172.21.15.189:0/1059989923 wait complete. 2020-11-20T12:05:07.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f4df8f700 1 Processor -- start 2020-11-20T12:05:07.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f4df8f700 1 -- start start 2020-11-20T12:05:07.084 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f4df8f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f48060310 0x7f9f480a3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.085 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f4df8f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f48066ba0 0x7f9f480adae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.085 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f4df8f700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9f480a0a10 0x7f9f48147e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.086 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f4df8f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f480ad680 con 0x7f9f48066ba0 2020-11-20T12:05:07.086 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f4df8f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f480a1f40 con 0x7f9f48060310 2020-11-20T12:05:07.086 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f4df8f700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f480a1510 con 0x7f9f480a0a10 2020-11-20T12:05:07.087 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f4cf8d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f48060310 0x7f9f480a3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.087 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f4cf8d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f48060310 0x7f9f480a3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48822/0 (socket says 172.21.15.189:48822) 2020-11-20T12:05:07.087 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f4cf8d700 1 -- 172.21.15.189:0/1628375709 learned_addr learned my addr 172.21.15.189:0/1628375709 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:07.088 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f3ffff700 1 --2- 172.21.15.189:0/1628375709 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f48066ba0 0x7f9f480adae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.088 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f4d78e700 1 --2- 172.21.15.189:0/1628375709 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9f480a0a10 0x7f9f48147e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.088 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f4cf8d700 1 -- 172.21.15.189:0/1628375709 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9f480a0a10 msgr2=0x7f9f48147e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.089 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f4cf8d700 1 --2- 172.21.15.189:0/1628375709 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9f480a0a10 0x7f9f48147e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.089 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f4cf8d700 1 -- 172.21.15.189:0/1628375709 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f48066ba0 msgr2=0x7f9f480adae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.089 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f4cf8d700 1 --2- 172.21.15.189:0/1628375709 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f48066ba0 0x7f9f480adae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.090 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f4cf8d700 1 -- 172.21.15.189:0/1628375709 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f480a17b0 con 0x7f9f48060310 2020-11-20T12:05:07.090 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.066+0000 7f9f4cf8d700 1 --2- 172.21.15.189:0/1628375709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f48060310 0x7f9f480a3540 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f9f4400c710 tx=0x7f9f44007a60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:07.090 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.070+0000 7f9f3cff9700 1 -- 172.21.15.189:0/1628375709 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f44007320 con 0x7f9f48060310 2020-11-20T12:05:07.090 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.070+0000 7f9f3cff9700 1 -- 172.21.15.189:0/1628375709 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f44005530 con 0x7f9f48060310 2020-11-20T12:05:07.090 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.070+0000 7f9f4df8f700 1 -- 172.21.15.189:0/1628375709 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f480a4130 con 0x7f9f48060310 2020-11-20T12:05:07.091 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.070+0000 7f9f3cff9700 1 -- 172.21.15.189:0/1628375709 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f440058f0 con 0x7f9f48060310 2020-11-20T12:05:07.091 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.070+0000 7f9f4df8f700 1 -- 172.21.15.189:0/1628375709 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f480aca20 con 0x7f9f48060310 2020-11-20T12:05:07.091 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.070+0000 7f9f4df8f700 1 -- 172.21.15.189:0/1628375709 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f480446d0 con 0x7f9f48060310 2020-11-20T12:05:07.091 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.070+0000 7f9f3cff9700 1 -- 172.21.15.189:0/1628375709 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f9f44010030 con 0x7f9f48060310 2020-11-20T12:05:07.092 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.070+0000 7f9f3cff9700 1 --2- 172.21.15.189:0/1628375709 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9f200557c0 0x7f9f20057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.092 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.070+0000 7f9f3cff9700 1 -- 172.21.15.189:0/1628375709 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9f4406e3c0 con 0x7f9f48060310 2020-11-20T12:05:07.092 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.070+0000 7f9f3ffff700 1 --2- 172.21.15.189:0/1628375709 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9f200557c0 0x7f9f20057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.092 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.070+0000 7f9f3ffff700 1 --2- 172.21.15.189:0/1628375709 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9f200557c0 0x7f9f20057c80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f9f3800b1b0 tx=0x7f9f38015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:07.093 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.078+0000 7f9f3cff9700 1 -- 172.21.15.189:0/1628375709 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9f44044260 con 0x7f9f48060310 2020-11-20T12:05:07.239 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.230+0000 7f9f4df8f700 1 -- 172.21.15.189:0/1628375709 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9f4805cdd0 con 0x7f9f48060310 2020-11-20T12:05:07.240 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.234+0000 7f9f3cff9700 1 -- 172.21.15.189:0/1628375709 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+10 (secure 0 0 0) 0x7f9f44047880 con 0x7f9f48060310 2020-11-20T12:05:07.240 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 8 2020-11-20T12:05:07.242 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.234+0000 7f9f4df8f700 1 -- 172.21.15.189:0/1628375709 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9f200557c0 msgr2=0x7f9f20057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.242 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.234+0000 7f9f4df8f700 1 --2- 172.21.15.189:0/1628375709 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9f200557c0 0x7f9f20057c80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f9f3800b1b0 tx=0x7f9f38015000).stop 2020-11-20T12:05:07.243 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.234+0000 7f9f4df8f700 1 -- 172.21.15.189:0/1628375709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f48060310 msgr2=0x7f9f480a3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.243 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.234+0000 7f9f4df8f700 1 --2- 172.21.15.189:0/1628375709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f48060310 0x7f9f480a3540 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f9f4400c710 tx=0x7f9f44007a60).stop 2020-11-20T12:05:07.243 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.234+0000 7f9f4df8f700 1 -- 172.21.15.189:0/1628375709 shutdown_connections 2020-11-20T12:05:07.243 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.234+0000 7f9f4df8f700 1 --2- 172.21.15.189:0/1628375709 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9f480a0a10 0x7f9f48147e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.244 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.234+0000 7f9f4df8f700 1 --2- 172.21.15.189:0/1628375709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f48060310 0x7f9f480a3540 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.244 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.234+0000 7f9f4df8f700 1 --2- 172.21.15.189:0/1628375709 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9f200557c0 0x7f9f20057c80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.244 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.234+0000 7f9f4df8f700 1 --2- 172.21.15.189:0/1628375709 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9f48066ba0 0x7f9f480adae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.244 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.234+0000 7f9f4df8f700 1 -- 172.21.15.189:0/1628375709 >> 172.21.15.189:0/1628375709 conn(0x7f9f48065910 msgr2=0x7f9f48139a50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:07.244 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.234+0000 7f9f4df8f700 1 -- 172.21.15.189:0/1628375709 shutdown_connections 2020-11-20T12:05:07.245 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.234+0000 7f9f4df8f700 1 -- 172.21.15.189:0/1628375709 wait complete. 2020-11-20T12:05:07.254 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:05:07.377 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.370+0000 7ffb6e420700 1 Processor -- start 2020-11-20T12:05:07.378 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.370+0000 7ffb6e420700 1 -- start start 2020-11-20T12:05:07.379 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.370+0000 7ffb6e420700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ffb680612b0 0x7ffb680a0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.380 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.370+0000 7ffb6e420700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ffb68060170 0x7ffb680a85a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.380 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.370+0000 7ffb6e420700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb680a0f60 con 0x7ffb680605f0 2020-11-20T12:05:07.380 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.370+0000 7ffb6e420700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb68068440 con 0x7ffb68060170 2020-11-20T12:05:07.380 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.370+0000 7ffb6e420700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb68067d00 con 0x7ffb680612b0 2020-11-20T12:05:07.380 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.370+0000 7ffb6c9bd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ffb680605f0 0x7ffb681339a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50456/0 (socket says 172.21.15.189:50456) 2020-11-20T12:05:07.381 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.370+0000 7ffb6c9bd700 1 -- 172.21.15.189:0/4069873621 learned_addr learned my addr 172.21.15.189:0/4069873621 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:07.381 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.370+0000 7ffb67fff700 1 --2- 172.21.15.189:0/4069873621 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ffb680612b0 0x7ffb680a0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.381 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.370+0000 7ffb677fe700 1 --2- 172.21.15.189:0/4069873621 >> v2:172.21.15.47:3300/0 conn(0x7ffb68060170 0x7ffb680a85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.381 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.374+0000 7ffb66ffd700 1 -- 172.21.15.189:0/4069873621 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18375991 0 0) 0x7ffb680a0f60 con 0x7ffb680605f0 2020-11-20T12:05:07.381 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.374+0000 7ffb66ffd700 1 -- 172.21.15.189:0/4069873621 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb54003240 con 0x7ffb680605f0 2020-11-20T12:05:07.382 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.374+0000 7ffb67fff700 1 -- 172.21.15.189:0/4069873621 >> v2:172.21.15.47:3300/0 conn(0x7ffb68060170 msgr2=0x7ffb680a85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:07.382 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.374+0000 7ffb67fff700 1 --2- 172.21.15.189:0/4069873621 >> v2:172.21.15.47:3300/0 conn(0x7ffb68060170 0x7ffb680a85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.382 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.374+0000 7ffb67fff700 1 -- 172.21.15.189:0/4069873621 >> v1:172.21.15.189:6789/0 conn(0x7ffb680605f0 legacy=0x7ffb681339a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.382 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.374+0000 7ffb67fff700 1 -- 172.21.15.189:0/4069873621 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb68067fa0 con 0x7ffb680612b0 2020-11-20T12:05:07.382 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.374+0000 7ffb677fe700 1 --2- 172.21.15.189:0/4069873621 >> v2:172.21.15.47:3300/0 conn(0x7ffb68060170 0x7ffb680a85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:07.383 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.374+0000 7ffb67fff700 1 --2- 172.21.15.189:0/4069873621 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ffb680612b0 0x7ffb680a0730 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7ffb5800b9d0 tx=0x7ffb58006df0).ready entity=mon.2 client_cookie=a0ec143652982bdd server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:07.383 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.374+0000 7ffb66ffd700 1 -- 172.21.15.189:0/4069873621 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb58009770 con 0x7ffb680612b0 2020-11-20T12:05:07.383 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.374+0000 7ffb66ffd700 1 -- 172.21.15.189:0/4069873621 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb580098d0 con 0x7ffb680612b0 2020-11-20T12:05:07.383 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.374+0000 7ffb66ffd700 1 -- 172.21.15.189:0/4069873621 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb58009c50 con 0x7ffb680612b0 2020-11-20T12:05:07.386 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.374+0000 7ffb6e420700 1 -- 172.21.15.189:0/4069873621 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ffb680612b0 msgr2=0x7ffb680a0730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.386 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.374+0000 7ffb6e420700 1 --2- 172.21.15.189:0/4069873621 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ffb680612b0 0x7ffb680a0730 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7ffb5800b9d0 tx=0x7ffb58006df0).stop 2020-11-20T12:05:07.387 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.374+0000 7ffb6e420700 1 -- 172.21.15.189:0/4069873621 shutdown_connections 2020-11-20T12:05:07.387 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.374+0000 7ffb6e420700 1 --2- 172.21.15.189:0/4069873621 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ffb680612b0 0x7ffb680a0730 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.387 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.374+0000 7ffb6e420700 1 --2- 172.21.15.189:0/4069873621 >> v2:172.21.15.47:3300/0 conn(0x7ffb68060170 0x7ffb680a85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.387 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.374+0000 7ffb6e420700 1 -- 172.21.15.189:0/4069873621 >> 172.21.15.189:0/4069873621 conn(0x7ffb68065910 msgr2=0x7ffb68060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:07.387 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.374+0000 7ffb6e420700 1 -- 172.21.15.189:0/4069873621 shutdown_connections 2020-11-20T12:05:07.388 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.374+0000 7ffb6e420700 1 -- 172.21.15.189:0/4069873621 wait complete. 2020-11-20T12:05:07.388 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.374+0000 7ffb6e420700 1 Processor -- start 2020-11-20T12:05:07.388 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.374+0000 7ffb6e420700 1 -- start start 2020-11-20T12:05:07.388 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb6e420700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb68060170 0x7ffb680a5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.389 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb6e420700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ffb680605f0 0x7ffb68139a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.390 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb6e420700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb680612b0 0x7ffb680a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.390 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb6e420700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb68139ef0 con 0x7ffb68060170 2020-11-20T12:05:07.390 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb6e420700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb68139300 con 0x7ffb680612b0 2020-11-20T12:05:07.391 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb6e420700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb681267b0 con 0x7ffb680605f0 2020-11-20T12:05:07.391 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb6c9bd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb680612b0 0x7ffb680a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.391 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb6c9bd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb680612b0 0x7ffb680a4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48838/0 (socket says 172.21.15.189:48838) 2020-11-20T12:05:07.391 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb6c9bd700 1 -- 172.21.15.189:0/1382826944 learned_addr learned my addr 172.21.15.189:0/1382826944 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:07.392 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb677fe700 1 --2- 172.21.15.189:0/1382826944 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ffb680605f0 0x7ffb68139a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.392 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb67fff700 1 --2- 172.21.15.189:0/1382826944 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb68060170 0x7ffb680a5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.392 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb6c9bd700 1 -- 172.21.15.189:0/1382826944 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ffb680605f0 msgr2=0x7ffb68139a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.392 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb6c9bd700 1 --2- 172.21.15.189:0/1382826944 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ffb680605f0 0x7ffb68139a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.392 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb6c9bd700 1 -- 172.21.15.189:0/1382826944 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb68060170 msgr2=0x7ffb680a5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.393 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb6c9bd700 1 --2- 172.21.15.189:0/1382826944 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb68060170 0x7ffb680a5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.393 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb6c9bd700 1 -- 172.21.15.189:0/1382826944 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb68139560 con 0x7ffb680612b0 2020-11-20T12:05:07.393 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb6c9bd700 1 --2- 172.21.15.189:0/1382826944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb680612b0 0x7ffb680a4f10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7ffb6000ca60 tx=0x7ffb60007de0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:07.393 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb47fff700 1 -- 172.21.15.189:0/1382826944 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb60005b80 con 0x7ffb680612b0 2020-11-20T12:05:07.393 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb47fff700 1 -- 172.21.15.189:0/1382826944 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb60005ce0 con 0x7ffb680612b0 2020-11-20T12:05:07.394 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb47fff700 1 -- 172.21.15.189:0/1382826944 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb60007320 con 0x7ffb680612b0 2020-11-20T12:05:07.394 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb6e420700 1 -- 172.21.15.189:0/1382826944 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb68137eb0 con 0x7ffb680612b0 2020-11-20T12:05:07.394 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb6e420700 1 -- 172.21.15.189:0/1382826944 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb680a5400 con 0x7ffb680612b0 2020-11-20T12:05:07.394 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb47fff700 1 -- 172.21.15.189:0/1382826944 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7ffb60010030 con 0x7ffb680612b0 2020-11-20T12:05:07.395 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb6e420700 1 -- 172.21.15.189:0/1382826944 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb681396f0 con 0x7ffb680612b0 2020-11-20T12:05:07.395 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb47fff700 1 --2- 172.21.15.189:0/1382826944 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ffb400557c0 0x7ffb40057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:07.395 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb47fff700 1 -- 172.21.15.189:0/1382826944 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ffb6006e6d0 con 0x7ffb680612b0 2020-11-20T12:05:07.395 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.378+0000 7ffb67fff700 1 --2- 172.21.15.189:0/1382826944 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ffb400557c0 0x7ffb40057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:07.396 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.382+0000 7ffb67fff700 1 --2- 172.21.15.189:0/1382826944 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ffb400557c0 0x7ffb40057c80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7ffb5800b150 tx=0x7ffb58015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:07.396 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.386+0000 7ffb47fff700 1 -- 172.21.15.189:0/1382826944 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ffb60044570 con 0x7ffb680612b0 2020-11-20T12:05:07.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.618+0000 7ffb6e420700 1 -- 172.21.15.189:0/1382826944 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb68061690 con 0x7ffb680612b0 2020-11-20T12:05:07.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.618+0000 7ffb47fff700 1 -- 172.21.15.189:0/1382826944 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2134 (secure 0 0 0) 0x7ffb60047b90 con 0x7ffb680612b0 2020-11-20T12:05:07.625 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:05:07.625 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2660417436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2660417436},{"type":"v1","addr":"172.21.15.47:6833","nonce":2660417436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:04:56.025055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.189:6835/1042140684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1042140684},{"type":"v1","addr":"172.21.15.189:6835","nonce":1042140684}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:05:07.627 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.618+0000 7ffb6e420700 1 -- 172.21.15.189:0/1382826944 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ffb400557c0 msgr2=0x7ffb40057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.627 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.618+0000 7ffb6e420700 1 --2- 172.21.15.189:0/1382826944 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ffb400557c0 0x7ffb40057c80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7ffb5800b150 tx=0x7ffb58015000).stop 2020-11-20T12:05:07.627 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.618+0000 7ffb6e420700 1 -- 172.21.15.189:0/1382826944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb680612b0 msgr2=0x7ffb680a4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:07.628 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.618+0000 7ffb6e420700 1 --2- 172.21.15.189:0/1382826944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb680612b0 0x7ffb680a4f10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7ffb6000ca60 tx=0x7ffb60007de0).stop 2020-11-20T12:05:07.628 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.618+0000 7ffb6e420700 1 -- 172.21.15.189:0/1382826944 shutdown_connections 2020-11-20T12:05:07.628 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.618+0000 7ffb6e420700 1 --2- 172.21.15.189:0/1382826944 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ffb400557c0 0x7ffb40057c80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.628 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.618+0000 7ffb6e420700 1 --2- 172.21.15.189:0/1382826944 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffb68060170 0x7ffb680a5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.628 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.618+0000 7ffb6e420700 1 --2- 172.21.15.189:0/1382826944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb680612b0 0x7ffb680a4f10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.618+0000 7ffb6e420700 1 --2- 172.21.15.189:0/1382826944 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ffb680605f0 0x7ffb68139a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:07.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.618+0000 7ffb6e420700 1 -- 172.21.15.189:0/1382826944 >> 172.21.15.189:0/1382826944 conn(0x7ffb68065910 msgr2=0x7ffb68133ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:07.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.618+0000 7ffb6e420700 1 -- 172.21.15.189:0/1382826944 shutdown_connections 2020-11-20T12:05:07.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:07.618+0000 7ffb6e420700 1 -- 172.21.15.189:0/1382826944 wait complete. 2020-11-20T12:05:07.630 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 6 2020-11-20T12:05:07.841 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:04:54.012110+0000', 'modified': '2020-11-20T12:04:56.025055+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4273}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4273': {'gid': 4273, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.189:6835/1042140684', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6834', 'nonce': 1042140684}, {'type': 'v1', 'addr': '172.21.15.189:6835', 'nonce': 1042140684}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:05:07.842 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:05:07.843 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:05:07.981 INFO:teuthology.orchestra.run.smithi189.stdout:{"cluster_fsid":"cda18d0e-64b3-41bb-964a-084069c3e648","whoami":0,"id":4273,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":13.952432491,"mdsmap_epoch":6,"osdmap_epoch":22,"osdmap_epoch_barrier":22,"uptime":17.366338922000001} 2020-11-20T12:05:07.982 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "cda18d0e-64b3-41bb-964a-084069c3e648", "fs_name": "cephfs", "id": 4273, "mdsmap_epoch": 6, "osdmap_epoch": 22, "osdmap_epoch_barrier": 22, "rank_uptime": 13.952432491, "state": "up:active", "uptime": 17.366338922, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:05:07.982 INFO:teuthology.run_tasks:Running task ceph-fuse... 2020-11-20T12:05:08.007 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2020-11-20T12:05:08.007 INFO:tasks.ceph_fuse:config is None 2020-11-20T12:05:08.007 INFO:tasks.ceph_fuse:new config is {'client.0': None} 2020-11-20T12:05:08.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:08.009 INFO:teuthology.orchestra.run.smithi189:> ip netns list 2020-11-20T12:05:08.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:08.016 INFO:teuthology.orchestra.run.smithi189:> sudo ip link delete ceph-brx 2020-11-20T12:05:08.077 INFO:teuthology.orchestra.run.smithi189.stderr:Cannot find device "ceph-brx" 2020-11-20T12:05:08.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-20T12:05:08.078 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2020-11-20T12:05:08.079 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2020-11-20T12:05:08.079 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi189.front.sepia.ceph.com 2020-11-20T12:05:08.079 INFO:tasks.cephfs.mount:self.client.name = client.0 2020-11-20T12:05:08.080 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2020-11-20T12:05:08.080 INFO:tasks.cephfs.mount:self.cephfs_name = None 2020-11-20T12:05:08.080 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2020-11-20T12:05:08.080 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2020-11-20T12:05:08.081 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:05:08.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:08.081 INFO:teuthology.orchestra.run.smithi189:> ip addr 2020-11-20T12:05:08.126 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T12:05:08.126 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T12:05:08.126 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T12:05:08.127 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:08.127 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2020-11-20T12:05:08.127 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:08.127 INFO:teuthology.orchestra.run.smithi189.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2020-11-20T12:05:08.128 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:18 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:05:08.128 INFO:teuthology.orchestra.run.smithi189.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2020-11-20T12:05:08.128 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:19 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:05:08.128 INFO:teuthology.orchestra.run.smithi189.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2020-11-20T12:05:08.129 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f0 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:05:08.129 INFO:teuthology.orchestra.run.smithi189.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2020-11-20T12:05:08.129 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f1 brd ff:ff:ff:ff:ff:ff 2020-11-20T12:05:08.129 INFO:teuthology.orchestra.run.smithi189.stdout: inet 172.21.15.189/20 brd 172.21.15.255 scope global enp3s0f1 2020-11-20T12:05:08.129 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:08.130 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::ec4:7aff:fe88:6df1/64 scope link 2020-11-20T12:05:08.130 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:08.130 INFO:teuthology.orchestra.run.smithi189.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2020-11-20T12:05:08.130 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 02:42:2d:71:2a:7c brd ff:ff:ff:ff:ff:ff 2020-11-20T12:05:08.130 INFO:teuthology.orchestra.run.smithi189.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2020-11-20T12:05:08.131 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2020-11-20T12:05:08.131 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2020-11-20T12:05:08.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:08.132 INFO:teuthology.orchestra.run.smithi189:> (cd / && exec sudo bash -c ' 2020-11-20T12:05:08.132 INFO:teuthology.orchestra.run.smithi189:> set -e 2020-11-20T12:05:08.133 INFO:teuthology.orchestra.run.smithi189:> sudo ip link add name ceph-brx type bridge 2020-11-20T12:05:08.133 INFO:teuthology.orchestra.run.smithi189:> sudo ip addr flush dev ceph-brx 2020-11-20T12:05:08.133 INFO:teuthology.orchestra.run.smithi189:> sudo ip link set ceph-brx up 2020-11-20T12:05:08.134 INFO:teuthology.orchestra.run.smithi189:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2020-11-20T12:05:08.134 INFO:teuthology.orchestra.run.smithi189:> ') 2020-11-20T12:05:08.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:08.219 INFO:teuthology.orchestra.run.smithi189:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2020-11-20T12:05:08.231 INFO:teuthology.orchestra.run.smithi189.stdout:1 2020-11-20T12:05:08.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:08.232 INFO:teuthology.orchestra.run.smithi189:> route 2020-11-20T12:05:08.456 INFO:teuthology.orchestra.run.smithi189.stdout:Kernel IP routing table 2020-11-20T12:05:08.457 INFO:teuthology.orchestra.run.smithi189.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2020-11-20T12:05:08.457 INFO:teuthology.orchestra.run.smithi189.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2020-11-20T12:05:08.457 INFO:teuthology.orchestra.run.smithi189.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2020-11-20T12:05:08.457 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2020-11-20T12:05:08.458 INFO:teuthology.orchestra.run.smithi189.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2020-11-20T12:05:08.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:08.459 INFO:teuthology.orchestra.run.smithi189:> (cd / && exec sudo bash -c ' 2020-11-20T12:05:08.459 INFO:teuthology.orchestra.run.smithi189:> set -e 2020-11-20T12:05:08.459 INFO:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2020-11-20T12:05:08.460 INFO:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2020-11-20T12:05:08.460 INFO:teuthology.orchestra.run.smithi189:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2020-11-20T12:05:08.460 INFO:teuthology.orchestra.run.smithi189:> ') 2020-11-20T12:05:08.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:08.503 INFO:teuthology.orchestra.run.smithi189:> ip netns list 2020-11-20T12:05:08.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:08.554 INFO:teuthology.orchestra.run.smithi189:> ip netns list-id 2020-11-20T12:05:08.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:08.602 INFO:teuthology.orchestra.run.smithi189:> (cd / && exec sudo bash -c ' 2020-11-20T12:05:08.602 INFO:teuthology.orchestra.run.smithi189:> set -e 2020-11-20T12:05:08.602 INFO:teuthology.orchestra.run.smithi189:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2020-11-20T12:05:08.603 INFO:teuthology.orchestra.run.smithi189:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2020-11-20T12:05:08.603 INFO:teuthology.orchestra.run.smithi189:> ') 2020-11-20T12:05:08.678 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2020-11-20T12:05:08.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:08.679 INFO:teuthology.orchestra.run.smithi189:> (cd / && exec sudo bash -c ' 2020-11-20T12:05:08.679 INFO:teuthology.orchestra.run.smithi189:> set -e 2020-11-20T12:05:08.679 INFO:teuthology.orchestra.run.smithi189:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2020-11-20T12:05:08.680 INFO:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2020-11-20T12:05:08.680 INFO:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2020-11-20T12:05:08.680 INFO:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2020-11-20T12:05:08.681 INFO:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2020-11-20T12:05:08.681 INFO:teuthology.orchestra.run.smithi189:> ') 2020-11-20T12:05:09.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T12:05:09.055 INFO:teuthology.orchestra.run.smithi189:> (cd / && exec sudo bash -c ' 2020-11-20T12:05:09.055 INFO:teuthology.orchestra.run.smithi189:> set -e 2020-11-20T12:05:09.055 INFO:teuthology.orchestra.run.smithi189:> sudo ip link set brx.0 up 2020-11-20T12:05:09.056 INFO:teuthology.orchestra.run.smithi189:> sudo ip link set dev brx.0 master ceph-brx 2020-11-20T12:05:09.056 INFO:teuthology.orchestra.run.smithi189:> ') 2020-11-20T12:05:09.091 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:05:09.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50ca89e700 1 Processor -- start 2020-11-20T12:05:09.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50ca89e700 1 -- start start 2020-11-20T12:05:09.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50ca89e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f50c40a8840 0x7f50c413dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50ca89e700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f50c413e100 0x7f50c41371b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50ca89e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50c4066800 con 0x7f50c41376a0 2020-11-20T12:05:09.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50ca89e700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50c4066aa0 con 0x7f50c413e100 2020-11-20T12:05:09.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50ca89e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50c40672f0 con 0x7f50c40a8840 2020-11-20T12:05:09.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50c989c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f50c40a8840 0x7f50c413dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50ca09d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f50c41376a0 0x7f50c413a680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50470/0 (socket says 172.21.15.189:50470) 2020-11-20T12:05:09.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50c989c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f50c40a8840 0x7f50c413dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48844/0 (socket says 172.21.15.189:48844) 2020-11-20T12:05:09.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50ca09d700 1 -- 172.21.15.189:0/1032374589 learned_addr learned my addr 172.21.15.189:0/1032374589 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:09.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50c909b700 1 --2- 172.21.15.189:0/1032374589 >> v2:172.21.15.47:3300/0 conn(0x7f50c413e100 0x7f50c41371b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50c989c700 1 -- 172.21.15.189:0/1032374589 >> v2:172.21.15.47:3300/0 conn(0x7f50c413e100 msgr2=0x7f50c41371b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:09.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50c989c700 1 --2- 172.21.15.189:0/1032374589 >> v2:172.21.15.47:3300/0 conn(0x7f50c413e100 0x7f50c41371b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.269 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50c989c700 1 -- 172.21.15.189:0/1032374589 >> v1:172.21.15.189:6789/0 conn(0x7f50c41376a0 legacy=0x7f50c413a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.269 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50c989c700 1 -- 172.21.15.189:0/1032374589 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c4067590 con 0x7f50c40a8840 2020-11-20T12:05:09.269 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50c989c700 1 --2- 172.21.15.189:0/1032374589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50c40a8840 0x7f50c413dc10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f50c4068350 tx=0x7f50c0006df0).ready entity=mon.1 client_cookie=2b7024617109188a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:09.270 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50c889a700 1 -- 172.21.15.189:0/1032374589 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c00097b0 con 0x7f50c40a8840 2020-11-20T12:05:09.270 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50c889a700 1 -- 172.21.15.189:0/1032374589 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c0009910 con 0x7f50c40a8840 2020-11-20T12:05:09.270 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50c889a700 1 -- 172.21.15.189:0/1032374589 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c0009c90 con 0x7f50c40a8840 2020-11-20T12:05:09.270 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50ca89e700 1 -- 172.21.15.189:0/1032374589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50c40a8840 msgr2=0x7f50c413dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.270 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50ca89e700 1 --2- 172.21.15.189:0/1032374589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50c40a8840 0x7f50c413dc10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f50c4068350 tx=0x7f50c0006df0).stop 2020-11-20T12:05:09.271 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50ca89e700 1 -- 172.21.15.189:0/1032374589 shutdown_connections 2020-11-20T12:05:09.271 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50ca89e700 1 --2- 172.21.15.189:0/1032374589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50c40a8840 0x7f50c413dc10 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.271 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50ca89e700 1 --2- 172.21.15.189:0/1032374589 >> v2:172.21.15.47:3300/0 conn(0x7f50c413e100 0x7f50c41371b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.271 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50ca89e700 1 -- 172.21.15.189:0/1032374589 >> 172.21.15.189:0/1032374589 conn(0x7f50c4060170 msgr2=0x7f50c4060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:09.271 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50ca89e700 1 -- 172.21.15.189:0/1032374589 shutdown_connections 2020-11-20T12:05:09.272 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.258+0000 7f50ca89e700 1 -- 172.21.15.189:0/1032374589 wait complete. 2020-11-20T12:05:09.274 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.262+0000 7f50ca89e700 1 Processor -- start 2020-11-20T12:05:09.274 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.262+0000 7f50ca89e700 1 -- start start 2020-11-20T12:05:09.274 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.262+0000 7f50ca89e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50c40a8840 0x7f50c40a5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.275 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.262+0000 7f50ca89e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50c41376a0 0x7f50c414a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.275 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.262+0000 7f50ca89e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f50c413e100 0x7f50c414a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.275 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.262+0000 7f50ca89e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c4149d80 con 0x7f50c40a8840 2020-11-20T12:05:09.275 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.262+0000 7f50ca89e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c40a5620 con 0x7f50c41376a0 2020-11-20T12:05:09.276 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.262+0000 7f50ca89e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50c4149f70 con 0x7f50c413e100 2020-11-20T12:05:09.276 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.262+0000 7f50c909b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50c41376a0 0x7f50c414a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.276 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.262+0000 7f50c909b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50c41376a0 0x7f50c414a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48852/0 (socket says 172.21.15.189:48852) 2020-11-20T12:05:09.276 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.262+0000 7f50c909b700 1 -- 172.21.15.189:0/411682679 learned_addr learned my addr 172.21.15.189:0/411682679 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:09.276 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.262+0000 7f50ca09d700 1 --2- 172.21.15.189:0/411682679 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f50c413e100 0x7f50c414a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.277 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.262+0000 7f50c989c700 1 --2- 172.21.15.189:0/411682679 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50c40a8840 0x7f50c40a5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.277 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.262+0000 7f50c909b700 1 -- 172.21.15.189:0/411682679 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f50c413e100 msgr2=0x7f50c414a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.277 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.262+0000 7f50c909b700 1 --2- 172.21.15.189:0/411682679 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f50c413e100 0x7f50c414a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.277 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.262+0000 7f50c909b700 1 -- 172.21.15.189:0/411682679 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50c40a8840 msgr2=0x7f50c40a5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.277 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.262+0000 7f50c909b700 1 --2- 172.21.15.189:0/411682679 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50c40a8840 0x7f50c40a5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.278 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.262+0000 7f50c909b700 1 -- 172.21.15.189:0/411682679 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c40a4aa0 con 0x7f50c41376a0 2020-11-20T12:05:09.278 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.262+0000 7f50c909b700 1 --2- 172.21.15.189:0/411682679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50c41376a0 0x7f50c414a410 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f50b400d8e0 tx=0x7f50b4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:09.278 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.262+0000 7f50b1ffb700 1 -- 172.21.15.189:0/411682679 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b40125f0 con 0x7f50c41376a0 2020-11-20T12:05:09.278 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.262+0000 7f50b1ffb700 1 -- 172.21.15.189:0/411682679 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b4012750 con 0x7f50c41376a0 2020-11-20T12:05:09.278 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.262+0000 7f50ca89e700 1 -- 172.21.15.189:0/411682679 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c40a4d50 con 0x7f50c41376a0 2020-11-20T12:05:09.279 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.262+0000 7f50b1ffb700 1 -- 172.21.15.189:0/411682679 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b4012af0 con 0x7f50c41376a0 2020-11-20T12:05:09.279 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.262+0000 7f50ca89e700 1 -- 172.21.15.189:0/411682679 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50c40a52a0 con 0x7f50c41376a0 2020-11-20T12:05:09.279 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.262+0000 7f50b1ffb700 1 -- 172.21.15.189:0/411682679 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f50b4013480 con 0x7f50c41376a0 2020-11-20T12:05:09.279 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.266+0000 7f50b1ffb700 1 --2- 172.21.15.189:0/411682679 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f509c0557c0 0x7f509c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.279 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.266+0000 7f50ca89e700 1 -- 172.21.15.189:0/411682679 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50c40a3b90 con 0x7f50c41376a0 2020-11-20T12:05:09.280 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.266+0000 7f50b1ffb700 1 -- 172.21.15.189:0/411682679 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f50b406e5e0 con 0x7f50c41376a0 2020-11-20T12:05:09.280 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.266+0000 7f50c989c700 1 --2- 172.21.15.189:0/411682679 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f509c0557c0 0x7f509c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.280 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.266+0000 7f50c989c700 1 --2- 172.21.15.189:0/411682679 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f509c0557c0 0x7f509c057c80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f50c000c010 tx=0x7f50c0015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:09.281 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.274+0000 7f50b1ffb700 1 -- 172.21.15.189:0/411682679 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f50b4044420 con 0x7f50c41376a0 2020-11-20T12:05:09.430 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.422+0000 7f50ca89e700 1 -- 172.21.15.189:0/411682679 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50c40a9230 con 0x7f50c41376a0 2020-11-20T12:05:09.431 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.426+0000 7f50b1ffb700 1 -- 172.21.15.189:0/411682679 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16826 (secure 0 0 0) 0x7f50b4012e60 con 0x7f50c41376a0 2020-11-20T12:05:09.432 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:05:09.432 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":22,"fsid":"cda18d0e-64b3-41bb-964a-084069c3e648","created":"2020-11-20T12:04:37.538706+0000","modified":"2020-11-20T12:04:55.015818+0000","last_up_change":"2020-11-20T12:04:43.006631+0000","last_in_change":"2020-11-20T12:04:43.006631+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:41.395395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:48.550454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:52.101837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:53.018196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c81214e1-4f84-4aaa-8ceb-99c398b2a89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6811","nonce":11760},{"type":"v1","addr":"172.21.15.189:6814","nonce":11760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":11760},{"type":"v1","addr":"172.21.15.189:6818","nonce":11760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":11760},{"type":"v1","addr":"172.21.15.189:6828","nonce":11760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6821","nonce":11760},{"type":"v1","addr":"172.21.15.189:6824","nonce":11760}]},"public_addr":"172.21.15.189:6814/11760","cluster_addr":"172.21.15.189:6818/11760","heartbeat_back_addr":"172.21.15.189:6828/11760","heartbeat_front_addr":"172.21.15.189:6824/11760","state":["exists","up"]},{"osd":1,"uuid":"d2f3ec7d-d403-484f-b5ed-b2e6b079d63d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6805","nonce":11761},{"type":"v1","addr":"172.21.15.189:6807","nonce":11761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":11761},{"type":"v1","addr":"172.21.15.189:6812","nonce":11761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6817","nonce":11761},{"type":"v1","addr":"172.21.15.189:6820","nonce":11761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6813","nonce":11761},{"type":"v1","addr":"172.21.15.189:6815","nonce":11761}]},"public_addr":"172.21.15.189:6807/11761","cluster_addr":"172.21.15.189:6812/11761","heartbeat_back_addr":"172.21.15.189:6820/11761","heartbeat_front_addr":"172.21.15.189:6815/11761","state":["exists","up"]},{"osd":2,"uuid":"e6cf4790-000c-43eb-8844-175f04c60503","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6819","nonce":11751},{"type":"v1","addr":"172.21.15.189:6822","nonce":11751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":11751},{"type":"v1","addr":"172.21.15.189:6825","nonce":11751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":11751},{"type":"v1","addr":"172.21.15.189:6831","nonce":11751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6827","nonce":11751},{"type":"v1","addr":"172.21.15.189:6829","nonce":11751}]},"public_addr":"172.21.15.189:6822/11751","cluster_addr":"172.21.15.189:6825/11751","heartbeat_back_addr":"172.21.15.189:6831/11751","heartbeat_front_addr":"172.21.15.189:6829/11751","state":["exists","up"]},{"osd":3,"uuid":"ba49dfac-f2b9-476e-853d-42f05e21a77a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":11762},{"type":"v1","addr":"172.21.15.189:6801","nonce":11762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":11762},{"type":"v1","addr":"172.21.15.189:6803","nonce":11762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":11762},{"type":"v1","addr":"172.21.15.189:6809","nonce":11762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":11762},{"type":"v1","addr":"172.21.15.189:6806","nonce":11762}]},"public_addr":"172.21.15.189:6801/11762","cluster_addr":"172.21.15.189:6803/11762","heartbeat_back_addr":"172.21.15.189:6809/11762","heartbeat_front_addr":"172.21.15.189:6806/11762","state":["exists","up"]},{"osd":4,"uuid":"3ca7b438-38b2-4689-8b18-fb5a8ddf6aee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":12114},{"type":"v1","addr":"172.21.15.47:6825","nonce":12114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":12114},{"type":"v1","addr":"172.21.15.47:6827","nonce":12114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":12114},{"type":"v1","addr":"172.21.15.47:6831","nonce":12114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":12114},{"type":"v1","addr":"172.21.15.47:6829","nonce":12114}]},"public_addr":"172.21.15.47:6825/12114","cluster_addr":"172.21.15.47:6827/12114","heartbeat_back_addr":"172.21.15.47:6831/12114","heartbeat_front_addr":"172.21.15.47:6829/12114","state":["exists","up"]},{"osd":5,"uuid":"6a45fca7-83d3-44fd-a1d4-b01e4201a101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":12111},{"type":"v1","addr":"172.21.15.47:6801","nonce":12111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":12111},{"type":"v1","addr":"172.21.15.47:6803","nonce":12111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":12111},{"type":"v1","addr":"172.21.15.47:6807","nonce":12111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":12111},{"type":"v1","addr":"172.21.15.47:6805","nonce":12111}]},"public_addr":"172.21.15.47:6801/12111","cluster_addr":"172.21.15.47:6803/12111","heartbeat_back_addr":"172.21.15.47:6807/12111","heartbeat_front_addr":"172.21.15.47:6805/12111","state":["exists","up"]},{"osd":6,"uuid":"c4b1b9ed-61f8-4677-9755-dff59a30e447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":12113},{"type":"v1","addr":"172.21.15.47:6817","nonce":12113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":12113},{"type":"v1","addr":"172.21.15.47:6819","nonce":12113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":12113},{"type":"v1","addr":"172.21.15.47:6823","nonce":12113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":12113},{"type":"v1","addr":"172.21.15.47:6821","nonce":12113}]},"public_addr":"172.21.15.47:6817/12113","cluster_addr":"172.21.15.47:6819/12113","heartbeat_back_addr":"172.21.15.47:6823/12113","heartbeat_front_addr":"172.21.15.47:6821/12113","state":["exists","up"]},{"osd":7,"uuid":"4608e4ed-cc40-4c5c-ba8b-3779bbc62f65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":12112},{"type":"v1","addr":"172.21.15.47:6809","nonce":12112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":12112},{"type":"v1","addr":"172.21.15.47:6811","nonce":12112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":12112},{"type":"v1","addr":"172.21.15.47:6815","nonce":12112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":12112},{"type":"v1","addr":"172.21.15.47:6813","nonce":12112}]},"public_addr":"172.21.15.47:6809/12112","cluster_addr":"172.21.15.47:6811/12112","heartbeat_back_addr":"172.21.15.47:6815/12112","heartbeat_front_addr":"172.21.15.47:6813/12112","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.435265+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.472776+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.388370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.417366+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.102392+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.116529+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.167430+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.148139+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:05:09.433 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.426+0000 7f50ca89e700 1 -- 172.21.15.189:0/411682679 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f509c0557c0 msgr2=0x7f509c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.433 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.426+0000 7f50ca89e700 1 --2- 172.21.15.189:0/411682679 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f509c0557c0 0x7f509c057c80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f50c000c010 tx=0x7f50c0015000).stop 2020-11-20T12:05:09.433 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.426+0000 7f50ca89e700 1 -- 172.21.15.189:0/411682679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50c41376a0 msgr2=0x7f50c414a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.426+0000 7f50ca89e700 1 --2- 172.21.15.189:0/411682679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50c41376a0 0x7f50c414a410 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f50b400d8e0 tx=0x7f50b4008040).stop 2020-11-20T12:05:09.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.426+0000 7f50ca89e700 1 -- 172.21.15.189:0/411682679 shutdown_connections 2020-11-20T12:05:09.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.426+0000 7f50ca89e700 1 --2- 172.21.15.189:0/411682679 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f509c0557c0 0x7f509c057c80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.426+0000 7f50ca89e700 1 --2- 172.21.15.189:0/411682679 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50c40a8840 0x7f50c40a5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.426+0000 7f50ca89e700 1 --2- 172.21.15.189:0/411682679 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f50c413e100 0x7f50c414a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.426+0000 7f50ca89e700 1 --2- 172.21.15.189:0/411682679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50c41376a0 0x7f50c414a410 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.426+0000 7f50ca89e700 1 -- 172.21.15.189:0/411682679 >> 172.21.15.189:0/411682679 conn(0x7f50c4060170 msgr2=0x7f50c4135000 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:09.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.426+0000 7f50ca89e700 1 -- 172.21.15.189:0/411682679 shutdown_connections 2020-11-20T12:05:09.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.426+0000 7f50ca89e700 1 -- 172.21.15.189:0/411682679 wait complete. 2020-11-20T12:05:09.446 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:41.395395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:48.550454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:04:52.101837+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:04:53.018196+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:05:09.447 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:05:09.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.574+0000 7f412aeb7700 1 Processor -- start 2020-11-20T12:05:09.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.574+0000 7f412aeb7700 1 -- start start 2020-11-20T12:05:09.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.574+0000 7f412aeb7700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f41240ae0d0 0x7f412413aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.574+0000 7f412aeb7700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4124060880 0x7f41241451e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.574+0000 7f412aeb7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4124066960 con 0x7f412413af60 2020-11-20T12:05:09.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.574+0000 7f412aeb7700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f412413a680 con 0x7f41240ae0d0 2020-11-20T12:05:09.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.574+0000 7f412aeb7700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4124067110 con 0x7f4124060880 2020-11-20T12:05:09.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.574+0000 7f4123fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f412413af60 0x7f4124060170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50482/0 (socket says 172.21.15.189:50482) 2020-11-20T12:05:09.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.574+0000 7f4123fff700 1 -- 172.21.15.189:0/3537413192 learned_addr learned my addr 172.21.15.189:0/3537413192 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:09.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.574+0000 7f4129454700 1 --2- 172.21.15.189:0/3537413192 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4124060880 0x7f41241451e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.574+0000 7f4128c53700 1 --2- 172.21.15.189:0/3537413192 >> v2:172.21.15.47:3300/0 conn(0x7f41240ae0d0 0x7f412413aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.574+0000 7f41237fe700 1 -- 172.21.15.189:0/3537413192 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807444050 0 0) 0x7f4124066960 con 0x7f412413af60 2020-11-20T12:05:09.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.574+0000 7f41237fe700 1 -- 172.21.15.189:0/3537413192 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4110003240 con 0x7f412413af60 2020-11-20T12:05:09.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.574+0000 7f4129454700 1 -- 172.21.15.189:0/3537413192 >> v2:172.21.15.47:3300/0 conn(0x7f41240ae0d0 msgr2=0x7f412413aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:09.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.574+0000 7f4129454700 1 --2- 172.21.15.189:0/3537413192 >> v2:172.21.15.47:3300/0 conn(0x7f41240ae0d0 0x7f412413aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.574+0000 7f4129454700 1 -- 172.21.15.189:0/3537413192 >> v1:172.21.15.189:6789/0 conn(0x7f412413af60 legacy=0x7f4124060170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.574+0000 7f4129454700 1 -- 172.21.15.189:0/3537413192 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41240673b0 con 0x7f4124060880 2020-11-20T12:05:09.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.574+0000 7f4129454700 1 --2- 172.21.15.189:0/3537413192 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4124060880 0x7f41241451e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f411c00b9d0 tx=0x7f411c006df0).ready entity=mon.2 client_cookie=e27796f2a8d37408 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:09.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.574+0000 7f41237fe700 1 -- 172.21.15.189:0/3537413192 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411c009640 con 0x7f4124060880 2020-11-20T12:05:09.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.574+0000 7f41237fe700 1 -- 172.21.15.189:0/3537413192 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411c0097a0 con 0x7f4124060880 2020-11-20T12:05:09.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.574+0000 7f41237fe700 1 -- 172.21.15.189:0/3537413192 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411c009b20 con 0x7f4124060880 2020-11-20T12:05:09.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f412aeb7700 1 -- 172.21.15.189:0/3537413192 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4124060880 msgr2=0x7f41241451e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f412aeb7700 1 --2- 172.21.15.189:0/3537413192 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4124060880 0x7f41241451e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f411c00b9d0 tx=0x7f411c006df0).stop 2020-11-20T12:05:09.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f412aeb7700 1 -- 172.21.15.189:0/3537413192 shutdown_connections 2020-11-20T12:05:09.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f412aeb7700 1 --2- 172.21.15.189:0/3537413192 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4124060880 0x7f41241451e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f412aeb7700 1 --2- 172.21.15.189:0/3537413192 >> v2:172.21.15.47:3300/0 conn(0x7f41240ae0d0 0x7f412413aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f412aeb7700 1 -- 172.21.15.189:0/3537413192 >> 172.21.15.189:0/3537413192 conn(0x7f41240b1850 msgr2=0x7f41240ad820 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:09.592 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f412aeb7700 1 -- 172.21.15.189:0/3537413192 shutdown_connections 2020-11-20T12:05:09.592 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f412aeb7700 1 -- 172.21.15.189:0/3537413192 wait complete. 2020-11-20T12:05:09.592 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f412aeb7700 1 Processor -- start 2020-11-20T12:05:09.592 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f412aeb7700 1 -- start start 2020-11-20T12:05:09.592 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f412aeb7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4124060880 0x7f4124135e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.593 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f412aeb7700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f41240ae0d0 0x7f412413d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.593 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f412aeb7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f412413af60 0x7f412413d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.593 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f412aeb7700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4124149700 con 0x7f412413af60 2020-11-20T12:05:09.593 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f412aeb7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412413e620 con 0x7f4124060880 2020-11-20T12:05:09.593 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f412aeb7700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f412413e8c0 con 0x7f41240ae0d0 2020-11-20T12:05:09.594 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f4128c53700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4124060880 0x7f4124135e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.594 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f4128c53700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4124060880 0x7f4124135e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48864/0 (socket says 172.21.15.189:48864) 2020-11-20T12:05:09.594 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f4128c53700 1 -- 172.21.15.189:0/786971996 learned_addr learned my addr 172.21.15.189:0/786971996 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:09.594 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f4123fff700 1 --2- 172.21.15.189:0/786971996 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f41240ae0d0 0x7f412413d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.594 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f4129454700 1 --2- 172.21.15.189:0/786971996 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f412413af60 0x7f412413d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.595 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f4128c53700 1 -- 172.21.15.189:0/786971996 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f41240ae0d0 msgr2=0x7f412413d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.595 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f4128c53700 1 --2- 172.21.15.189:0/786971996 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f41240ae0d0 0x7f412413d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.595 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f4128c53700 1 -- 172.21.15.189:0/786971996 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f412413af60 msgr2=0x7f412413d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.595 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f4128c53700 1 --2- 172.21.15.189:0/786971996 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f412413af60 0x7f412413d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.595 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f4128c53700 1 -- 172.21.15.189:0/786971996 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41240a9910 con 0x7f4124060880 2020-11-20T12:05:09.596 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f4129454700 1 --2- 172.21.15.189:0/786971996 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f412413af60 0x7f412413d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:09.596 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f4128c53700 1 --2- 172.21.15.189:0/786971996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4124060880 0x7f4124135e70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f411400ef50 tx=0x7f41140091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:09.596 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.578+0000 7f4120ff9700 1 -- 172.21.15.189:0/786971996 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411400cd20 con 0x7f4124060880 2020-11-20T12:05:09.596 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.582+0000 7f4120ff9700 1 -- 172.21.15.189:0/786971996 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411400ce80 con 0x7f4124060880 2020-11-20T12:05:09.596 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.582+0000 7f4120ff9700 1 -- 172.21.15.189:0/786971996 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4114012600 con 0x7f4124060880 2020-11-20T12:05:09.597 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.582+0000 7f412aeb7700 1 -- 172.21.15.189:0/786971996 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f412413e180 con 0x7f4124060880 2020-11-20T12:05:09.597 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.582+0000 7f412aeb7700 1 -- 172.21.15.189:0/786971996 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f412413deb0 con 0x7f4124060880 2020-11-20T12:05:09.597 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.582+0000 7f412aeb7700 1 -- 172.21.15.189:0/786971996 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f412413ea80 con 0x7f4124060880 2020-11-20T12:05:09.597 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.582+0000 7f4120ff9700 1 -- 172.21.15.189:0/786971996 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f4114022070 con 0x7f4124060880 2020-11-20T12:05:09.597 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.582+0000 7f4120ff9700 1 --2- 172.21.15.189:0/786971996 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f40fc0557c0 0x7f40fc057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.598 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.582+0000 7f4123fff700 1 --2- 172.21.15.189:0/786971996 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f40fc0557c0 0x7f40fc057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.598 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.582+0000 7f4120ff9700 1 -- 172.21.15.189:0/786971996 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f411406efe0 con 0x7f4124060880 2020-11-20T12:05:09.599 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.582+0000 7f4123fff700 1 --2- 172.21.15.189:0/786971996 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f40fc0557c0 0x7f40fc057c80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f4124060520 tx=0x7f4118009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:09.599 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.586+0000 7f4120ff9700 1 -- 172.21.15.189:0/786971996 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4114044e40 con 0x7f4124060880 2020-11-20T12:05:09.744 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.738+0000 7f412aeb7700 1 -- 172.21.15.189:0/786971996 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f41240a9b00 con 0x7f4124060880 2020-11-20T12:05:09.744 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.738+0000 7f4120ff9700 1 -- 172.21.15.189:0/786971996 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v22) v1 ==== 110+0+10 (secure 0 0 0) 0x7f4114016020 con 0x7f4124060880 2020-11-20T12:05:09.745 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2020-11-20T12:05:09.746 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.738+0000 7f412aeb7700 1 -- 172.21.15.189:0/786971996 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f40fc0557c0 msgr2=0x7f40fc057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.746 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.738+0000 7f412aeb7700 1 --2- 172.21.15.189:0/786971996 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f40fc0557c0 0x7f40fc057c80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f4124060520 tx=0x7f4118009040).stop 2020-11-20T12:05:09.747 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.738+0000 7f412aeb7700 1 -- 172.21.15.189:0/786971996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4124060880 msgr2=0x7f4124135e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.747 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.738+0000 7f412aeb7700 1 --2- 172.21.15.189:0/786971996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4124060880 0x7f4124135e70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f411400ef50 tx=0x7f41140091d0).stop 2020-11-20T12:05:09.747 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.738+0000 7f412aeb7700 1 -- 172.21.15.189:0/786971996 shutdown_connections 2020-11-20T12:05:09.747 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.738+0000 7f412aeb7700 1 --2- 172.21.15.189:0/786971996 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f40fc0557c0 0x7f40fc057c80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.747 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.738+0000 7f412aeb7700 1 --2- 172.21.15.189:0/786971996 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f412413af60 0x7f412413d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.738+0000 7f412aeb7700 1 --2- 172.21.15.189:0/786971996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4124060880 0x7f4124135e70 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.738+0000 7f412aeb7700 1 --2- 172.21.15.189:0/786971996 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f41240ae0d0 0x7f412413d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.738+0000 7f412aeb7700 1 -- 172.21.15.189:0/786971996 >> 172.21.15.189:0/786971996 conn(0x7f41240b1850 msgr2=0x7f4124138b30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:09.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.738+0000 7f412aeb7700 1 -- 172.21.15.189:0/786971996 shutdown_connections 2020-11-20T12:05:09.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.738+0000 7f412aeb7700 1 -- 172.21.15.189:0/786971996 wait complete. 2020-11-20T12:05:09.765 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:05:09.889 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.882+0000 7f1bb9d99700 1 Processor -- start 2020-11-20T12:05:09.890 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.882+0000 7f1bb9d99700 1 -- start start 2020-11-20T12:05:09.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.882+0000 7f1bb9d99700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1bb4062930 0x7f1bb40b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.893 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.882+0000 7f1bb9d99700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1bb4066a30 0x7f1bb40b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.893 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.882+0000 7f1bb9d99700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1bb41373e0 0x7f1bb4137870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.893 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.882+0000 7f1bb9d99700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bb40666b0 con 0x7f1bb41373e0 2020-11-20T12:05:09.893 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.882+0000 7f1bb9d99700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bb40b1cb0 con 0x7f1bb4062930 2020-11-20T12:05:09.894 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.882+0000 7f1bb9d99700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb4066fb0 con 0x7f1bb4066a30 2020-11-20T12:05:09.894 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.882+0000 7f1bb8d97700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1bb4062930 0x7f1bb40b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.894 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.882+0000 7f1bb8d97700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1bb4062930 0x7f1bb40b12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48874/0 (socket says 172.21.15.189:48874) 2020-11-20T12:05:09.895 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.882+0000 7f1bb8d97700 1 -- 172.21.15.189:0/679418320 learned_addr learned my addr 172.21.15.189:0/679418320 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:09.896 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.882+0000 7f1bb3fff700 1 --2- 172.21.15.189:0/679418320 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1bb4066a30 0x7f1bb40b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.896 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.882+0000 7f1bb9598700 1 --2- 172.21.15.189:0/679418320 >> v2:172.21.15.47:3300/0 conn(0x7f1bb41373e0 0x7f1bb4137870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.896 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.882+0000 7f1bb8d97700 1 -- 172.21.15.189:0/679418320 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1bb4066a30 msgr2=0x7f1bb40b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:09.896 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.882+0000 7f1bb8d97700 1 --2- 172.21.15.189:0/679418320 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1bb4066a30 0x7f1bb40b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.897 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.882+0000 7f1bb8d97700 1 -- 172.21.15.189:0/679418320 >> v2:172.21.15.47:3300/0 conn(0x7f1bb41373e0 msgr2=0x7f1bb4137870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:09.897 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.882+0000 7f1bb8d97700 1 --2- 172.21.15.189:0/679418320 >> v2:172.21.15.47:3300/0 conn(0x7f1bb41373e0 0x7f1bb4137870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.897 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.882+0000 7f1bb8d97700 1 -- 172.21.15.189:0/679418320 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb4067250 con 0x7f1bb4062930 2020-11-20T12:05:09.897 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.882+0000 7f1bb8d97700 1 --2- 172.21.15.189:0/679418320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb4062930 0x7f1bb40b12d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f1bb4065880 tx=0x7f1ba4006eb0).ready entity=mon.1 client_cookie=fc08b7dd509f5330 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:09.897 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.882+0000 7f1bb37fe700 1 -- 172.21.15.189:0/679418320 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba40098d0 con 0x7f1bb4062930 2020-11-20T12:05:09.898 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.882+0000 7f1bb37fe700 1 -- 172.21.15.189:0/679418320 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba4009a30 con 0x7f1bb4062930 2020-11-20T12:05:09.898 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.882+0000 7f1bb37fe700 1 -- 172.21.15.189:0/679418320 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4009db0 con 0x7f1bb4062930 2020-11-20T12:05:09.898 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.886+0000 7f1bb9d99700 1 -- 172.21.15.189:0/679418320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb4062930 msgr2=0x7f1bb40b12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.898 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.886+0000 7f1bb9d99700 1 --2- 172.21.15.189:0/679418320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb4062930 0x7f1bb40b12d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f1bb4065880 tx=0x7f1ba4006eb0).stop 2020-11-20T12:05:09.898 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.886+0000 7f1bb9d99700 1 -- 172.21.15.189:0/679418320 shutdown_connections 2020-11-20T12:05:09.899 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.886+0000 7f1bb9d99700 1 --2- 172.21.15.189:0/679418320 >> v2:172.21.15.47:3300/0 conn(0x7f1bb41373e0 0x7f1bb4137870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.899 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.886+0000 7f1bb9d99700 1 --2- 172.21.15.189:0/679418320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb4062930 0x7f1bb40b12d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.899 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.886+0000 7f1bb9d99700 1 --2- 172.21.15.189:0/679418320 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1bb4066a30 0x7f1bb40b17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.899 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.886+0000 7f1bb9d99700 1 -- 172.21.15.189:0/679418320 >> 172.21.15.189:0/679418320 conn(0x7f1bb4065080 msgr2=0x7f1bb4062080 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:09.900 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.886+0000 7f1bb9d99700 1 -- 172.21.15.189:0/679418320 shutdown_connections 2020-11-20T12:05:09.900 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.886+0000 7f1bb9d99700 1 -- 172.21.15.189:0/679418320 wait complete. 2020-11-20T12:05:09.903 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.886+0000 7f1bb9d99700 1 Processor -- start 2020-11-20T12:05:09.904 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.886+0000 7f1bb9d99700 1 -- start start 2020-11-20T12:05:09.904 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.886+0000 7f1bb9d99700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb4062930 0x7f1bb414a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.904 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.886+0000 7f1bb9d99700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1bb4066a30 0x7f1bb4149af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.904 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.886+0000 7f1bb9d99700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1bb41373e0 0x7f1bb413bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.905 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.886+0000 7f1bb9d99700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb40a4cd0 con 0x7f1bb4066a30 2020-11-20T12:05:09.905 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.886+0000 7f1bb9d99700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb4138500 con 0x7f1bb4062930 2020-11-20T12:05:09.905 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.886+0000 7f1bb9d99700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb40a4f00 con 0x7f1bb41373e0 2020-11-20T12:05:09.905 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.886+0000 7f1bb8d97700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb4062930 0x7f1bb414a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.905 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.886+0000 7f1bb8d97700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb4062930 0x7f1bb414a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48882/0 (socket says 172.21.15.189:48882) 2020-11-20T12:05:09.906 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.886+0000 7f1bb8d97700 1 -- 172.21.15.189:0/1707926918 learned_addr learned my addr 172.21.15.189:0/1707926918 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:09.906 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.886+0000 7f1bb9598700 1 --2- 172.21.15.189:0/1707926918 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1bb41373e0 0x7f1bb413bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.906 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.886+0000 7f1bb3fff700 1 --2- 172.21.15.189:0/1707926918 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1bb4066a30 0x7f1bb4149af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.906 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.890+0000 7f1bb8d97700 1 -- 172.21.15.189:0/1707926918 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1bb41373e0 msgr2=0x7f1bb413bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.906 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.890+0000 7f1bb8d97700 1 --2- 172.21.15.189:0/1707926918 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1bb41373e0 0x7f1bb413bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.907 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.890+0000 7f1bb8d97700 1 -- 172.21.15.189:0/1707926918 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1bb4066a30 msgr2=0x7f1bb4149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:09.908 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.890+0000 7f1bb8d97700 1 --2- 172.21.15.189:0/1707926918 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1bb4066a30 0x7f1bb4149af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:09.908 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.890+0000 7f1bb8d97700 1 -- 172.21.15.189:0/1707926918 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb40a51a0 con 0x7f1bb4062930 2020-11-20T12:05:09.908 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.890+0000 7f1bb8d97700 1 --2- 172.21.15.189:0/1707926918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb4062930 0x7f1bb414a600 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f1ba4006fd0 tx=0x7f1ba4015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:09.908 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.890+0000 7f1bb0ff9700 1 -- 172.21.15.189:0/1707926918 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4011d50 con 0x7f1bb4062930 2020-11-20T12:05:09.908 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.890+0000 7f1bb9d99700 1 -- 172.21.15.189:0/1707926918 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bb4149fe0 con 0x7f1bb4062930 2020-11-20T12:05:09.909 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.890+0000 7f1bb0ff9700 1 -- 172.21.15.189:0/1707926918 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba401e460 con 0x7f1bb4062930 2020-11-20T12:05:09.909 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.890+0000 7f1bb0ff9700 1 -- 172.21.15.189:0/1707926918 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba401e7e0 con 0x7f1bb4062930 2020-11-20T12:05:09.909 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.890+0000 7f1bb9d99700 1 -- 172.21.15.189:0/1707926918 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb413c200 con 0x7f1bb4062930 2020-11-20T12:05:09.909 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.890+0000 7f1bb0ff9700 1 -- 172.21.15.189:0/1707926918 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f1ba4025030 con 0x7f1bb4062930 2020-11-20T12:05:09.909 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.890+0000 7f1bb9d99700 1 -- 172.21.15.189:0/1707926918 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bb40a5330 con 0x7f1bb4062930 2020-11-20T12:05:09.910 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.890+0000 7f1bb0ff9700 1 --2- 172.21.15.189:0/1707926918 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1b8c0557c0 0x7f1b8c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:09.910 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.890+0000 7f1bb0ff9700 1 -- 172.21.15.189:0/1707926918 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1ba406fd30 con 0x7f1bb4062930 2020-11-20T12:05:09.911 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.890+0000 7f1bb3fff700 1 --2- 172.21.15.189:0/1707926918 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1b8c0557c0 0x7f1b8c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:09.911 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.890+0000 7f1bb3fff700 1 --2- 172.21.15.189:0/1707926918 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1b8c0557c0 0x7f1b8c057c80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f1ba800ba00 tx=0x7f1ba8008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:09.911 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:09.898+0000 7f1bb0ff9700 1 -- 172.21.15.189:0/1707926918 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1ba4045d00 con 0x7f1bb4062930 2020-11-20T12:05:10.055 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.046+0000 7f1bb9d99700 1 -- 172.21.15.189:0/1707926918 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f1bb40ab200 con 0x7f1bb4062930 2020-11-20T12:05:10.056 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.050+0000 7f1bb0ff9700 1 -- 172.21.15.189:0/1707926918 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v22) v1 ==== 92+0+10 (secure 0 0 0) 0x7f1ba4023030 con 0x7f1bb4062930 2020-11-20T12:05:10.056 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 8 2020-11-20T12:05:10.057 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.050+0000 7f1bb9d99700 1 -- 172.21.15.189:0/1707926918 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1b8c0557c0 msgr2=0x7f1b8c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.058 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.050+0000 7f1bb9d99700 1 --2- 172.21.15.189:0/1707926918 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1b8c0557c0 0x7f1b8c057c80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f1ba800ba00 tx=0x7f1ba8008040).stop 2020-11-20T12:05:10.058 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.050+0000 7f1bb9d99700 1 -- 172.21.15.189:0/1707926918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb4062930 msgr2=0x7f1bb414a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.058 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.050+0000 7f1bb9d99700 1 --2- 172.21.15.189:0/1707926918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb4062930 0x7f1bb414a600 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f1ba4006fd0 tx=0x7f1ba4015000).stop 2020-11-20T12:05:10.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.050+0000 7f1bb9d99700 1 -- 172.21.15.189:0/1707926918 shutdown_connections 2020-11-20T12:05:10.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.050+0000 7f1bb9d99700 1 --2- 172.21.15.189:0/1707926918 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1bb41373e0 0x7f1bb413bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.050+0000 7f1bb9d99700 1 --2- 172.21.15.189:0/1707926918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb4062930 0x7f1bb414a600 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.050+0000 7f1bb9d99700 1 --2- 172.21.15.189:0/1707926918 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1b8c0557c0 0x7f1b8c057c80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.050+0000 7f1bb9d99700 1 --2- 172.21.15.189:0/1707926918 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1bb4066a30 0x7f1bb4149af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.050+0000 7f1bb9d99700 1 -- 172.21.15.189:0/1707926918 >> 172.21.15.189:0/1707926918 conn(0x7f1bb4065080 msgr2=0x7f1bb4134400 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:10.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.050+0000 7f1bb9d99700 1 -- 172.21.15.189:0/1707926918 shutdown_connections 2020-11-20T12:05:10.061 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.050+0000 7f1bb9d99700 1 -- 172.21.15.189:0/1707926918 wait complete. 2020-11-20T12:05:10.070 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:05:10.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f72c73700 1 Processor -- start 2020-11-20T12:05:10.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f72c73700 1 -- start start 2020-11-20T12:05:10.203 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f72c73700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0f6c060310 0x7f0f6c13a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f72c73700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f6c0a8fc0 con 0x7f0f6c0a0c30 2020-11-20T12:05:10.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f72c73700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f6c0a9260 con 0x7f0f6c066ba0 2020-11-20T12:05:10.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f72c73700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6c0674b0 con 0x7f0f6c060310 2020-11-20T12:05:10.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f71470700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0f6c066ba0 0x7f0f6c0a0520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50510/0 (socket says 172.21.15.189:50510) 2020-11-20T12:05:10.205 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f71470700 1 -- 172.21.15.189:0/2887493007 learned_addr learned my addr 172.21.15.189:0/2887493007 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:10.205 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f71c71700 1 --2- 172.21.15.189:0/2887493007 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0f6c060310 0x7f0f6c13a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.205 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f70c6f700 1 -- 172.21.15.189:0/2887493007 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184642004 0 0) 0x7f0f6c0a9260 con 0x7f0f6c066ba0 2020-11-20T12:05:10.205 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f70c6f700 1 -- 172.21.15.189:0/2887493007 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f54003240 con 0x7f0f6c066ba0 2020-11-20T12:05:10.206 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f70c6f700 1 -- 172.21.15.189:0/2887493007 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3104866298 0 0) 0x7f0f6c0a8fc0 con 0x7f0f6c0a0c30 2020-11-20T12:05:10.206 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f70c6f700 1 -- 172.21.15.189:0/2887493007 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f6c0a9260 con 0x7f0f6c0a0c30 2020-11-20T12:05:10.206 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f70c6f700 1 -- 172.21.15.189:0/2887493007 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 911040248 0 0) 0x7f0f54003240 con 0x7f0f6c066ba0 2020-11-20T12:05:10.206 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f70c6f700 1 -- 172.21.15.189:0/2887493007 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0f6c060310 msgr2=0x7f0f6c13a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:10.206 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f70c6f700 1 --2- 172.21.15.189:0/2887493007 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0f6c060310 0x7f0f6c13a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.207 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f70c6f700 1 -- 172.21.15.189:0/2887493007 >> v1:172.21.15.47:6789/0 conn(0x7f0f6c0a0c30 legacy=0x7f0f6c141c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.207 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f70c6f700 1 -- 172.21.15.189:0/2887493007 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f6c068460 con 0x7f0f6c066ba0 2020-11-20T12:05:10.207 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f70c6f700 1 -- 172.21.15.189:0/2887493007 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f0f5c0014e0 con 0x7f0f6c066ba0 2020-11-20T12:05:10.207 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f70c6f700 1 -- 172.21.15.189:0/2887493007 >> v1:172.21.15.189:6789/0 conn(0x7f0f6c066ba0 legacy=0x7f0f6c0a0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.207 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f70c6f700 1 --2- 172.21.15.189:0/2887493007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f540038c0 0x7f0f54003cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.208 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f70c6f700 1 --2- 172.21.15.189:0/2887493007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f540044e0 0x7f0f54004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.208 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f70c6f700 1 --2- 172.21.15.189:0/2887493007 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0f6c060310 0x7f0f54004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.208 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f70c6f700 1 -- 172.21.15.189:0/2887493007 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6c0674b0 con 0x7f0f540044e0 2020-11-20T12:05:10.208 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f70c6f700 1 -- 172.21.15.189:0/2887493007 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f54005340 con 0x7f0f540038c0 2020-11-20T12:05:10.208 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f70c6f700 1 -- 172.21.15.189:0/2887493007 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5400b730 con 0x7f0f6c060310 2020-11-20T12:05:10.209 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f71c71700 1 --2- 172.21.15.189:0/2887493007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f540038c0 0x7f0f54003cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.209 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f72472700 1 --2- 172.21.15.189:0/2887493007 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0f6c060310 0x7f0f54004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.209 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.194+0000 7f0f71470700 1 --2- 172.21.15.189:0/2887493007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f540044e0 0x7f0f54004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.209 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.198+0000 7f0f71c71700 1 -- 172.21.15.189:0/2887493007 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0f6c060310 msgr2=0x7f0f54004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:10.209 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.198+0000 7f0f71c71700 1 --2- 172.21.15.189:0/2887493007 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0f6c060310 0x7f0f54004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.210 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.198+0000 7f0f71c71700 1 -- 172.21.15.189:0/2887493007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f540044e0 msgr2=0x7f0f54004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:10.210 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.198+0000 7f0f71c71700 1 --2- 172.21.15.189:0/2887493007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f540044e0 0x7f0f54004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.210 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.198+0000 7f0f71c71700 1 -- 172.21.15.189:0/2887493007 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f54003240 con 0x7f0f540038c0 2020-11-20T12:05:10.210 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.198+0000 7f0f71470700 1 --2- 172.21.15.189:0/2887493007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f540044e0 0x7f0f54004960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:10.210 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.198+0000 7f0f71c71700 1 --2- 172.21.15.189:0/2887493007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f540038c0 0x7f0f54003cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f0f6800bc00 tx=0x7f0f68006df0).ready entity=mon.1 client_cookie=800702ae986860cf server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:10.211 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.198+0000 7f0f70c6f700 1 -- 172.21.15.189:0/2887493007 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6800b310 con 0x7f0f540038c0 2020-11-20T12:05:10.211 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.198+0000 7f0f70c6f700 1 -- 172.21.15.189:0/2887493007 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f6800ea90 con 0x7f0f540038c0 2020-11-20T12:05:10.211 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.198+0000 7f0f70c6f700 1 -- 172.21.15.189:0/2887493007 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6800ec00 con 0x7f0f540038c0 2020-11-20T12:05:10.214 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.198+0000 7f0f72c73700 1 -- 172.21.15.189:0/2887493007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f540038c0 msgr2=0x7f0f54003cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.215 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.198+0000 7f0f72c73700 1 --2- 172.21.15.189:0/2887493007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f540038c0 0x7f0f54003cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f0f6800bc00 tx=0x7f0f68006df0).stop 2020-11-20T12:05:10.215 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.198+0000 7f0f72472700 1 -- 172.21.15.189:0/2887493007 reap_dead start 2020-11-20T12:05:10.215 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.198+0000 7f0f72c73700 1 -- 172.21.15.189:0/2887493007 shutdown_connections 2020-11-20T12:05:10.215 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.198+0000 7f0f72c73700 1 -- 172.21.15.189:0/2887493007 >> 172.21.15.189:0/2887493007 conn(0x7f0f6c065910 msgr2=0x7f0f6c060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:10.215 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.198+0000 7f0f72c73700 1 -- 172.21.15.189:0/2887493007 shutdown_connections 2020-11-20T12:05:10.216 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.198+0000 7f0f72c73700 1 -- 172.21.15.189:0/2887493007 wait complete. 2020-11-20T12:05:10.216 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.198+0000 7f0f72c73700 1 Processor -- start 2020-11-20T12:05:10.216 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.198+0000 7f0f72c73700 1 -- start start 2020-11-20T12:05:10.216 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.202+0000 7f0f72c73700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f6c13a5c0 0x7f0f6c0a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.217 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.202+0000 7f0f72c73700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f6c13a9a0 0x7f0f6c0a3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.217 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.202+0000 7f0f72c73700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0f6c149130 0x7f0f6c149590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.217 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.202+0000 7f0f72c73700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6c0a2ee0 con 0x7f0f6c13a9a0 2020-11-20T12:05:10.217 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.202+0000 7f0f72c73700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6c0a27a0 con 0x7f0f6c13a5c0 2020-11-20T12:05:10.217 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.202+0000 7f0f72c73700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6c0a2a40 con 0x7f0f6c149130 2020-11-20T12:05:10.218 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.202+0000 7f0f71c71700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f6c13a5c0 0x7f0f6c0a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.218 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.202+0000 7f0f71c71700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f6c13a5c0 0x7f0f6c0a38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48898/0 (socket says 172.21.15.189:48898) 2020-11-20T12:05:10.218 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.202+0000 7f0f71c71700 1 -- 172.21.15.189:0/1242280192 learned_addr learned my addr 172.21.15.189:0/1242280192 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:10.218 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.202+0000 7f0f71470700 1 --2- 172.21.15.189:0/1242280192 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f6c13a9a0 0x7f0f6c0a3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.218 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.202+0000 7f0f72472700 1 --2- 172.21.15.189:0/1242280192 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0f6c149130 0x7f0f6c149590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.219 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.202+0000 7f0f71c71700 1 -- 172.21.15.189:0/1242280192 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0f6c149130 msgr2=0x7f0f6c149590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.219 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.202+0000 7f0f71c71700 1 --2- 172.21.15.189:0/1242280192 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0f6c149130 0x7f0f6c149590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.219 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.202+0000 7f0f71c71700 1 -- 172.21.15.189:0/1242280192 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f6c13a9a0 msgr2=0x7f0f6c0a3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.219 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.202+0000 7f0f71c71700 1 --2- 172.21.15.189:0/1242280192 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f6c13a9a0 0x7f0f6c0a3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.219 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.202+0000 7f0f71c71700 1 -- 172.21.15.189:0/1242280192 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f6c0a3480 con 0x7f0f6c13a5c0 2020-11-20T12:05:10.220 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.202+0000 7f0f71c71700 1 --2- 172.21.15.189:0/1242280192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f6c13a5c0 0x7f0f6c0a38b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f0f68015000 tx=0x7f0f68019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:10.220 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.202+0000 7f0f59ffb700 1 -- 172.21.15.189:0/1242280192 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6801f070 con 0x7f0f6c13a5c0 2020-11-20T12:05:10.220 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.202+0000 7f0f59ffb700 1 -- 172.21.15.189:0/1242280192 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f6800b310 con 0x7f0f6c13a5c0 2020-11-20T12:05:10.220 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.202+0000 7f0f59ffb700 1 -- 172.21.15.189:0/1242280192 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6801a650 con 0x7f0f6c13a5c0 2020-11-20T12:05:10.220 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.202+0000 7f0f72c73700 1 -- 172.21.15.189:0/1242280192 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f6c0ae160 con 0x7f0f6c13a5c0 2020-11-20T12:05:10.221 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.202+0000 7f0f72c73700 1 -- 172.21.15.189:0/1242280192 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f6c13afc0 con 0x7f0f6c13a5c0 2020-11-20T12:05:10.221 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.202+0000 7f0f72c73700 1 -- 172.21.15.189:0/1242280192 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f6c0a2c00 con 0x7f0f6c13a5c0 2020-11-20T12:05:10.221 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.206+0000 7f0f59ffb700 1 -- 172.21.15.189:0/1242280192 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f0f6801aa00 con 0x7f0f6c13a5c0 2020-11-20T12:05:10.221 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.206+0000 7f0f59ffb700 1 --2- 172.21.15.189:0/1242280192 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0f440557c0 0x7f0f44057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.221 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.206+0000 7f0f71470700 1 --2- 172.21.15.189:0/1242280192 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0f440557c0 0x7f0f44057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.222 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.206+0000 7f0f59ffb700 1 -- 172.21.15.189:0/1242280192 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0f680722d0 con 0x7f0f6c13a5c0 2020-11-20T12:05:10.222 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.206+0000 7f0f71470700 1 --2- 172.21.15.189:0/1242280192 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0f440557c0 0x7f0f44057c80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f0f6c1349c0 tx=0x7f0f6c138370).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:10.223 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.210+0000 7f0f59ffb700 1 -- 172.21.15.189:0/1242280192 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0f6804ba90 con 0x7f0f6c13a5c0 2020-11-20T12:05:10.372 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.366+0000 7f0f72c73700 1 -- 172.21.15.189:0/1242280192 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0f6c0446d0 con 0x7f0f6c13a5c0 2020-11-20T12:05:10.373 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.366+0000 7f0f59ffb700 1 -- 172.21.15.189:0/1242280192 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+10 (secure 0 0 0) 0x7f0f680310a0 con 0x7f0f6c13a5c0 2020-11-20T12:05:10.373 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 8 2020-11-20T12:05:10.375 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.370+0000 7f0f72c73700 1 -- 172.21.15.189:0/1242280192 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0f440557c0 msgr2=0x7f0f44057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.376 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.370+0000 7f0f72c73700 1 --2- 172.21.15.189:0/1242280192 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0f440557c0 0x7f0f44057c80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f0f6c1349c0 tx=0x7f0f6c138370).stop 2020-11-20T12:05:10.376 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.370+0000 7f0f72c73700 1 -- 172.21.15.189:0/1242280192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f6c13a5c0 msgr2=0x7f0f6c0a38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.376 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.370+0000 7f0f72c73700 1 --2- 172.21.15.189:0/1242280192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f6c13a5c0 0x7f0f6c0a38b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f0f68015000 tx=0x7f0f68019000).stop 2020-11-20T12:05:10.376 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.370+0000 7f0f72c73700 1 -- 172.21.15.189:0/1242280192 shutdown_connections 2020-11-20T12:05:10.377 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.370+0000 7f0f72c73700 1 --2- 172.21.15.189:0/1242280192 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0f6c149130 0x7f0f6c149590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.377 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.370+0000 7f0f72c73700 1 --2- 172.21.15.189:0/1242280192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f6c13a5c0 0x7f0f6c0a38b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.377 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.370+0000 7f0f72c73700 1 --2- 172.21.15.189:0/1242280192 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0f440557c0 0x7f0f44057c80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.377 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.370+0000 7f0f72c73700 1 --2- 172.21.15.189:0/1242280192 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0f6c13a9a0 0x7f0f6c0a3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.377 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.370+0000 7f0f72c73700 1 -- 172.21.15.189:0/1242280192 >> 172.21.15.189:0/1242280192 conn(0x7f0f6c065910 msgr2=0x7f0f6c0a9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:10.378 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.370+0000 7f0f72c73700 1 -- 172.21.15.189:0/1242280192 shutdown_connections 2020-11-20T12:05:10.378 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.370+0000 7f0f72c73700 1 -- 172.21.15.189:0/1242280192 wait complete. 2020-11-20T12:05:10.393 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:05:10.529 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.518+0000 7f12e324a700 1 Processor -- start 2020-11-20T12:05:10.529 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.518+0000 7f12e324a700 1 -- start start 2020-11-20T12:05:10.530 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.518+0000 7f12e324a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f12dc0612b0 0x7f12dc0a0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.530 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.518+0000 7f12e324a700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f12dc1340b0 0x7f12dc13a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.530 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.518+0000 7f12e324a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12dc134500 con 0x7f12dc13a090 2020-11-20T12:05:10.530 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.518+0000 7f12e324a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12dc067b80 con 0x7f12dc0612b0 2020-11-20T12:05:10.531 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.518+0000 7f12e324a700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12dc067e20 con 0x7f12dc1340b0 2020-11-20T12:05:10.531 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.518+0000 7f12e2248700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f12dc0612b0 0x7f12dc0a0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.531 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.518+0000 7f12e2248700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f12dc0612b0 0x7f12dc0a0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48906/0 (socket says 172.21.15.189:48906) 2020-11-20T12:05:10.531 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.518+0000 7f12e2248700 1 -- 172.21.15.189:0/1274760686 learned_addr learned my addr 172.21.15.189:0/1274760686 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:10.531 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.518+0000 7f12e1a47700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f12dc13a090 0x7f12dc1339a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50530/0 (socket says 172.21.15.189:50530) 2020-11-20T12:05:10.532 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.518+0000 7f12e2a49700 1 --2- 172.21.15.189:0/1274760686 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f12dc1340b0 0x7f12dc13a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.532 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.518+0000 7f12e2248700 1 -- 172.21.15.189:0/1274760686 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f12dc1340b0 msgr2=0x7f12dc13a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:10.532 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.518+0000 7f12e2248700 1 --2- 172.21.15.189:0/1274760686 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f12dc1340b0 0x7f12dc13a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.532 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.518+0000 7f12e2248700 1 -- 172.21.15.189:0/1274760686 >> v1:172.21.15.189:6789/0 conn(0x7f12dc13a090 legacy=0x7f12dc1339a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.532 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.518+0000 7f12e1246700 1 -- 172.21.15.189:0/1274760686 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 422998494 0 0) 0x7f12dc134500 con 0x7f12dc13a090 2020-11-20T12:05:10.533 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.518+0000 7f12e2248700 1 -- 172.21.15.189:0/1274760686 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12dc0680c0 con 0x7f12dc0612b0 2020-11-20T12:05:10.533 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.518+0000 7f12e2248700 1 --2- 172.21.15.189:0/1274760686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12dc0612b0 0x7f12dc0a0730 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f12d8006fd0 tx=0x7f12d8006eb0).ready entity=mon.1 client_cookie=de753440c7a1d277 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:10.533 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.518+0000 7f12e1246700 1 -- 172.21.15.189:0/1274760686 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d8009810 con 0x7f12dc0612b0 2020-11-20T12:05:10.533 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.518+0000 7f12e1246700 1 -- 172.21.15.189:0/1274760686 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12d8009970 con 0x7f12dc0612b0 2020-11-20T12:05:10.534 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.518+0000 7f12e1246700 1 -- 172.21.15.189:0/1274760686 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d8009cf0 con 0x7f12dc0612b0 2020-11-20T12:05:10.534 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.518+0000 7f12e324a700 1 -- 172.21.15.189:0/1274760686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12dc0612b0 msgr2=0x7f12dc0a0730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.534 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.518+0000 7f12e324a700 1 --2- 172.21.15.189:0/1274760686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12dc0612b0 0x7f12dc0a0730 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f12d8006fd0 tx=0x7f12d8006eb0).stop 2020-11-20T12:05:10.534 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.522+0000 7f12e324a700 1 -- 172.21.15.189:0/1274760686 shutdown_connections 2020-11-20T12:05:10.534 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.522+0000 7f12e324a700 1 --2- 172.21.15.189:0/1274760686 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f12dc1340b0 0x7f12dc13a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.535 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.522+0000 7f12e324a700 1 --2- 172.21.15.189:0/1274760686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12dc0612b0 0x7f12dc0a0730 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.535 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.522+0000 7f12e324a700 1 -- 172.21.15.189:0/1274760686 >> 172.21.15.189:0/1274760686 conn(0x7f12dc065910 msgr2=0x7f12dc060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:10.535 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.522+0000 7f12e324a700 1 -- 172.21.15.189:0/1274760686 shutdown_connections 2020-11-20T12:05:10.535 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.522+0000 7f12e324a700 1 -- 172.21.15.189:0/1274760686 wait complete. 2020-11-20T12:05:10.535 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.522+0000 7f12e324a700 1 Processor -- start 2020-11-20T12:05:10.536 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.522+0000 7f12e324a700 1 -- start start 2020-11-20T12:05:10.538 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.522+0000 7f12e324a700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f12dc0612b0 0x7f12dc0a5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.539 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.522+0000 7f12e324a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12dc1340b0 0x7f12dc0a1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.539 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.522+0000 7f12e324a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12dc13a090 0x7f12dc147780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.539 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.522+0000 7f12e324a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12dc0a64a0 con 0x7f12dc1340b0 2020-11-20T12:05:10.539 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.522+0000 7f12e324a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12dc0a6700 con 0x7f12dc13a090 2020-11-20T12:05:10.539 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.522+0000 7f12e324a700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12dc0a69a0 con 0x7f12dc0612b0 2020-11-20T12:05:10.540 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.522+0000 7f12e2a49700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12dc13a090 0x7f12dc147780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.540 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.522+0000 7f12e2a49700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12dc13a090 0x7f12dc147780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48916/0 (socket says 172.21.15.189:48916) 2020-11-20T12:05:10.540 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.522+0000 7f12e2a49700 1 -- 172.21.15.189:0/2337675485 learned_addr learned my addr 172.21.15.189:0/2337675485 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:10.540 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.522+0000 7f12e2248700 1 --2- 172.21.15.189:0/2337675485 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f12dc0612b0 0x7f12dc0a5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.541 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.522+0000 7f12e1a47700 1 --2- 172.21.15.189:0/2337675485 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12dc1340b0 0x7f12dc0a1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.541 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.522+0000 7f12e2a49700 1 -- 172.21.15.189:0/2337675485 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f12dc0612b0 msgr2=0x7f12dc0a5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.541 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.522+0000 7f12e2a49700 1 --2- 172.21.15.189:0/2337675485 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f12dc0612b0 0x7f12dc0a5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.541 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.522+0000 7f12e2a49700 1 -- 172.21.15.189:0/2337675485 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12dc1340b0 msgr2=0x7f12dc0a1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.542 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.522+0000 7f12e2a49700 1 --2- 172.21.15.189:0/2337675485 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12dc1340b0 0x7f12dc0a1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.542 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.522+0000 7f12e2a49700 1 -- 172.21.15.189:0/2337675485 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12dc0a08a0 con 0x7f12dc13a090 2020-11-20T12:05:10.542 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.526+0000 7f12e2a49700 1 --2- 172.21.15.189:0/2337675485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12dc13a090 0x7f12dc147780 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f12d400ef50 tx=0x7f12d40091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:10.542 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.526+0000 7f12ca7fc700 1 -- 172.21.15.189:0/2337675485 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d400ccc0 con 0x7f12dc13a090 2020-11-20T12:05:10.542 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.526+0000 7f12ca7fc700 1 -- 172.21.15.189:0/2337675485 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12d400ce20 con 0x7f12dc13a090 2020-11-20T12:05:10.543 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.526+0000 7f12ca7fc700 1 -- 172.21.15.189:0/2337675485 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d4012620 con 0x7f12dc13a090 2020-11-20T12:05:10.543 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.526+0000 7f12e324a700 1 -- 172.21.15.189:0/2337675485 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12dc0a6e40 con 0x7f12dc13a090 2020-11-20T12:05:10.543 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.526+0000 7f12e324a700 1 -- 172.21.15.189:0/2337675485 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12dc0a47c0 con 0x7f12dc13a090 2020-11-20T12:05:10.543 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.526+0000 7f12e324a700 1 -- 172.21.15.189:0/2337675485 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12dc0a6b60 con 0x7f12dc13a090 2020-11-20T12:05:10.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.526+0000 7f12ca7fc700 1 -- 172.21.15.189:0/2337675485 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f12d4022070 con 0x7f12dc13a090 2020-11-20T12:05:10.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.526+0000 7f12ca7fc700 1 --2- 172.21.15.189:0/2337675485 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f12b40557f0 0x7f12b4057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.526+0000 7f12ca7fc700 1 -- 172.21.15.189:0/2337675485 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f12d406ef10 con 0x7f12dc13a090 2020-11-20T12:05:10.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.526+0000 7f12e2248700 1 --2- 172.21.15.189:0/2337675485 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f12b40557f0 0x7f12b4057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.545 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.530+0000 7f12e2248700 1 --2- 172.21.15.189:0/2337675485 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f12b40557f0 0x7f12b4057cb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f12d800b1b0 tx=0x7f12d8015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:10.545 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.534+0000 7f12ca7fc700 1 -- 172.21.15.189:0/2337675485 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f12d4044db0 con 0x7f12dc13a090 2020-11-20T12:05:10.691 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.682+0000 7f12e324a700 1 -- 172.21.15.189:0/2337675485 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f12dc0446d0 con 0x7f12dc13a090 2020-11-20T12:05:10.692 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.686+0000 7f12ca7fc700 1 -- 172.21.15.189:0/2337675485 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+10 (secure 0 0 0) 0x7f12d4016070 con 0x7f12dc13a090 2020-11-20T12:05:10.692 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 8 2020-11-20T12:05:10.694 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.686+0000 7f12e324a700 1 -- 172.21.15.189:0/2337675485 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f12b40557f0 msgr2=0x7f12b4057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.694 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.686+0000 7f12e324a700 1 --2- 172.21.15.189:0/2337675485 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f12b40557f0 0x7f12b4057cb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f12d800b1b0 tx=0x7f12d8015000).stop 2020-11-20T12:05:10.694 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.686+0000 7f12e324a700 1 -- 172.21.15.189:0/2337675485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12dc13a090 msgr2=0x7f12dc147780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.694 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.686+0000 7f12e324a700 1 --2- 172.21.15.189:0/2337675485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12dc13a090 0x7f12dc147780 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f12d400ef50 tx=0x7f12d40091d0).stop 2020-11-20T12:05:10.695 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.686+0000 7f12e324a700 1 -- 172.21.15.189:0/2337675485 shutdown_connections 2020-11-20T12:05:10.695 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.686+0000 7f12e324a700 1 --2- 172.21.15.189:0/2337675485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12dc13a090 0x7f12dc147780 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.695 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.686+0000 7f12e324a700 1 --2- 172.21.15.189:0/2337675485 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f12dc0612b0 0x7f12dc0a5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.695 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.686+0000 7f12e324a700 1 --2- 172.21.15.189:0/2337675485 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f12b40557f0 0x7f12b4057cb0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.695 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.686+0000 7f12e324a700 1 --2- 172.21.15.189:0/2337675485 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f12dc1340b0 0x7f12dc0a1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.696 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.686+0000 7f12e324a700 1 -- 172.21.15.189:0/2337675485 >> 172.21.15.189:0/2337675485 conn(0x7f12dc065910 msgr2=0x7f12dc135210 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:10.696 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.686+0000 7f12e324a700 1 -- 172.21.15.189:0/2337675485 shutdown_connections 2020-11-20T12:05:10.696 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.686+0000 7f12e324a700 1 -- 172.21.15.189:0/2337675485 wait complete. 2020-11-20T12:05:10.706 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2020-11-20T12:05:10.707 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:05:10.832 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.826+0000 7f5cf8888700 1 Processor -- start 2020-11-20T12:05:10.832 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.826+0000 7f5cf8888700 1 -- start start 2020-11-20T12:05:10.834 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.826+0000 7f5cf8888700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5cf4060310 0x7f5cf40a9200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.834 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.826+0000 7f5cf8888700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f5cf413c370 0x7f5cf4149440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.834 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.826+0000 7f5cf8888700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cf40a0eb0 con 0x7f5cf40a8b00 2020-11-20T12:05:10.834 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.826+0000 7f5cf8888700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cf40673b0 con 0x7f5cf413c370 2020-11-20T12:05:10.835 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.826+0000 7f5cf8888700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cf40675a0 con 0x7f5cf4060310 2020-11-20T12:05:10.835 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.826+0000 7f5cf37fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5cf4060310 0x7f5cf40a9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.835 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.826+0000 7f5cf37fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5cf4060310 0x7f5cf40a9200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48924/0 (socket says 172.21.15.189:48924) 2020-11-20T12:05:10.835 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.826+0000 7f5cf37fe700 1 -- 172.21.15.189:0/2113992945 learned_addr learned my addr 172.21.15.189:0/2113992945 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:10.836 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.826+0000 7f5cf3fff700 1 --2- 172.21.15.189:0/2113992945 >> v2:172.21.15.47:3300/0 conn(0x7f5cf413c370 0x7f5cf4149440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.836 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.826+0000 7f5cf37fe700 1 -- 172.21.15.189:0/2113992945 >> v2:172.21.15.47:3300/0 conn(0x7f5cf413c370 msgr2=0x7f5cf4149440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:10.836 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.826+0000 7f5cf37fe700 1 --2- 172.21.15.189:0/2113992945 >> v2:172.21.15.47:3300/0 conn(0x7f5cf413c370 0x7f5cf4149440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.836 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.826+0000 7f5cf37fe700 1 -- 172.21.15.189:0/2113992945 >> v1:172.21.15.47:6789/0 conn(0x7f5cf40a8b00 legacy=0x7f5cf413bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.836 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.826+0000 7f5cf37fe700 1 -- 172.21.15.189:0/2113992945 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf4067840 con 0x7f5cf4060310 2020-11-20T12:05:10.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.826+0000 7f5cf37fe700 1 --2- 172.21.15.189:0/2113992945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cf4060310 0x7f5cf40a9200 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f5cf40ae330 tx=0x7f5ce4006eb0).ready entity=mon.1 client_cookie=1acdd272122f8819 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:10.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.826+0000 7f5cf27fc700 1 -- 172.21.15.189:0/2113992945 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce4009810 con 0x7f5cf4060310 2020-11-20T12:05:10.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.826+0000 7f5cf27fc700 1 -- 172.21.15.189:0/2113992945 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce4009970 con 0x7f5cf4060310 2020-11-20T12:05:10.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.826+0000 7f5cf27fc700 1 -- 172.21.15.189:0/2113992945 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce4009cf0 con 0x7f5cf4060310 2020-11-20T12:05:10.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.830+0000 7f5cf8888700 1 -- 172.21.15.189:0/2113992945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cf4060310 msgr2=0x7f5cf40a9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.830+0000 7f5cf8888700 1 --2- 172.21.15.189:0/2113992945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cf4060310 0x7f5cf40a9200 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f5cf40ae330 tx=0x7f5ce4006eb0).stop 2020-11-20T12:05:10.839 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.830+0000 7f5cf8888700 1 -- 172.21.15.189:0/2113992945 shutdown_connections 2020-11-20T12:05:10.839 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.830+0000 7f5cf8888700 1 --2- 172.21.15.189:0/2113992945 >> v2:172.21.15.47:3300/0 conn(0x7f5cf413c370 0x7f5cf4149440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.839 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.830+0000 7f5cf8888700 1 --2- 172.21.15.189:0/2113992945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cf4060310 0x7f5cf40a9200 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.839 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.830+0000 7f5cf8888700 1 -- 172.21.15.189:0/2113992945 >> 172.21.15.189:0/2113992945 conn(0x7f5cf4063210 msgr2=0x7f5cf4060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:10.840 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.830+0000 7f5cf8888700 1 -- 172.21.15.189:0/2113992945 shutdown_connections 2020-11-20T12:05:10.840 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.830+0000 7f5cf8888700 1 -- 172.21.15.189:0/2113992945 wait complete. 2020-11-20T12:05:10.840 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.830+0000 7f5cf8888700 1 Processor -- start 2020-11-20T12:05:10.840 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.830+0000 7f5cf8888700 1 -- start start 2020-11-20T12:05:10.842 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.834+0000 7f5cf8888700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5cf4060310 0x7f5cf40a7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.843 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.834+0000 7f5cf8888700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cf40a8b00 0x7f5cf4138ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.843 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.834+0000 7f5cf8888700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5cf413c370 0x7f5cf40a60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.843 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.834+0000 7f5cf8888700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf40a53d0 con 0x7f5cf4060310 2020-11-20T12:05:10.843 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.834+0000 7f5cf8888700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf40a65c0 con 0x7f5cf40a8b00 2020-11-20T12:05:10.844 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.834+0000 7f5cf8888700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf40a6be0 con 0x7f5cf413c370 2020-11-20T12:05:10.844 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.834+0000 7f5cf2ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cf40a8b00 0x7f5cf4138ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.844 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.834+0000 7f5cf2ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cf40a8b00 0x7f5cf4138ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48928/0 (socket says 172.21.15.189:48928) 2020-11-20T12:05:10.844 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.834+0000 7f5cf2ffd700 1 -- 172.21.15.189:0/3774684553 learned_addr learned my addr 172.21.15.189:0/3774684553 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:10.844 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.834+0000 7f5cf3fff700 1 --2- 172.21.15.189:0/3774684553 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5cf413c370 0x7f5cf40a60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.834+0000 7f5cf37fe700 1 --2- 172.21.15.189:0/3774684553 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5cf4060310 0x7f5cf40a7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.834+0000 7f5cf2ffd700 1 -- 172.21.15.189:0/3774684553 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5cf413c370 msgr2=0x7f5cf40a60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.834+0000 7f5cf2ffd700 1 --2- 172.21.15.189:0/3774684553 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5cf413c370 0x7f5cf40a60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.834+0000 7f5cf2ffd700 1 -- 172.21.15.189:0/3774684553 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5cf4060310 msgr2=0x7f5cf40a7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:10.846 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.834+0000 7f5cf2ffd700 1 --2- 172.21.15.189:0/3774684553 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5cf4060310 0x7f5cf40a7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:10.846 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.834+0000 7f5cf2ffd700 1 -- 172.21.15.189:0/3774684553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf4141ce0 con 0x7f5cf40a8b00 2020-11-20T12:05:10.846 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.834+0000 7f5cf37fe700 1 --2- 172.21.15.189:0/3774684553 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5cf4060310 0x7f5cf40a7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:10.846 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.834+0000 7f5cf2ffd700 1 --2- 172.21.15.189:0/3774684553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cf40a8b00 0x7f5cf4138ad0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f5cdc00ca90 tx=0x7f5cdc007f90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:10.846 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.834+0000 7f5ceb7fe700 1 -- 172.21.15.189:0/3774684553 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc00a730 con 0x7f5cf40a8b00 2020-11-20T12:05:10.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.834+0000 7f5ceb7fe700 1 -- 172.21.15.189:0/3774684553 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cdc00a890 con 0x7f5cf40a8b00 2020-11-20T12:05:10.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.834+0000 7f5cf8888700 1 -- 172.21.15.189:0/3774684553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cf4142da0 con 0x7f5cf40a8b00 2020-11-20T12:05:10.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.834+0000 7f5ceb7fe700 1 -- 172.21.15.189:0/3774684553 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc00ac30 con 0x7f5cf40a8b00 2020-11-20T12:05:10.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.834+0000 7f5cf8888700 1 -- 172.21.15.189:0/3774684553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cf40a4ae0 con 0x7f5cf40a8b00 2020-11-20T12:05:10.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.834+0000 7f5cf8888700 1 -- 172.21.15.189:0/3774684553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cf40446d0 con 0x7f5cf40a8b00 2020-11-20T12:05:10.848 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.838+0000 7f5ceb7fe700 1 -- 172.21.15.189:0/3774684553 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f5cdc019030 con 0x7f5cf40a8b00 2020-11-20T12:05:10.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.838+0000 7f5ceb7fe700 1 --2- 172.21.15.189:0/3774684553 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5cc40557c0 0x7f5cc4057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:10.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.838+0000 7f5ceb7fe700 1 -- 172.21.15.189:0/3774684553 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5cdc06dfe0 con 0x7f5cf40a8b00 2020-11-20T12:05:10.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.838+0000 7f5cf37fe700 1 --2- 172.21.15.189:0/3774684553 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5cc40557c0 0x7f5cc4057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:10.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.838+0000 7f5cf37fe700 1 --2- 172.21.15.189:0/3774684553 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5cc40557c0 0x7f5cc4057c80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f5ce4006ef0 tx=0x7f5ce4015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:10.850 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:10.842+0000 7f5ceb7fe700 1 -- 172.21.15.189:0/3774684553 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5cdc043e80 con 0x7f5cf40a8b00 2020-11-20T12:05:11.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:11.042+0000 7f5cf8888700 1 -- 172.21.15.189:0/3774684553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5cf405cdd0 con 0x7f5cf40a8b00 2020-11-20T12:05:11.093 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:11.042+0000 7f5ceb7fe700 1 -- 172.21.15.189:0/3774684553 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2134 (secure 0 0 0) 0x7f5cdc017070 con 0x7f5cf40a8b00 2020-11-20T12:05:11.093 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:11.046+0000 7f5cf8888700 1 -- 172.21.15.189:0/3774684553 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5cc40557c0 msgr2=0x7f5cc4057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:11.093 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:11.046+0000 7f5cf8888700 1 --2- 172.21.15.189:0/3774684553 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5cc40557c0 0x7f5cc4057c80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f5ce4006ef0 tx=0x7f5ce4015000).stop 2020-11-20T12:05:11.094 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:11.046+0000 7f5cf8888700 1 -- 172.21.15.189:0/3774684553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cf40a8b00 msgr2=0x7f5cf4138ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:11.094 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:11.046+0000 7f5cf8888700 1 --2- 172.21.15.189:0/3774684553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cf40a8b00 0x7f5cf4138ad0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f5cdc00ca90 tx=0x7f5cdc007f90).stop 2020-11-20T12:05:11.094 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:11.046+0000 7f5cf8888700 1 -- 172.21.15.189:0/3774684553 shutdown_connections 2020-11-20T12:05:11.094 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:11.046+0000 7f5cf8888700 1 --2- 172.21.15.189:0/3774684553 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5cc40557c0 0x7f5cc4057c80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:11.094 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:11.046+0000 7f5cf8888700 1 --2- 172.21.15.189:0/3774684553 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5cf4060310 0x7f5cf40a7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:11.095 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:11.046+0000 7f5cf8888700 1 --2- 172.21.15.189:0/3774684553 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5cf413c370 0x7f5cf40a60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:11.095 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:11.046+0000 7f5cf8888700 1 --2- 172.21.15.189:0/3774684553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cf40a8b00 0x7f5cf4138ad0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:11.095 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:11.046+0000 7f5cf8888700 1 -- 172.21.15.189:0/3774684553 >> 172.21.15.189:0/3774684553 conn(0x7f5cf4063210 msgr2=0x7f5cf40a7960 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:11.095 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:11.046+0000 7f5cf8888700 1 -- 172.21.15.189:0/3774684553 shutdown_connections 2020-11-20T12:05:11.095 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:11.046+0000 7f5cf8888700 1 -- 172.21.15.189:0/3774684553 wait complete. 2020-11-20T12:05:11.096 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 6 2020-11-20T12:05:11.096 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:05:11.096 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2660417436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2660417436},{"type":"v1","addr":"172.21.15.47:6833","nonce":2660417436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:04:56.025055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.189:6835/1042140684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1042140684},{"type":"v1","addr":"172.21.15.189:6835","nonce":1042140684}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:05:11.298 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T12:04:54.012110+0000', 'modified': '2020-11-20T12:04:56.025055+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4273}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4273': {'gid': 4273, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.189:6835/1042140684', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6834', 'nonce': 1042140684}, {'type': 'v1', 'addr': '172.21.15.189:6835', 'nonce': 1042140684}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T12:05:11.298 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T12:05:11.299 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T12:05:11.444 INFO:teuthology.orchestra.run.smithi189.stdout:{"cluster_fsid":"cda18d0e-64b3-41bb-964a-084069c3e648","whoami":0,"id":4273,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":17.414812820000002,"mdsmap_epoch":6,"osdmap_epoch":22,"osdmap_epoch_barrier":22,"uptime":20.828716315000001} 2020-11-20T12:05:11.445 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "cda18d0e-64b3-41bb-964a-084069c3e648", "fs_name": "cephfs", "id": 4273, "mdsmap_epoch": 6, "osdmap_epoch": 22, "osdmap_epoch_barrier": 22, "rank_uptime": 17.41481282, "state": "up:active", "uptime": 20.828716315, "want_state": "up:active", "whoami": 0 } 2020-11-20T12:05:11.446 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2020-11-20T12:05:11.446 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2020-11-20T12:05:11.446 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:05:11.447 INFO:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p -v /home/ubuntu/cephtest/mnt.0) 2020-11-20T12:05:11.453 INFO:teuthology.orchestra.run.smithi189.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2020-11-20T12:05:11.454 INFO:teuthology.orchestra.run.smithi189:> sudo modprobe fuse 2020-11-20T12:05:11.515 INFO:teuthology.orchestra.run:Running command with timeout 30 2020-11-20T12:05:11.515 INFO:teuthology.orchestra.run.smithi189:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2020-11-20T12:05:11.572 INFO:teuthology.orchestra.run.smithi189.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2020-11-20T12:05:11.573 DEBUG:teuthology.orchestra.run:got remote process result: 32 2020-11-20T12:05:11.574 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:05:11.574 INFO:teuthology.orchestra.run.smithi189:> ls /sys/fs/fuse/connections 2020-11-20T12:05:11.621 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2020-11-20T12:05:11.621 INFO:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --id 0 /home/ubuntu/cephtest/mnt.0) 2020-11-20T12:05:11.664 INFO:teuthology.orchestra.run.smithi189:> sudo modprobe fuse 2020-11-20T12:05:11.679 INFO:teuthology.orchestra.run:Running command with timeout 30 2020-11-20T12:05:11.679 INFO:teuthology.orchestra.run.smithi189:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2020-11-20T12:05:11.738 INFO:teuthology.orchestra.run.smithi189.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2020-11-20T12:05:11.739 DEBUG:teuthology.orchestra.run:got remote process result: 32 2020-11-20T12:05:11.739 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:05:11.740 INFO:teuthology.orchestra.run.smithi189:> ls /sys/fs/fuse/connections 2020-11-20T12:05:11.744 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi189.stderr:2020-11-20T12:05:11.738+0000 7f9644977100 -1 init, newargv = 0x55bd9eba4660 newargc=15 2020-11-20T12:05:11.745 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi189.stderr:ceph-fuse[13958]: starting ceph client 2020-11-20T12:05:11.753 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi189.stderr:ceph-fuse[13958]: starting fuse 2020-11-20T12:05:12.851 INFO:teuthology.orchestra.run.smithi189:> sudo modprobe fuse 2020-11-20T12:05:12.868 INFO:teuthology.orchestra.run:Running command with timeout 30 2020-11-20T12:05:12.868 INFO:teuthology.orchestra.run.smithi189:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2020-11-20T12:05:12.925 INFO:teuthology.orchestra.run.smithi189.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2020-11-20T12:05:12.926 DEBUG:teuthology.orchestra.run:got remote process result: 32 2020-11-20T12:05:12.927 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:05:12.927 INFO:teuthology.orchestra.run.smithi189:> ls /sys/fs/fuse/connections 2020-11-20T12:05:12.976 INFO:teuthology.orchestra.run.smithi189.stdout:49 2020-11-20T12:05:12.977 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [49] 2020-11-20T12:05:12.977 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2020-11-20T12:05:12.977 INFO:teuthology.orchestra.run.smithi189:> import glob 2020-11-20T12:05:12.978 INFO:teuthology.orchestra.run.smithi189:> import re 2020-11-20T12:05:12.978 INFO:teuthology.orchestra.run.smithi189:> import os 2020-11-20T12:05:12.978 INFO:teuthology.orchestra.run.smithi189:> import subprocess 2020-11-20T12:05:12.979 INFO:teuthology.orchestra.run.smithi189:> 2020-11-20T12:05:12.979 INFO:teuthology.orchestra.run.smithi189:> def _find_admin_socket(client_name): 2020-11-20T12:05:12.979 INFO:teuthology.orchestra.run.smithi189:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2020-11-20T12:05:12.980 INFO:teuthology.orchestra.run.smithi189:> files = glob.glob(asok_path) 2020-11-20T12:05:12.980 INFO:teuthology.orchestra.run.smithi189:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2020-11-20T12:05:12.980 INFO:teuthology.orchestra.run.smithi189:> 2020-11-20T12:05:12.981 INFO:teuthology.orchestra.run.smithi189:> # Given a non-glob path, it better be there 2020-11-20T12:05:12.981 INFO:teuthology.orchestra.run.smithi189:> if "*" not in asok_path: 2020-11-20T12:05:12.981 INFO:teuthology.orchestra.run.smithi189:> assert(len(files) == 1) 2020-11-20T12:05:12.982 INFO:teuthology.orchestra.run.smithi189:> return files[0] 2020-11-20T12:05:12.982 INFO:teuthology.orchestra.run.smithi189:> 2020-11-20T12:05:12.982 INFO:teuthology.orchestra.run.smithi189:> for f in files: 2020-11-20T12:05:12.982 INFO:teuthology.orchestra.run.smithi189:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2020-11-20T12:05:12.983 INFO:teuthology.orchestra.run.smithi189:> if os.path.exists("/proc/{0}".format(pid)): 2020-11-20T12:05:12.983 INFO:teuthology.orchestra.run.smithi189:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2020-11-20T12:05:12.983 INFO:teuthology.orchestra.run.smithi189:> contents = proc_f.read() 2020-11-20T12:05:12.984 INFO:teuthology.orchestra.run.smithi189:> if mountpoint in contents: 2020-11-20T12:05:12.984 INFO:teuthology.orchestra.run.smithi189:> return f 2020-11-20T12:05:12.984 INFO:teuthology.orchestra.run.smithi189:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2020-11-20T12:05:12.985 INFO:teuthology.orchestra.run.smithi189:> 2020-11-20T12:05:12.985 INFO:teuthology.orchestra.run.smithi189:> print(_find_admin_socket("client.0")) 2020-11-20T12:05:12.985 INFO:teuthology.orchestra.run.smithi189:> ' 2020-11-20T12:05:13.113 INFO:teuthology.orchestra.run.smithi189.stdout:/var/run/ceph/ceph-client.0.13958.asok 2020-11-20T12:05:13.292 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.13958.asok 2020-11-20T12:05:13.293 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:05:13.293 INFO:teuthology.orchestra.run.smithi189:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.13958.asok status 2020-11-20T12:05:13.455 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2020-11-20T12:05:13.455 INFO:teuthology.orchestra.run.smithi189.stdout: "metadata": { 2020-11-20T12:05:13.456 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_sha1": "b86e691bf191566922a8f08e2a5abf64fde8cfcb", 2020-11-20T12:05:13.456 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_version": "ceph version 16.0.0-7397-gb86e691b (b86e691bf191566922a8f08e2a5abf64fde8cfcb) pacific (dev)", 2020-11-20T12:05:13.456 INFO:teuthology.orchestra.run.smithi189.stdout: "entity_id": "0", 2020-11-20T12:05:13.456 INFO:teuthology.orchestra.run.smithi189.stdout: "hostname": "smithi189", 2020-11-20T12:05:13.457 INFO:teuthology.orchestra.run.smithi189.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2020-11-20T12:05:13.457 INFO:teuthology.orchestra.run.smithi189.stdout: "pid": "13958", 2020-11-20T12:05:13.457 INFO:teuthology.orchestra.run.smithi189.stdout: "root": "/" 2020-11-20T12:05:13.457 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2020-11-20T12:05:13.457 INFO:teuthology.orchestra.run.smithi189.stdout: "dentry_count": 0, 2020-11-20T12:05:13.458 INFO:teuthology.orchestra.run.smithi189.stdout: "dentry_pinned_count": 0, 2020-11-20T12:05:13.458 INFO:teuthology.orchestra.run.smithi189.stdout: "id": 4651, 2020-11-20T12:05:13.458 INFO:teuthology.orchestra.run.smithi189.stdout: "inst": { 2020-11-20T12:05:13.458 INFO:teuthology.orchestra.run.smithi189.stdout: "name": { 2020-11-20T12:05:13.458 INFO:teuthology.orchestra.run.smithi189.stdout: "type": "client", 2020-11-20T12:05:13.459 INFO:teuthology.orchestra.run.smithi189.stdout: "num": 4651 2020-11-20T12:05:13.459 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2020-11-20T12:05:13.459 INFO:teuthology.orchestra.run.smithi189.stdout: "addr": { 2020-11-20T12:05:13.459 INFO:teuthology.orchestra.run.smithi189.stdout: "type": "v1", 2020-11-20T12:05:13.459 INFO:teuthology.orchestra.run.smithi189.stdout: "addr": "192.168.0.1:0", 2020-11-20T12:05:13.460 INFO:teuthology.orchestra.run.smithi189.stdout: "nonce": 3466502623 2020-11-20T12:05:13.460 INFO:teuthology.orchestra.run.smithi189.stdout: } 2020-11-20T12:05:13.460 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2020-11-20T12:05:13.460 INFO:teuthology.orchestra.run.smithi189.stdout: "addr": { 2020-11-20T12:05:13.461 INFO:teuthology.orchestra.run.smithi189.stdout: "type": "v1", 2020-11-20T12:05:13.461 INFO:teuthology.orchestra.run.smithi189.stdout: "addr": "192.168.0.1:0", 2020-11-20T12:05:13.461 INFO:teuthology.orchestra.run.smithi189.stdout: "nonce": 3466502623 2020-11-20T12:05:13.461 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2020-11-20T12:05:13.461 INFO:teuthology.orchestra.run.smithi189.stdout: "inst_str": "client.4651 192.168.0.1:0/3466502623", 2020-11-20T12:05:13.462 INFO:teuthology.orchestra.run.smithi189.stdout: "addr_str": "192.168.0.1:0/3466502623", 2020-11-20T12:05:13.462 INFO:teuthology.orchestra.run.smithi189.stdout: "inode_count": 1, 2020-11-20T12:05:13.462 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_epoch": 6, 2020-11-20T12:05:13.462 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_epoch": 22, 2020-11-20T12:05:13.463 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_epoch_barrier": 0, 2020-11-20T12:05:13.463 INFO:teuthology.orchestra.run.smithi189.stdout: "blocklisted": false, 2020-11-20T12:05:13.463 INFO:teuthology.orchestra.run.smithi189.stdout: "fs_name": "cephfs" 2020-11-20T12:05:13.463 INFO:teuthology.orchestra.run.smithi189.stdout:} 2020-11-20T12:05:13.465 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:05:13.466 INFO:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2020-11-20T12:05:13.466 INFO:teuthology.orchestra.run.smithi189:> ' -- /home/ubuntu/cephtest/mnt.0) 2020-11-20T12:05:13.486 INFO:teuthology.orchestra.run.smithi189.stdout:fuseblk 2020-11-20T12:05:13.487 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2020-11-20T12:05:13.487 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T12:05:13.488 INFO:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2020-11-20T12:05:13.556 INFO:teuthology.run_tasks:Running task mds_thrash... 2020-11-20T12:05:15.029 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:05:15.166 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.158+0000 7f23b2c48700 1 Processor -- start 2020-11-20T12:05:15.167 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.158+0000 7f23b2c48700 1 -- start start 2020-11-20T12:05:15.167 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.158+0000 7f23b2c48700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f23ac13b220 0x7f23ac1461d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.167 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.158+0000 7f23b2c48700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23ac13e320 con 0x7f23ac0a8840 2020-11-20T12:05:15.168 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.158+0000 7f23b2c48700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23ac066800 con 0x7f23ac13a680 2020-11-20T12:05:15.168 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.158+0000 7f23b2c48700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23ac066aa0 con 0x7f23ac13b220 2020-11-20T12:05:15.168 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.158+0000 7f23b1445700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f23ac13a680 0x7f23ac13ab10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50578/0 (socket says 172.21.15.189:50578) 2020-11-20T12:05:15.168 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.158+0000 7f23b1445700 1 -- 172.21.15.189:0/3290827499 learned_addr learned my addr 172.21.15.189:0/3290827499 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:15.168 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.158+0000 7f23b2447700 1 --2- 172.21.15.189:0/3290827499 >> v2:172.21.15.47:3300/0 conn(0x7f23ac13b220 0x7f23ac1461d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.170 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.158+0000 7f23b0c44700 1 -- 172.21.15.189:0/3290827499 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 175581338 0 0) 0x7f23ac066800 con 0x7f23ac13a680 2020-11-20T12:05:15.170 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.158+0000 7f23b0c44700 1 -- 172.21.15.189:0/3290827499 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f238c003240 con 0x7f23ac13a680 2020-11-20T12:05:15.171 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.158+0000 7f23b0c44700 1 -- 172.21.15.189:0/3290827499 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1596969167 0 0) 0x7f23ac13e320 con 0x7f23ac0a8840 2020-11-20T12:05:15.171 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.158+0000 7f23b0c44700 1 -- 172.21.15.189:0/3290827499 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23ac066800 con 0x7f23ac0a8840 2020-11-20T12:05:15.172 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.158+0000 7f23b2447700 1 -- 172.21.15.189:0/3290827499 >> v1:172.21.15.189:6789/0 conn(0x7f23ac13a680 legacy=0x7f23ac13ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.172 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.158+0000 7f23b0c44700 1 -- 172.21.15.189:0/3290827499 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4112359747 0 0) 0x7f238c003240 con 0x7f23ac13a680 2020-11-20T12:05:15.172 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.158+0000 7f23b2447700 1 -- 172.21.15.189:0/3290827499 >> v1:172.21.15.47:6789/0 conn(0x7f23ac0a8840 legacy=0x7f23ac13dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.173 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.158+0000 7f23b2447700 1 -- 172.21.15.189:0/3290827499 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ac0672f0 con 0x7f23ac13b220 2020-11-20T12:05:15.173 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.158+0000 7f23b2447700 1 --2- 172.21.15.189:0/3290827499 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23ac13b220 0x7f23ac1461d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f23a4009c10 tx=0x7f23a40091d0).ready entity=mon.0 client_cookie=1ca647e75edc9e75 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:15.180 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b0c44700 1 -- 172.21.15.189:0/3290827499 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23940014e0 con 0x7f23ac13b220 2020-11-20T12:05:15.180 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b0c44700 1 -- 172.21.15.189:0/3290827499 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a400b6b0 con 0x7f23ac13b220 2020-11-20T12:05:15.180 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b0c44700 1 -- 172.21.15.189:0/3290827499 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239c0014e0 con 0x7f23ac13b220 2020-11-20T12:05:15.181 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b2c48700 1 -- 172.21.15.189:0/3290827499 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23ac13b220 msgr2=0x7f23ac1461d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.181 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b2c48700 1 --2- 172.21.15.189:0/3290827499 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23ac13b220 0x7f23ac1461d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f23a4009c10 tx=0x7f23a40091d0).stop 2020-11-20T12:05:15.181 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b2c48700 1 -- 172.21.15.189:0/3290827499 shutdown_connections 2020-11-20T12:05:15.182 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b2c48700 1 --2- 172.21.15.189:0/3290827499 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23ac13b220 0x7f23ac1461d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.183 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b2c48700 1 -- 172.21.15.189:0/3290827499 >> 172.21.15.189:0/3290827499 conn(0x7f23ac060170 msgr2=0x7f23ac060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:15.183 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b2c48700 1 -- 172.21.15.189:0/3290827499 shutdown_connections 2020-11-20T12:05:15.183 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b2c48700 1 -- 172.21.15.189:0/3290827499 wait complete. 2020-11-20T12:05:15.183 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b2c48700 1 Processor -- start 2020-11-20T12:05:15.184 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b2c48700 1 -- start start 2020-11-20T12:05:15.184 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b2c48700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23ac0a8840 0x7f23ac0a4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.184 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b2c48700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23ac13a680 0x7f23ac0a4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.184 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b2c48700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f23ac13b220 0x7f23ac0a52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.185 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b2c48700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac0a4e50 con 0x7f23ac0a8840 2020-11-20T12:05:15.185 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b2c48700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac132a70 con 0x7f23ac13a680 2020-11-20T12:05:15.185 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b2c48700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac13c6b0 con 0x7f23ac13b220 2020-11-20T12:05:15.185 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b1445700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23ac13a680 0x7f23ac0a4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.185 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b1445700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23ac13a680 0x7f23ac0a4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48960/0 (socket says 172.21.15.189:48960) 2020-11-20T12:05:15.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b1445700 1 -- 172.21.15.189:0/341363458 learned_addr learned my addr 172.21.15.189:0/341363458 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:15.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b1c46700 1 --2- 172.21.15.189:0/341363458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23ac0a8840 0x7f23ac0a4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b2447700 1 --2- 172.21.15.189:0/341363458 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f23ac13b220 0x7f23ac0a52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b1445700 1 -- 172.21.15.189:0/341363458 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f23ac13b220 msgr2=0x7f23ac0a52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b1445700 1 --2- 172.21.15.189:0/341363458 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f23ac13b220 0x7f23ac0a52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b1445700 1 -- 172.21.15.189:0/341363458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23ac0a8840 msgr2=0x7f23ac0a4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b1445700 1 --2- 172.21.15.189:0/341363458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23ac0a8840 0x7f23ac0a4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b1445700 1 -- 172.21.15.189:0/341363458 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ac13c950 con 0x7f23ac13a680 2020-11-20T12:05:15.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b1c46700 1 --2- 172.21.15.189:0/341363458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23ac0a8840 0x7f23ac0a4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:15.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b2447700 1 --2- 172.21.15.189:0/341363458 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f23ac13b220 0x7f23ac0a52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:15.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.162+0000 7f23b1445700 1 --2- 172.21.15.189:0/341363458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23ac13a680 0x7f23ac0a4900 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f239c00c7c0 tx=0x7f239c0014c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:15.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.166+0000 7f23a9ffb700 1 -- 172.21.15.189:0/341363458 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239c007170 con 0x7f23ac13a680 2020-11-20T12:05:15.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.166+0000 7f23a9ffb700 1 -- 172.21.15.189:0/341363458 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239c0072f0 con 0x7f23ac13a680 2020-11-20T12:05:15.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.166+0000 7f23b2c48700 1 -- 172.21.15.189:0/341363458 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23ac138310 con 0x7f23ac13a680 2020-11-20T12:05:15.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.166+0000 7f23a9ffb700 1 -- 172.21.15.189:0/341363458 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239c004960 con 0x7f23ac13a680 2020-11-20T12:05:15.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.166+0000 7f23b2c48700 1 -- 172.21.15.189:0/341363458 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23ac13c280 con 0x7f23ac13a680 2020-11-20T12:05:15.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.166+0000 7f23b2c48700 1 -- 172.21.15.189:0/341363458 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23ac0a4fc0 con 0x7f23ac13a680 2020-11-20T12:05:15.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.166+0000 7f23a9ffb700 1 -- 172.21.15.189:0/341363458 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f239c010070 con 0x7f23ac13a680 2020-11-20T12:05:15.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.166+0000 7f23a9ffb700 1 --2- 172.21.15.189:0/341363458 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f23880557c0 0x7f2388057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.166+0000 7f23a9ffb700 1 -- 172.21.15.189:0/341363458 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f239c06e2e0 con 0x7f23ac13a680 2020-11-20T12:05:15.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.166+0000 7f23b1c46700 1 --2- 172.21.15.189:0/341363458 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f23880557c0 0x7f2388057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.166+0000 7f23b1c46700 1 --2- 172.21.15.189:0/341363458 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f23880557c0 0x7f2388057c80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f23940011a0 tx=0x7f2394008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:15.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.170+0000 7f23a9ffb700 1 -- 172.21.15.189:0/341363458 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f239c044180 con 0x7f23ac13a680 2020-11-20T12:05:15.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.322+0000 7f23b2c48700 1 -- 172.21.15.189:0/341363458 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23ac13c4d0 con 0x7f23ac13a680 2020-11-20T12:05:15.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.326+0000 7f23a9ffb700 1 -- 172.21.15.189:0/341363458 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16826 (secure 0 0 0) 0x7f239c0477a0 con 0x7f23ac13a680 2020-11-20T12:05:15.333 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:05:15.333 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":22,"fsid":"cda18d0e-64b3-41bb-964a-084069c3e648","created":"2020-11-20T12:04:37.538706+0000","modified":"2020-11-20T12:04:55.015818+0000","last_up_change":"2020-11-20T12:04:43.006631+0000","last_in_change":"2020-11-20T12:04:43.006631+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:41.395395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:48.550454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:52.101837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:53.018196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c81214e1-4f84-4aaa-8ceb-99c398b2a89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6811","nonce":11760},{"type":"v1","addr":"172.21.15.189:6814","nonce":11760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":11760},{"type":"v1","addr":"172.21.15.189:6818","nonce":11760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":11760},{"type":"v1","addr":"172.21.15.189:6828","nonce":11760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6821","nonce":11760},{"type":"v1","addr":"172.21.15.189:6824","nonce":11760}]},"public_addr":"172.21.15.189:6814/11760","cluster_addr":"172.21.15.189:6818/11760","heartbeat_back_addr":"172.21.15.189:6828/11760","heartbeat_front_addr":"172.21.15.189:6824/11760","state":["exists","up"]},{"osd":1,"uuid":"d2f3ec7d-d403-484f-b5ed-b2e6b079d63d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6805","nonce":11761},{"type":"v1","addr":"172.21.15.189:6807","nonce":11761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":11761},{"type":"v1","addr":"172.21.15.189:6812","nonce":11761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6817","nonce":11761},{"type":"v1","addr":"172.21.15.189:6820","nonce":11761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6813","nonce":11761},{"type":"v1","addr":"172.21.15.189:6815","nonce":11761}]},"public_addr":"172.21.15.189:6807/11761","cluster_addr":"172.21.15.189:6812/11761","heartbeat_back_addr":"172.21.15.189:6820/11761","heartbeat_front_addr":"172.21.15.189:6815/11761","state":["exists","up"]},{"osd":2,"uuid":"e6cf4790-000c-43eb-8844-175f04c60503","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6819","nonce":11751},{"type":"v1","addr":"172.21.15.189:6822","nonce":11751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":11751},{"type":"v1","addr":"172.21.15.189:6825","nonce":11751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":11751},{"type":"v1","addr":"172.21.15.189:6831","nonce":11751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6827","nonce":11751},{"type":"v1","addr":"172.21.15.189:6829","nonce":11751}]},"public_addr":"172.21.15.189:6822/11751","cluster_addr":"172.21.15.189:6825/11751","heartbeat_back_addr":"172.21.15.189:6831/11751","heartbeat_front_addr":"172.21.15.189:6829/11751","state":["exists","up"]},{"osd":3,"uuid":"ba49dfac-f2b9-476e-853d-42f05e21a77a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":11762},{"type":"v1","addr":"172.21.15.189:6801","nonce":11762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":11762},{"type":"v1","addr":"172.21.15.189:6803","nonce":11762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":11762},{"type":"v1","addr":"172.21.15.189:6809","nonce":11762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":11762},{"type":"v1","addr":"172.21.15.189:6806","nonce":11762}]},"public_addr":"172.21.15.189:6801/11762","cluster_addr":"172.21.15.189:6803/11762","heartbeat_back_addr":"172.21.15.189:6809/11762","heartbeat_front_addr":"172.21.15.189:6806/11762","state":["exists","up"]},{"osd":4,"uuid":"3ca7b438-38b2-4689-8b18-fb5a8ddf6aee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":12114},{"type":"v1","addr":"172.21.15.47:6825","nonce":12114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":12114},{"type":"v1","addr":"172.21.15.47:6827","nonce":12114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":12114},{"type":"v1","addr":"172.21.15.47:6831","nonce":12114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":12114},{"type":"v1","addr":"172.21.15.47:6829","nonce":12114}]},"public_addr":"172.21.15.47:6825/12114","cluster_addr":"172.21.15.47:6827/12114","heartbeat_back_addr":"172.21.15.47:6831/12114","heartbeat_front_addr":"172.21.15.47:6829/12114","state":["exists","up"]},{"osd":5,"uuid":"6a45fca7-83d3-44fd-a1d4-b01e4201a101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":12111},{"type":"v1","addr":"172.21.15.47:6801","nonce":12111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":12111},{"type":"v1","addr":"172.21.15.47:6803","nonce":12111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":12111},{"type":"v1","addr":"172.21.15.47:6807","nonce":12111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":12111},{"type":"v1","addr":"172.21.15.47:6805","nonce":12111}]},"public_addr":"172.21.15.47:6801/12111","cluster_addr":"172.21.15.47:6803/12111","heartbeat_back_addr":"172.21.15.47:6807/12111","heartbeat_front_addr":"172.21.15.47:6805/12111","state":["exists","up"]},{"osd":6,"uuid":"c4b1b9ed-61f8-4677-9755-dff59a30e447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":12113},{"type":"v1","addr":"172.21.15.47:6817","nonce":12113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":12113},{"type":"v1","addr":"172.21.15.47:6819","nonce":12113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":12113},{"type":"v1","addr":"172.21.15.47:6823","nonce":12113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":12113},{"type":"v1","addr":"172.21.15.47:6821","nonce":12113}]},"public_addr":"172.21.15.47:6817/12113","cluster_addr":"172.21.15.47:6819/12113","heartbeat_back_addr":"172.21.15.47:6823/12113","heartbeat_front_addr":"172.21.15.47:6821/12113","state":["exists","up"]},{"osd":7,"uuid":"4608e4ed-cc40-4c5c-ba8b-3779bbc62f65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":12112},{"type":"v1","addr":"172.21.15.47:6809","nonce":12112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":12112},{"type":"v1","addr":"172.21.15.47:6811","nonce":12112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":12112},{"type":"v1","addr":"172.21.15.47:6815","nonce":12112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":12112},{"type":"v1","addr":"172.21.15.47:6813","nonce":12112}]},"public_addr":"172.21.15.47:6809/12112","cluster_addr":"172.21.15.47:6811/12112","heartbeat_back_addr":"172.21.15.47:6815/12112","heartbeat_front_addr":"172.21.15.47:6813/12112","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.435265+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.472776+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.388370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.417366+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.102392+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.116529+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.167430+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.148139+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:05:15.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.326+0000 7f23b2c48700 1 -- 172.21.15.189:0/341363458 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f23880557c0 msgr2=0x7f2388057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.326+0000 7f23b2c48700 1 --2- 172.21.15.189:0/341363458 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f23880557c0 0x7f2388057c80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f23940011a0 tx=0x7f2394008040).stop 2020-11-20T12:05:15.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.326+0000 7f23b2c48700 1 -- 172.21.15.189:0/341363458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23ac13a680 msgr2=0x7f23ac0a4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.326+0000 7f23b2c48700 1 --2- 172.21.15.189:0/341363458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23ac13a680 0x7f23ac0a4900 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f239c00c7c0 tx=0x7f239c0014c0).stop 2020-11-20T12:05:15.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.326+0000 7f23b2c48700 1 -- 172.21.15.189:0/341363458 shutdown_connections 2020-11-20T12:05:15.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.326+0000 7f23b2c48700 1 --2- 172.21.15.189:0/341363458 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f23880557c0 0x7f2388057c80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.336 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.326+0000 7f23b2c48700 1 --2- 172.21.15.189:0/341363458 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f23ac0a8840 0x7f23ac0a4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.336 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.326+0000 7f23b2c48700 1 --2- 172.21.15.189:0/341363458 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f23ac13b220 0x7f23ac0a52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.336 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.326+0000 7f23b2c48700 1 --2- 172.21.15.189:0/341363458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23ac13a680 0x7f23ac0a4900 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.336 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.326+0000 7f23b2c48700 1 -- 172.21.15.189:0/341363458 >> 172.21.15.189:0/341363458 conn(0x7f23ac060170 msgr2=0x7f23ac1341c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:15.337 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.326+0000 7f23b2c48700 1 -- 172.21.15.189:0/341363458 shutdown_connections 2020-11-20T12:05:15.337 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.326+0000 7f23b2c48700 1 -- 172.21.15.189:0/341363458 wait complete. 2020-11-20T12:05:15.348 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:41.395395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:48.550454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:04:52.101837+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:04:53.018196+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:05:15.349 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:05:15.477 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.470+0000 7fc66334e700 1 Processor -- start 2020-11-20T12:05:15.477 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.470+0000 7fc66334e700 1 -- start start 2020-11-20T12:05:15.480 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.470+0000 7fc66334e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc65c0b19e0 0x7fc65c13a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.480 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.470+0000 7fc66334e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc65c066a30 0x7fc65c13ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.480 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.470+0000 7fc66334e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc65c066fb0 con 0x7fc65c062930 2020-11-20T12:05:15.480 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.470+0000 7fc66334e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc65c0666b0 con 0x7fc65c0b19e0 2020-11-20T12:05:15.480 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.470+0000 7fc66334e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c067250 con 0x7fc65c066a30 2020-11-20T12:05:15.481 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.470+0000 7fc661b4b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc65c0b19e0 0x7fc65c13a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.481 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.470+0000 7fc661b4b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc65c0b19e0 0x7fc65c13a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48970/0 (socket says 172.21.15.189:48970) 2020-11-20T12:05:15.481 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.470+0000 7fc661b4b700 1 -- 172.21.15.189:0/1785208580 learned_addr learned my addr 172.21.15.189:0/1785208580 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:15.481 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.470+0000 7fc662b4d700 1 --2- 172.21.15.189:0/1785208580 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc65c066a30 0x7fc65c13ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.482 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.470+0000 7fc661b4b700 1 -- 172.21.15.189:0/1785208580 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc65c066a30 msgr2=0x7fc65c13ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:15.482 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.470+0000 7fc661b4b700 1 --2- 172.21.15.189:0/1785208580 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc65c066a30 0x7fc65c13ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.482 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.470+0000 7fc661b4b700 1 -- 172.21.15.189:0/1785208580 >> v1:172.21.15.47:6789/0 conn(0x7fc65c062930 legacy=0x7fc65c0b12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.482 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.470+0000 7fc661b4b700 1 -- 172.21.15.189:0/1785208580 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65c0b0e30 con 0x7fc65c0b19e0 2020-11-20T12:05:15.482 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.470+0000 7fc661b4b700 1 --2- 172.21.15.189:0/1785208580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc65c0b19e0 0x7fc65c13a7f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fc64c006fd0 tx=0x7fc64c006eb0).ready entity=mon.1 client_cookie=8cbf60db9de4addd server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:15.483 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc66134a700 1 -- 172.21.15.189:0/1785208580 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c0098d0 con 0x7fc65c0b19e0 2020-11-20T12:05:15.483 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc66134a700 1 -- 172.21.15.189:0/1785208580 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64c009a30 con 0x7fc65c0b19e0 2020-11-20T12:05:15.483 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc66134a700 1 -- 172.21.15.189:0/1785208580 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c009db0 con 0x7fc65c0b19e0 2020-11-20T12:05:15.483 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc66334e700 1 -- 172.21.15.189:0/1785208580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc65c0b19e0 msgr2=0x7fc65c13a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.483 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc66334e700 1 --2- 172.21.15.189:0/1785208580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc65c0b19e0 0x7fc65c13a7f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fc64c006fd0 tx=0x7fc64c006eb0).stop 2020-11-20T12:05:15.484 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc66334e700 1 -- 172.21.15.189:0/1785208580 shutdown_connections 2020-11-20T12:05:15.484 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc66334e700 1 --2- 172.21.15.189:0/1785208580 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc65c066a30 0x7fc65c13ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.484 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc66334e700 1 --2- 172.21.15.189:0/1785208580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc65c0b19e0 0x7fc65c13a7f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.484 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc66334e700 1 -- 172.21.15.189:0/1785208580 >> 172.21.15.189:0/1785208580 conn(0x7fc65c065080 msgr2=0x7fc65c062080 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:15.484 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc66334e700 1 -- 172.21.15.189:0/1785208580 shutdown_connections 2020-11-20T12:05:15.485 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc66334e700 1 -- 172.21.15.189:0/1785208580 wait complete. 2020-11-20T12:05:15.487 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc66334e700 1 Processor -- start 2020-11-20T12:05:15.487 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc66334e700 1 -- start start 2020-11-20T12:05:15.487 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc66334e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc65c062930 0x7fc65c14cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.487 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc66334e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc65c066a30 0x7fc65c0a9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.487 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc66334e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc65c0b19e0 0x7fc65c14d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.488 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc66334e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c14d450 con 0x7fc65c066a30 2020-11-20T12:05:15.488 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc66334e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c0a9f90 con 0x7fc65c0b19e0 2020-11-20T12:05:15.488 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc66334e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c0a8430 con 0x7fc65c062930 2020-11-20T12:05:15.488 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc662b4d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc65c0b19e0 0x7fc65c14d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.489 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc662b4d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc65c0b19e0 0x7fc65c14d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48978/0 (socket says 172.21.15.189:48978) 2020-11-20T12:05:15.489 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc662b4d700 1 -- 172.21.15.189:0/4267051085 learned_addr learned my addr 172.21.15.189:0/4267051085 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:15.489 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc661b4b700 1 --2- 172.21.15.189:0/4267051085 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc65c066a30 0x7fc65c0a9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.489 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc66234c700 1 --2- 172.21.15.189:0/4267051085 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc65c062930 0x7fc65c14cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.489 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc662b4d700 1 -- 172.21.15.189:0/4267051085 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc65c062930 msgr2=0x7fc65c14cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.490 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc662b4d700 1 --2- 172.21.15.189:0/4267051085 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc65c062930 0x7fc65c14cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.490 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc662b4d700 1 -- 172.21.15.189:0/4267051085 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc65c066a30 msgr2=0x7fc65c0a9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.490 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc662b4d700 1 --2- 172.21.15.189:0/4267051085 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc65c066a30 0x7fc65c0a9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.490 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc662b4d700 1 -- 172.21.15.189:0/4267051085 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65c0a86d0 con 0x7fc65c0b19e0 2020-11-20T12:05:15.490 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc662b4d700 1 --2- 172.21.15.189:0/4267051085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc65c0b19e0 0x7fc65c14d860 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fc65400eb70 tx=0x7fc654008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:15.491 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc64a7fc700 1 -- 172.21.15.189:0/4267051085 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65400c830 con 0x7fc65c0b19e0 2020-11-20T12:05:15.491 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc66334e700 1 -- 172.21.15.189:0/4267051085 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc65c0a9c50 con 0x7fc65c0b19e0 2020-11-20T12:05:15.491 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc66334e700 1 -- 172.21.15.189:0/4267051085 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc65c13aea0 con 0x7fc65c0b19e0 2020-11-20T12:05:15.491 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc64a7fc700 1 -- 172.21.15.189:0/4267051085 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65400cba0 con 0x7fc65c0b19e0 2020-11-20T12:05:15.491 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.474+0000 7fc64a7fc700 1 -- 172.21.15.189:0/4267051085 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65400cd10 con 0x7fc65c0b19e0 2020-11-20T12:05:15.492 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.478+0000 7fc66334e700 1 -- 172.21.15.189:0/4267051085 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc65c0a9010 con 0x7fc65c0b19e0 2020-11-20T12:05:15.493 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.482+0000 7fc64a7fc700 1 -- 172.21.15.189:0/4267051085 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fc65401d460 con 0x7fc65c0b19e0 2020-11-20T12:05:15.493 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.482+0000 7fc64a7fc700 1 --2- 172.21.15.189:0/4267051085 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc6340557f0 0x7fc634057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.493 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.482+0000 7fc64a7fc700 1 -- 172.21.15.189:0/4267051085 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc65406f650 con 0x7fc65c0b19e0 2020-11-20T12:05:15.493 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.482+0000 7fc66234c700 1 --2- 172.21.15.189:0/4267051085 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc6340557f0 0x7fc634057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.493 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.482+0000 7fc66234c700 1 --2- 172.21.15.189:0/4267051085 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc6340557f0 0x7fc634057cb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fc658003510 tx=0x7fc658009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:15.495 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.486+0000 7fc64a7fc700 1 -- 172.21.15.189:0/4267051085 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc654045490 con 0x7fc65c0b19e0 2020-11-20T12:05:15.641 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.634+0000 7fc66334e700 1 -- 172.21.15.189:0/4267051085 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fc65c0446d0 con 0x7fc65c0b19e0 2020-11-20T12:05:15.642 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.634+0000 7fc64a7fc700 1 -- 172.21.15.189:0/4267051085 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v22) v1 ==== 110+0+10 (secure 0 0 0) 0x7fc654048ab0 con 0x7fc65c0b19e0 2020-11-20T12:05:15.642 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2020-11-20T12:05:15.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.638+0000 7fc66334e700 1 -- 172.21.15.189:0/4267051085 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc6340557f0 msgr2=0x7fc634057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.638+0000 7fc66334e700 1 --2- 172.21.15.189:0/4267051085 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc6340557f0 0x7fc634057cb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fc658003510 tx=0x7fc658009040).stop 2020-11-20T12:05:15.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.638+0000 7fc66334e700 1 -- 172.21.15.189:0/4267051085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc65c0b19e0 msgr2=0x7fc65c14d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.638+0000 7fc66334e700 1 --2- 172.21.15.189:0/4267051085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc65c0b19e0 0x7fc65c14d860 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fc65400eb70 tx=0x7fc654008040).stop 2020-11-20T12:05:15.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.638+0000 7fc66334e700 1 -- 172.21.15.189:0/4267051085 shutdown_connections 2020-11-20T12:05:15.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.638+0000 7fc66334e700 1 --2- 172.21.15.189:0/4267051085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc65c0b19e0 0x7fc65c14d860 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.638+0000 7fc66334e700 1 --2- 172.21.15.189:0/4267051085 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc65c062930 0x7fc65c14cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.646 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.638+0000 7fc66334e700 1 --2- 172.21.15.189:0/4267051085 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc6340557f0 0x7fc634057cb0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.646 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.638+0000 7fc66334e700 1 --2- 172.21.15.189:0/4267051085 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc65c066a30 0x7fc65c0a9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.646 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.638+0000 7fc66334e700 1 -- 172.21.15.189:0/4267051085 >> 172.21.15.189:0/4267051085 conn(0x7fc65c065080 msgr2=0x7fc65c0b11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:15.646 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.638+0000 7fc66334e700 1 -- 172.21.15.189:0/4267051085 shutdown_connections 2020-11-20T12:05:15.647 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.638+0000 7fc66334e700 1 -- 172.21.15.189:0/4267051085 wait complete. 2020-11-20T12:05:15.656 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:05:15.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.778+0000 7f4f17432700 1 Processor -- start 2020-11-20T12:05:15.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.778+0000 7f4f17432700 1 -- start start 2020-11-20T12:05:15.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.778+0000 7f4f17432700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4f100ae0d0 0x7f4f1013aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.778+0000 7f4f17432700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f4f1013af60 0x7f4f10060170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.778+0000 7f4f17432700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f10066960 con 0x7f4f10060660 2020-11-20T12:05:15.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.778+0000 7f4f17432700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f1013a680 con 0x7f4f1013af60 2020-11-20T12:05:15.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.778+0000 7f4f17432700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f10067110 con 0x7f4f100ae0d0 2020-11-20T12:05:15.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.778+0000 7f4f151ce700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4f100ae0d0 0x7f4f1013aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.778+0000 7f4f151ce700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4f100ae0d0 0x7f4f1013aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48982/0 (socket says 172.21.15.189:48982) 2020-11-20T12:05:15.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.778+0000 7f4f151ce700 1 -- 172.21.15.189:0/2215185154 learned_addr learned my addr 172.21.15.189:0/2215185154 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:15.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.778+0000 7f4f151ce700 1 -- 172.21.15.189:0/2215185154 >> v2:172.21.15.47:3300/0 conn(0x7f4f1013af60 msgr2=0x7f4f10060170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:15.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.778+0000 7f4f149cd700 1 --2- 172.21.15.189:0/2215185154 >> v2:172.21.15.47:3300/0 conn(0x7f4f1013af60 0x7f4f10060170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.778+0000 7f4f151ce700 1 --2- 172.21.15.189:0/2215185154 >> v2:172.21.15.47:3300/0 conn(0x7f4f1013af60 0x7f4f10060170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.778+0000 7f4f151ce700 1 -- 172.21.15.189:0/2215185154 >> v1:172.21.15.47:6789/0 conn(0x7f4f10060660 legacy=0x7f4f10137170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.778+0000 7f4f151ce700 1 -- 172.21.15.189:0/2215185154 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f100673b0 con 0x7f4f100ae0d0 2020-11-20T12:05:15.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.778+0000 7f4f149cd700 1 --2- 172.21.15.189:0/2215185154 >> v2:172.21.15.47:3300/0 conn(0x7f4f1013af60 0x7f4f10060170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:15.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.778+0000 7f4f151ce700 1 --2- 172.21.15.189:0/2215185154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f100ae0d0 0x7f4f1013aa70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f4f0c006fd0 tx=0x7f4f0c006eb0).ready entity=mon.1 client_cookie=47d588070859ce7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:15.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.778+0000 7f4f07fff700 1 -- 172.21.15.189:0/2215185154 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c009890 con 0x7f4f100ae0d0 2020-11-20T12:05:15.790 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.778+0000 7f4f07fff700 1 -- 172.21.15.189:0/2215185154 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f0c0099f0 con 0x7f4f100ae0d0 2020-11-20T12:05:15.790 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.778+0000 7f4f07fff700 1 -- 172.21.15.189:0/2215185154 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c009d70 con 0x7f4f100ae0d0 2020-11-20T12:05:15.792 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.782+0000 7f4f17432700 1 -- 172.21.15.189:0/2215185154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f100ae0d0 msgr2=0x7f4f1013aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.782+0000 7f4f17432700 1 --2- 172.21.15.189:0/2215185154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f100ae0d0 0x7f4f1013aa70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f4f0c006fd0 tx=0x7f4f0c006eb0).stop 2020-11-20T12:05:15.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.782+0000 7f4f17432700 1 -- 172.21.15.189:0/2215185154 shutdown_connections 2020-11-20T12:05:15.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.782+0000 7f4f17432700 1 --2- 172.21.15.189:0/2215185154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f100ae0d0 0x7f4f1013aa70 secure :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0x7f4f0c006fd0 tx=0x7f4f0c006eb0).stop 2020-11-20T12:05:15.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.782+0000 7f4f17432700 1 --2- 172.21.15.189:0/2215185154 >> v2:172.21.15.47:3300/0 conn(0x7f4f1013af60 0x7f4f10060170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.782+0000 7f4f17432700 1 -- 172.21.15.189:0/2215185154 >> 172.21.15.189:0/2215185154 conn(0x7f4f100b1850 msgr2=0x7f4f100ad820 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:15.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.782+0000 7f4f17432700 1 -- 172.21.15.189:0/2215185154 shutdown_connections 2020-11-20T12:05:15.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.782+0000 7f4f17432700 1 -- 172.21.15.189:0/2215185154 wait complete. 2020-11-20T12:05:15.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.782+0000 7f4f17432700 1 Processor -- start 2020-11-20T12:05:15.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.782+0000 7f4f17432700 1 -- start start 2020-11-20T12:05:15.796 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.782+0000 7f4f17432700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f10060660 0x7f4f1013e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.796 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.782+0000 7f4f17432700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f1013af60 0x7f4f100aabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.796 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.782+0000 7f4f17432700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4f10149ce0 0x7f4f1013c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.796 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.782+0000 7f4f17432700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1013d710 con 0x7f4f1013af60 2020-11-20T12:05:15.796 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.782+0000 7f4f17432700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f100a9710 con 0x7f4f10060660 2020-11-20T12:05:15.797 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.782+0000 7f4f17432700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1013e9e0 con 0x7f4f10149ce0 2020-11-20T12:05:15.797 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.782+0000 7f4f151ce700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f10060660 0x7f4f1013e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.797 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.786+0000 7f4f151ce700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f10060660 0x7f4f1013e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48988/0 (socket says 172.21.15.189:48988) 2020-11-20T12:05:15.797 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.786+0000 7f4f151ce700 1 -- 172.21.15.189:0/1451150202 learned_addr learned my addr 172.21.15.189:0/1451150202 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:15.797 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.786+0000 7f4f159cf700 1 --2- 172.21.15.189:0/1451150202 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4f10149ce0 0x7f4f1013c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.786+0000 7f4f149cd700 1 --2- 172.21.15.189:0/1451150202 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f1013af60 0x7f4f100aabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.786+0000 7f4f151ce700 1 -- 172.21.15.189:0/1451150202 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4f10149ce0 msgr2=0x7f4f1013c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.786+0000 7f4f151ce700 1 --2- 172.21.15.189:0/1451150202 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4f10149ce0 0x7f4f1013c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.786+0000 7f4f151ce700 1 -- 172.21.15.189:0/1451150202 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f1013af60 msgr2=0x7f4f100aabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.786+0000 7f4f151ce700 1 --2- 172.21.15.189:0/1451150202 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f1013af60 0x7f4f100aabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.786+0000 7f4f151ce700 1 -- 172.21.15.189:0/1451150202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1013e740 con 0x7f4f10060660 2020-11-20T12:05:15.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.786+0000 7f4f149cd700 1 --2- 172.21.15.189:0/1451150202 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f1013af60 0x7f4f100aabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:15.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.786+0000 7f4f159cf700 1 --2- 172.21.15.189:0/1451150202 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4f10149ce0 0x7f4f1013c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:15.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.786+0000 7f4f151ce700 1 --2- 172.21.15.189:0/1451150202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f10060660 0x7f4f1013e150 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f4f0c010000 tx=0x7f4f0c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:15.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.786+0000 7f4f057fa700 1 -- 172.21.15.189:0/1451150202 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c01e460 con 0x7f4f10060660 2020-11-20T12:05:15.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.786+0000 7f4f057fa700 1 -- 172.21.15.189:0/1451150202 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f0c01e5c0 con 0x7f4f10060660 2020-11-20T12:05:15.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.786+0000 7f4f17432700 1 -- 172.21.15.189:0/1451150202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f1013c340 con 0x7f4f10060660 2020-11-20T12:05:15.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.786+0000 7f4f057fa700 1 -- 172.21.15.189:0/1451150202 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c01e960 con 0x7f4f10060660 2020-11-20T12:05:15.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.786+0000 7f4f17432700 1 -- 172.21.15.189:0/1451150202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f100a9f20 con 0x7f4f10060660 2020-11-20T12:05:15.801 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.786+0000 7f4f17432700 1 -- 172.21.15.189:0/1451150202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f1013c500 con 0x7f4f10060660 2020-11-20T12:05:15.801 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.786+0000 7f4f057fa700 1 -- 172.21.15.189:0/1451150202 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f4f0c025030 con 0x7f4f10060660 2020-11-20T12:05:15.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.786+0000 7f4f057fa700 1 --2- 172.21.15.189:0/1451150202 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4ee80557c0 0x7f4ee8057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:15.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.786+0000 7f4f057fa700 1 -- 172.21.15.189:0/1451150202 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4f0c06ffa0 con 0x7f4f10060660 2020-11-20T12:05:15.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.786+0000 7f4f149cd700 1 --2- 172.21.15.189:0/1451150202 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4ee80557c0 0x7f4ee8057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:15.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.790+0000 7f4f149cd700 1 --2- 172.21.15.189:0/1451150202 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4ee80557c0 0x7f4ee8057c80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f4f00006fd0 tx=0x7f4f00008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:15.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.794+0000 7f4f057fa700 1 -- 172.21.15.189:0/1451150202 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4f0c045ef0 con 0x7f4f10060660 2020-11-20T12:05:15.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.946+0000 7f4f17432700 1 -- 172.21.15.189:0/1451150202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4f10044f60 con 0x7f4f10060660 2020-11-20T12:05:15.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.946+0000 7f4f057fa700 1 -- 172.21.15.189:0/1451150202 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v22) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4f0c023030 con 0x7f4f10060660 2020-11-20T12:05:15.953 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 8 2020-11-20T12:05:15.955 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.946+0000 7f4f17432700 1 -- 172.21.15.189:0/1451150202 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4ee80557c0 msgr2=0x7f4ee8057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.955 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.946+0000 7f4f17432700 1 --2- 172.21.15.189:0/1451150202 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4ee80557c0 0x7f4ee8057c80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f4f00006fd0 tx=0x7f4f00008040).stop 2020-11-20T12:05:15.955 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.946+0000 7f4f17432700 1 -- 172.21.15.189:0/1451150202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f10060660 msgr2=0x7f4f1013e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:15.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.946+0000 7f4f17432700 1 --2- 172.21.15.189:0/1451150202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f10060660 0x7f4f1013e150 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f4f0c010000 tx=0x7f4f0c015000).stop 2020-11-20T12:05:15.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.946+0000 7f4f17432700 1 -- 172.21.15.189:0/1451150202 shutdown_connections 2020-11-20T12:05:15.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.946+0000 7f4f17432700 1 --2- 172.21.15.189:0/1451150202 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4f10149ce0 0x7f4f1013c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.946+0000 7f4f17432700 1 --2- 172.21.15.189:0/1451150202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f10060660 0x7f4f1013e150 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.946+0000 7f4f17432700 1 --2- 172.21.15.189:0/1451150202 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4ee80557c0 0x7f4ee8057c80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.946+0000 7f4f17432700 1 --2- 172.21.15.189:0/1451150202 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4f1013af60 0x7f4f100aabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:15.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.946+0000 7f4f17432700 1 -- 172.21.15.189:0/1451150202 >> 172.21.15.189:0/1451150202 conn(0x7f4f100b1850 msgr2=0x7f4f100ad800 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:15.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.950+0000 7f4f17432700 1 -- 172.21.15.189:0/1451150202 shutdown_connections 2020-11-20T12:05:15.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:15.950+0000 7f4f17432700 1 -- 172.21.15.189:0/1451150202 wait complete. 2020-11-20T12:05:15.967 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:05:16.093 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.086+0000 7f86501b0700 1 Processor -- start 2020-11-20T12:05:16.094 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.086+0000 7f86501b0700 1 -- start start 2020-11-20T12:05:16.095 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.086+0000 7f86501b0700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f86480a3ad0 0x7f864813a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:16.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.086+0000 7f86501b0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8648068450 0x7f864813ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:16.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.086+0000 7f86501b0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8648066ae0 con 0x7f8648060310 2020-11-20T12:05:16.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.086+0000 7f86501b0700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8648066d80 con 0x7f86480a3ad0 2020-11-20T12:05:16.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.086+0000 7f86501b0700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86480606f0 con 0x7f8648068450 2020-11-20T12:05:16.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.090+0000 7f864df4c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8648060310 0x7f86480a9040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50618/0 (socket says 172.21.15.189:50618) 2020-11-20T12:05:16.097 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.090+0000 7f864df4c700 1 -- 172.21.15.189:0/1066992094 learned_addr learned my addr 172.21.15.189:0/1066992094 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:16.097 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.090+0000 7f864d74b700 1 --2- 172.21.15.189:0/1066992094 >> v2:172.21.15.47:3300/0 conn(0x7f86480a3ad0 0x7f864813a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:16.097 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.090+0000 7f864e74d700 1 --2- 172.21.15.189:0/1066992094 >> v2:172.21.15.189:3300/0 conn(0x7f8648068450 0x7f864813ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:16.098 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.090+0000 7f864cf4a700 1 -- 172.21.15.189:0/1066992094 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3270348821 0 0) 0x7f8648066ae0 con 0x7f8648060310 2020-11-20T12:05:16.098 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.090+0000 7f864cf4a700 1 -- 172.21.15.189:0/1066992094 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8630003240 con 0x7f8648060310 2020-11-20T12:05:16.098 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.090+0000 7f864d74b700 1 -- 172.21.15.189:0/1066992094 >> v2:172.21.15.189:3300/0 conn(0x7f8648068450 msgr2=0x7f864813ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:16.098 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.090+0000 7f864d74b700 1 --2- 172.21.15.189:0/1066992094 >> v2:172.21.15.189:3300/0 conn(0x7f8648068450 0x7f864813ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.099 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.090+0000 7f864d74b700 1 -- 172.21.15.189:0/1066992094 >> v1:172.21.15.189:6789/0 conn(0x7f8648060310 legacy=0x7f86480a9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:16.099 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.090+0000 7f864d74b700 1 -- 172.21.15.189:0/1066992094 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86480a3fa0 con 0x7f86480a3ad0 2020-11-20T12:05:16.099 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.090+0000 7f864d74b700 1 --2- 172.21.15.189:0/1066992094 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f86480a3ad0 0x7f864813a830 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f864400b9a0 tx=0x7f8644006df0).ready entity=mon.0 client_cookie=2b01113bfa215ec1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:16.102 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.090+0000 7f864cf4a700 1 -- 172.21.15.189:0/1066992094 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8644009640 con 0x7f86480a3ad0 2020-11-20T12:05:16.102 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.090+0000 7f864cf4a700 1 -- 172.21.15.189:0/1066992094 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86440097a0 con 0x7f86480a3ad0 2020-11-20T12:05:16.103 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.090+0000 7f864cf4a700 1 -- 172.21.15.189:0/1066992094 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8644009b20 con 0x7f86480a3ad0 2020-11-20T12:05:16.103 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.090+0000 7f86501b0700 1 -- 172.21.15.189:0/1066992094 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f86480a3ad0 msgr2=0x7f864813a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:16.103 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.090+0000 7f86501b0700 1 --2- 172.21.15.189:0/1066992094 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f86480a3ad0 0x7f864813a830 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f864400b9a0 tx=0x7f8644006df0).stop 2020-11-20T12:05:16.103 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.090+0000 7f86501b0700 1 -- 172.21.15.189:0/1066992094 shutdown_connections 2020-11-20T12:05:16.103 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.090+0000 7f86501b0700 1 --2- 172.21.15.189:0/1066992094 >> v2:172.21.15.189:3300/0 conn(0x7f8648068450 0x7f864813ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.104 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.090+0000 7f86501b0700 1 --2- 172.21.15.189:0/1066992094 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f86480a3ad0 0x7f864813a830 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.104 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.090+0000 7f86501b0700 1 -- 172.21.15.189:0/1066992094 >> 172.21.15.189:0/1066992094 conn(0x7f8648063dc0 msgr2=0x7f86480ad1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:16.104 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.090+0000 7f86501b0700 1 -- 172.21.15.189:0/1066992094 shutdown_connections 2020-11-20T12:05:16.105 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.090+0000 7f86501b0700 1 -- 172.21.15.189:0/1066992094 wait complete. 2020-11-20T12:05:16.105 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.094+0000 7f86501b0700 1 Processor -- start 2020-11-20T12:05:16.105 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.094+0000 7f86501b0700 1 -- start start 2020-11-20T12:05:16.105 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.094+0000 7f86501b0700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8648060310 0x7f8648135180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:16.105 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.094+0000 7f86501b0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8648068450 0x7f86480a2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:16.106 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.094+0000 7f86501b0700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f86480a3ad0 0x7f864813cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:16.106 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.094+0000 7f86501b0700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864813ad70 con 0x7f8648060310 2020-11-20T12:05:16.106 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.094+0000 7f86501b0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864813b010 con 0x7f8648068450 2020-11-20T12:05:16.106 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.094+0000 7f86501b0700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86480a32d0 con 0x7f86480a3ad0 2020-11-20T12:05:16.106 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.094+0000 7f864d74b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8648068450 0x7f86480a2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:16.107 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.094+0000 7f864d74b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8648068450 0x7f86480a2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49004/0 (socket says 172.21.15.189:49004) 2020-11-20T12:05:16.107 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.094+0000 7f864d74b700 1 -- 172.21.15.189:0/3076895774 learned_addr learned my addr 172.21.15.189:0/3076895774 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:16.107 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.094+0000 7f864df4c700 1 --2- 172.21.15.189:0/3076895774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8648060310 0x7f8648135180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:16.107 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.094+0000 7f864e74d700 1 --2- 172.21.15.189:0/3076895774 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f86480a3ad0 0x7f864813cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:16.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.094+0000 7f864d74b700 1 -- 172.21.15.189:0/3076895774 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f86480a3ad0 msgr2=0x7f864813cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:16.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.094+0000 7f864d74b700 1 --2- 172.21.15.189:0/3076895774 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f86480a3ad0 0x7f864813cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.094+0000 7f864d74b700 1 -- 172.21.15.189:0/3076895774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8648060310 msgr2=0x7f8648135180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:16.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.094+0000 7f864d74b700 1 --2- 172.21.15.189:0/3076895774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8648060310 0x7f8648135180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.094+0000 7f864d74b700 1 -- 172.21.15.189:0/3076895774 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864813cfb0 con 0x7f8648068450 2020-11-20T12:05:16.109 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.094+0000 7f864df4c700 1 --2- 172.21.15.189:0/3076895774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8648060310 0x7f8648135180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:16.109 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.094+0000 7f864e74d700 1 --2- 172.21.15.189:0/3076895774 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f86480a3ad0 0x7f864813cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:16.109 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.094+0000 7f864d74b700 1 --2- 172.21.15.189:0/3076895774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8648068450 0x7f86480a2de0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f8644006e30 tx=0x7f8644015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:16.109 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.094+0000 7f8639ffb700 1 -- 172.21.15.189:0/3076895774 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8644021070 con 0x7f8648068450 2020-11-20T12:05:16.110 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.094+0000 7f8639ffb700 1 -- 172.21.15.189:0/3076895774 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8644011e20 con 0x7f8648068450 2020-11-20T12:05:16.110 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.094+0000 7f8639ffb700 1 -- 172.21.15.189:0/3076895774 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864401d6a0 con 0x7f8648068450 2020-11-20T12:05:16.110 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.094+0000 7f86501b0700 1 -- 172.21.15.189:0/3076895774 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86480a2220 con 0x7f8648068450 2020-11-20T12:05:16.110 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.094+0000 7f86501b0700 1 -- 172.21.15.189:0/3076895774 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f864813b650 con 0x7f8648068450 2020-11-20T12:05:16.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.098+0000 7f86501b0700 1 -- 172.21.15.189:0/3076895774 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86480a23b0 con 0x7f8648068450 2020-11-20T12:05:16.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.098+0000 7f8639ffb700 1 -- 172.21.15.189:0/3076895774 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f864400f070 con 0x7f8648068450 2020-11-20T12:05:16.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.098+0000 7f8639ffb700 1 --2- 172.21.15.189:0/3076895774 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f86200557c0 0x7f8620057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:16.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.098+0000 7f8639ffb700 1 -- 172.21.15.189:0/3076895774 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f864406fe90 con 0x7f8648068450 2020-11-20T12:05:16.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.098+0000 7f864df4c700 1 --2- 172.21.15.189:0/3076895774 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f86200557c0 0x7f8620057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:16.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.098+0000 7f864df4c700 1 --2- 172.21.15.189:0/3076895774 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f86200557c0 0x7f8620057c80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f8640005920 tx=0x7f8640009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:16.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.102+0000 7f8639ffb700 1 -- 172.21.15.189:0/3076895774 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8644045d30 con 0x7f8648068450 2020-11-20T12:05:16.262 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.254+0000 7f86501b0700 1 -- 172.21.15.189:0/3076895774 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8648044f60 con 0x7f8648068450 2020-11-20T12:05:16.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.254+0000 7f8639ffb700 1 -- 172.21.15.189:0/3076895774 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+10 (secure 0 0 0) 0x7f864402c020 con 0x7f8648068450 2020-11-20T12:05:16.263 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 8 2020-11-20T12:05:16.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.258+0000 7f86501b0700 1 -- 172.21.15.189:0/3076895774 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f86200557c0 msgr2=0x7f8620057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:16.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.258+0000 7f86501b0700 1 --2- 172.21.15.189:0/3076895774 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f86200557c0 0x7f8620057c80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f8640005920 tx=0x7f8640009040).stop 2020-11-20T12:05:16.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.258+0000 7f86501b0700 1 -- 172.21.15.189:0/3076895774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8648068450 msgr2=0x7f86480a2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:16.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.258+0000 7f86501b0700 1 --2- 172.21.15.189:0/3076895774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8648068450 0x7f86480a2de0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f8644006e30 tx=0x7f8644015000).stop 2020-11-20T12:05:16.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.258+0000 7f86501b0700 1 -- 172.21.15.189:0/3076895774 shutdown_connections 2020-11-20T12:05:16.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.258+0000 7f86501b0700 1 --2- 172.21.15.189:0/3076895774 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f86200557c0 0x7f8620057c80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.258+0000 7f86501b0700 1 --2- 172.21.15.189:0/3076895774 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8648060310 0x7f8648135180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.258+0000 7f86501b0700 1 --2- 172.21.15.189:0/3076895774 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f86480a3ad0 0x7f864813cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.258+0000 7f86501b0700 1 --2- 172.21.15.189:0/3076895774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8648068450 0x7f86480a2de0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.258+0000 7f86501b0700 1 -- 172.21.15.189:0/3076895774 >> 172.21.15.189:0/3076895774 conn(0x7f8648063dc0 msgr2=0x7f864813da00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:16.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.258+0000 7f86501b0700 1 -- 172.21.15.189:0/3076895774 shutdown_connections 2020-11-20T12:05:16.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.258+0000 7f86501b0700 1 -- 172.21.15.189:0/3076895774 wait complete. 2020-11-20T12:05:16.277 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:05:16.405 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.398+0000 7f6c70f84700 1 Processor -- start 2020-11-20T12:05:16.405 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.398+0000 7f6c70f84700 1 -- start start 2020-11-20T12:05:16.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.398+0000 7f6c70f84700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6c6c0adaa0 0x7f6c6c13a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:16.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.398+0000 7f6c70f84700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c6c0a8830 con 0x7f6c6c0a80c0 2020-11-20T12:05:16.409 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.398+0000 7f6c70f84700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c6c0a8ad0 con 0x7f6c6c0a8f70 2020-11-20T12:05:16.409 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.398+0000 7f6c70f84700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c6c0ade80 con 0x7f6c6c0adaa0 2020-11-20T12:05:16.409 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.398+0000 7f6c6ad9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6c6c0a8f70 0x7f6c6c134030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50632/0 (socket says 172.21.15.189:50632) 2020-11-20T12:05:16.409 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.398+0000 7f6c6ad9d700 1 -- 172.21.15.189:0/2591806567 learned_addr learned my addr 172.21.15.189:0/2591806567 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:16.410 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.398+0000 7f6c6a59c700 1 --2- 172.21.15.189:0/2591806567 >> v2:172.21.15.47:3300/0 conn(0x7f6c6c0adaa0 0x7f6c6c13a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:16.410 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.398+0000 7f6c6959a700 1 -- 172.21.15.189:0/2591806567 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358037788 0 0) 0x7f6c6c0a8ad0 con 0x7f6c6c0a8f70 2020-11-20T12:05:16.410 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.398+0000 7f6c6959a700 1 -- 172.21.15.189:0/2591806567 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c4c003240 con 0x7f6c6c0a8f70 2020-11-20T12:05:16.410 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.398+0000 7f6c6959a700 1 -- 172.21.15.189:0/2591806567 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3021419174 0 0) 0x7f6c4c003240 con 0x7f6c6c0a8f70 2020-11-20T12:05:16.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.398+0000 7f6c6959a700 1 -- 172.21.15.189:0/2591806567 >> v2:172.21.15.47:3300/0 conn(0x7f6c6c0adaa0 msgr2=0x7f6c6c13a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:16.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c6959a700 1 --2- 172.21.15.189:0/2591806567 >> v2:172.21.15.47:3300/0 conn(0x7f6c6c0adaa0 0x7f6c6c13a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c6959a700 1 -- 172.21.15.189:0/2591806567 >> v1:172.21.15.47:6789/0 conn(0x7f6c6c0a80c0 legacy=0x7f6c6c066c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:16.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c6959a700 1 -- 172.21.15.189:0/2591806567 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c6c13aa40 con 0x7f6c6c0a8f70 2020-11-20T12:05:16.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c6959a700 1 -- 172.21.15.189:0/2591806567 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f6c64001860 con 0x7f6c6c0a8f70 2020-11-20T12:05:16.412 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c6959a700 1 -- 172.21.15.189:0/2591806567 >> v1:172.21.15.189:6789/0 conn(0x7f6c6c0a8f70 legacy=0x7f6c6c134030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:16.412 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c6959a700 1 --2- 172.21.15.189:0/2591806567 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c4c003730 0x7f6c4c003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:16.412 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c6959a700 1 --2- 172.21.15.189:0/2591806567 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c4c0044e0 0x7f6c4c004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:16.412 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c6959a700 1 --2- 172.21.15.189:0/2591806567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c4c004e50 0x7f6c4c00b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:16.412 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c6959a700 1 -- 172.21.15.189:0/2591806567 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c0ade80 con 0x7f6c4c0044e0 2020-11-20T12:05:16.413 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c6959a700 1 -- 172.21.15.189:0/2591806567 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c004260 con 0x7f6c4c004e50 2020-11-20T12:05:16.413 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c6959a700 1 -- 172.21.15.189:0/2591806567 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c00bc20 con 0x7f6c4c003730 2020-11-20T12:05:16.413 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c6ad9d700 1 --2- 172.21.15.189:0/2591806567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c4c004e50 0x7f6c4c00b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:16.413 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c6a59c700 1 --2- 172.21.15.189:0/2591806567 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c4c003730 0x7f6c4c003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:16.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c69d9b700 1 --2- 172.21.15.189:0/2591806567 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c4c0044e0 0x7f6c4c004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:16.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c6ad9d700 1 -- 172.21.15.189:0/2591806567 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c4c003730 msgr2=0x7f6c4c003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:16.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c6ad9d700 1 --2- 172.21.15.189:0/2591806567 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c4c003730 0x7f6c4c003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c6ad9d700 1 -- 172.21.15.189:0/2591806567 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c4c0044e0 msgr2=0x7f6c4c004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:16.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c6ad9d700 1 --2- 172.21.15.189:0/2591806567 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c4c0044e0 0x7f6c4c004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c6ad9d700 1 -- 172.21.15.189:0/2591806567 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c4c003240 con 0x7f6c4c004e50 2020-11-20T12:05:16.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c6ad9d700 1 -- 172.21.15.189:0/2591806567 reap_dead start 2020-11-20T12:05:16.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c6ad9d700 1 --2- 172.21.15.189:0/2591806567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c4c004e50 0x7f6c4c00b730 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f6c6400c620 tx=0x7f6c64007b50).ready entity=mon.1 client_cookie=78aaabf6b756de6b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:16.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c6959a700 1 -- 172.21.15.189:0/2591806567 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c64005b80 con 0x7f6c4c004e50 2020-11-20T12:05:16.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c6959a700 1 -- 172.21.15.189:0/2591806567 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c64005ce0 con 0x7f6c4c004e50 2020-11-20T12:05:16.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c6959a700 1 -- 172.21.15.189:0/2591806567 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c64007320 con 0x7f6c4c004e50 2020-11-20T12:05:16.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c70f84700 1 -- 172.21.15.189:0/2591806567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c4c004e50 msgr2=0x7f6c4c00b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:16.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c70f84700 1 --2- 172.21.15.189:0/2591806567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c4c004e50 0x7f6c4c00b730 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f6c6400c620 tx=0x7f6c64007b50).stop 2020-11-20T12:05:16.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c70f84700 1 -- 172.21.15.189:0/2591806567 shutdown_connections 2020-11-20T12:05:16.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c70f84700 1 --2- 172.21.15.189:0/2591806567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c4c004e50 0x7f6c4c00b730 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c70f84700 1 -- 172.21.15.189:0/2591806567 >> 172.21.15.189:0/2591806567 conn(0x7f6c6c063dc0 msgr2=0x7f6c6c0ad1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:16.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c70f84700 1 -- 172.21.15.189:0/2591806567 shutdown_connections 2020-11-20T12:05:16.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c70f84700 1 -- 172.21.15.189:0/2591806567 wait complete. 2020-11-20T12:05:16.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c70f84700 1 Processor -- start 2020-11-20T12:05:16.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c70f84700 1 -- start start 2020-11-20T12:05:16.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c70f84700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c4c004e50 0x7f6c6c0a3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:16.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c70f84700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c6c0a1910 0x7f6c6c0a6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:16.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c70f84700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c6c131bc0 0x7f6c6c132040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:16.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c70f84700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c0a3ae0 con 0x7f6c4c004e50 2020-11-20T12:05:16.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c70f84700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c0a1d90 con 0x7f6c6c0a1910 2020-11-20T12:05:16.423 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.402+0000 7f6c70f84700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c0a1530 con 0x7f6c6c131bc0 2020-11-20T12:05:16.423 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.406+0000 7f6c69d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c6c0a1910 0x7f6c6c0a6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:16.423 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.406+0000 7f6c6ad9d700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c6c131bc0 0x7f6c6c132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:16.423 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.406+0000 7f6c69d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c6c0a1910 0x7f6c6c0a6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49026/0 (socket says 172.21.15.189:49026) 2020-11-20T12:05:16.424 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.406+0000 7f6c69d9b700 1 -- 172.21.15.189:0/3846482818 learned_addr learned my addr 172.21.15.189:0/3846482818 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:16.424 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.406+0000 7f6c6a59c700 1 --2- 172.21.15.189:0/3846482818 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c4c004e50 0x7f6c6c0a3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:16.424 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.406+0000 7f6c69d9b700 1 -- 172.21.15.189:0/3846482818 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c6c131bc0 msgr2=0x7f6c6c132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:16.424 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.406+0000 7f6c69d9b700 1 --2- 172.21.15.189:0/3846482818 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c6c131bc0 0x7f6c6c132040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.425 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.406+0000 7f6c69d9b700 1 -- 172.21.15.189:0/3846482818 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c4c004e50 msgr2=0x7f6c6c0a3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:16.425 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.406+0000 7f6c69d9b700 1 --2- 172.21.15.189:0/3846482818 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c4c004e50 0x7f6c6c0a3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.425 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.406+0000 7f6c69d9b700 1 -- 172.21.15.189:0/3846482818 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c6c0a3840 con 0x7f6c6c0a1910 2020-11-20T12:05:16.425 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.406+0000 7f6c69d9b700 1 --2- 172.21.15.189:0/3846482818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c6c0a1910 0x7f6c6c0a6f70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f6c5400f980 tx=0x7f6c54007b90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:16.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.406+0000 7f6c627fc700 1 -- 172.21.15.189:0/3846482818 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c54002730 con 0x7f6c6c0a1910 2020-11-20T12:05:16.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.406+0000 7f6c627fc700 1 -- 172.21.15.189:0/3846482818 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c54002890 con 0x7f6c6c0a1910 2020-11-20T12:05:16.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.406+0000 7f6c70f84700 1 -- 172.21.15.189:0/3846482818 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c6c0a6ad0 con 0x7f6c6c0a1910 2020-11-20T12:05:16.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.406+0000 7f6c627fc700 1 -- 172.21.15.189:0/3846482818 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c54002c10 con 0x7f6c6c0a1910 2020-11-20T12:05:16.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.406+0000 7f6c70f84700 1 -- 172.21.15.189:0/3846482818 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c6c0a5e00 con 0x7f6c6c0a1910 2020-11-20T12:05:16.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.406+0000 7f6c70f84700 1 -- 172.21.15.189:0/3846482818 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c6c0a6cb0 con 0x7f6c6c0a1910 2020-11-20T12:05:16.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.410+0000 7f6c627fc700 1 -- 172.21.15.189:0/3846482818 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f6c5401b030 con 0x7f6c6c0a1910 2020-11-20T12:05:16.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.410+0000 7f6c627fc700 1 --2- 172.21.15.189:0/3846482818 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6c480557f0 0x7f6c48057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:16.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.410+0000 7f6c627fc700 1 -- 172.21.15.189:0/3846482818 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6c5406e970 con 0x7f6c6c0a1910 2020-11-20T12:05:16.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.410+0000 7f6c6a59c700 1 --2- 172.21.15.189:0/3846482818 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6c480557f0 0x7f6c48057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:16.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.410+0000 7f6c6a59c700 1 --2- 172.21.15.189:0/3846482818 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6c480557f0 0x7f6c48057cb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f6c5c006860 tx=0x7f6c5c006df0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:16.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.414+0000 7f6c627fc700 1 -- 172.21.15.189:0/3846482818 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6c540447b0 con 0x7f6c6c0a1910 2020-11-20T12:05:16.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.566+0000 7f6c70f84700 1 -- 172.21.15.189:0/3846482818 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6c6c0a6c60 con 0x7f6c6c0a1910 2020-11-20T12:05:16.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.570+0000 7f6c627fc700 1 -- 172.21.15.189:0/3846482818 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+10 (secure 0 0 0) 0x7f6c54047dd0 con 0x7f6c6c0a1910 2020-11-20T12:05:16.575 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 8 2020-11-20T12:05:16.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.570+0000 7f6c70f84700 1 -- 172.21.15.189:0/3846482818 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6c480557f0 msgr2=0x7f6c48057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:16.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.570+0000 7f6c70f84700 1 --2- 172.21.15.189:0/3846482818 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6c480557f0 0x7f6c48057cb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f6c5c006860 tx=0x7f6c5c006df0).stop 2020-11-20T12:05:16.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.570+0000 7f6c70f84700 1 -- 172.21.15.189:0/3846482818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c6c0a1910 msgr2=0x7f6c6c0a6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:16.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.570+0000 7f6c70f84700 1 --2- 172.21.15.189:0/3846482818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c6c0a1910 0x7f6c6c0a6f70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f6c5400f980 tx=0x7f6c54007b90).stop 2020-11-20T12:05:16.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.570+0000 7f6c70f84700 1 -- 172.21.15.189:0/3846482818 shutdown_connections 2020-11-20T12:05:16.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.570+0000 7f6c70f84700 1 --2- 172.21.15.189:0/3846482818 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6c480557f0 0x7f6c48057cb0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.570+0000 7f6c70f84700 1 --2- 172.21.15.189:0/3846482818 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6c4c004e50 0x7f6c6c0a3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.570+0000 7f6c70f84700 1 --2- 172.21.15.189:0/3846482818 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6c6c131bc0 0x7f6c6c132040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.570+0000 7f6c70f84700 1 --2- 172.21.15.189:0/3846482818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c6c0a1910 0x7f6c6c0a6f70 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.570+0000 7f6c70f84700 1 -- 172.21.15.189:0/3846482818 >> 172.21.15.189:0/3846482818 conn(0x7f6c6c063dc0 msgr2=0x7f6c6c0aa230 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:16.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.570+0000 7f6c70f84700 1 -- 172.21.15.189:0/3846482818 shutdown_connections 2020-11-20T12:05:16.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.570+0000 7f6c70f84700 1 -- 172.21.15.189:0/3846482818 wait complete. 2020-11-20T12:05:16.590 INFO:tasks.mds_thrash:mds thrasher using random seed: 1605873916 2020-11-20T12:05:16.591 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:05:16.709 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.702+0000 7f304ce39700 1 Processor -- start 2020-11-20T12:05:16.710 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.702+0000 7f304ce39700 1 -- start start 2020-11-20T12:05:16.711 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.702+0000 7f304ce39700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f30480612b0 0x7f30480a0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:16.711 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.702+0000 7f304ce39700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f30481376c0 0x7f3048133a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:16.712 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.702+0000 7f304ce39700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3048068020 con 0x7f3048067b50 2020-11-20T12:05:16.712 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.702+0000 7f304ce39700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30480682c0 con 0x7f30481376c0 2020-11-20T12:05:16.712 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.702+0000 7f304ce39700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3048061690 con 0x7f30480612b0 2020-11-20T12:05:16.712 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f30477fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f30480612b0 0x7f30480a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:16.712 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f30477fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f30480612b0 0x7f30480a0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49030/0 (socket says 172.21.15.189:49030) 2020-11-20T12:05:16.713 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f30477fe700 1 -- 172.21.15.189:0/1035360923 learned_addr learned my addr 172.21.15.189:0/1035360923 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:16.713 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f3047fff700 1 --2- 172.21.15.189:0/1035360923 >> v2:172.21.15.47:3300/0 conn(0x7f30481376c0 0x7f3048133a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:16.713 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f30477fe700 1 -- 172.21.15.189:0/1035360923 >> v2:172.21.15.47:3300/0 conn(0x7f30481376c0 msgr2=0x7f3048133a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:16.713 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f30477fe700 1 --2- 172.21.15.189:0/1035360923 >> v2:172.21.15.47:3300/0 conn(0x7f30481376c0 0x7f3048133a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.714 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f30477fe700 1 -- 172.21.15.189:0/1035360923 >> v1:172.21.15.47:6789/0 conn(0x7f3048067b50 legacy=0x7f3048136fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:16.714 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f30477fe700 1 -- 172.21.15.189:0/1035360923 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3048136aa0 con 0x7f30480612b0 2020-11-20T12:05:16.714 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f30477fe700 1 --2- 172.21.15.189:0/1035360923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30480612b0 0x7f30480a0960 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f30480ae330 tx=0x7f3038006eb0).ready entity=mon.1 client_cookie=5caf99732649db8f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:16.716 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f30467fc700 1 -- 172.21.15.189:0/1035360923 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3038009810 con 0x7f30480612b0 2020-11-20T12:05:16.716 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f30467fc700 1 -- 172.21.15.189:0/1035360923 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3038009970 con 0x7f30480612b0 2020-11-20T12:05:16.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f30467fc700 1 -- 172.21.15.189:0/1035360923 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3038009cf0 con 0x7f30480612b0 2020-11-20T12:05:16.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f304ce39700 1 -- 172.21.15.189:0/1035360923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30480612b0 msgr2=0x7f30480a0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:16.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f304ce39700 1 --2- 172.21.15.189:0/1035360923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30480612b0 0x7f30480a0960 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f30480ae330 tx=0x7f3038006eb0).stop 2020-11-20T12:05:16.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f304ce39700 1 -- 172.21.15.189:0/1035360923 shutdown_connections 2020-11-20T12:05:16.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f304ce39700 1 --2- 172.21.15.189:0/1035360923 >> v2:172.21.15.47:3300/0 conn(0x7f30481376c0 0x7f3048133a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.718 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f304ce39700 1 --2- 172.21.15.189:0/1035360923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30480612b0 0x7f30480a0960 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.718 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f304ce39700 1 -- 172.21.15.189:0/1035360923 >> 172.21.15.189:0/1035360923 conn(0x7f3048063210 msgr2=0x7f3048060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:16.718 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f304ce39700 1 -- 172.21.15.189:0/1035360923 shutdown_connections 2020-11-20T12:05:16.718 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f304ce39700 1 -- 172.21.15.189:0/1035360923 wait complete. 2020-11-20T12:05:16.718 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f304ce39700 1 Processor -- start 2020-11-20T12:05:16.719 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f304ce39700 1 -- start start 2020-11-20T12:05:16.719 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f304ce39700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f30480612b0 0x7f30480acaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:16.719 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f304ce39700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3048067b50 0x7f30480a67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:16.719 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f304ce39700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30481376c0 0x7f30480a4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:16.719 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f304ce39700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30480a6fc0 con 0x7f30480612b0 2020-11-20T12:05:16.720 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f304ce39700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30480a33a0 con 0x7f30481376c0 2020-11-20T12:05:16.720 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f304ce39700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30480a3640 con 0x7f3048067b50 2020-11-20T12:05:16.720 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.706+0000 7f3047fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30481376c0 0x7f30480a4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:16.720 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.710+0000 7f3047fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30481376c0 0x7f30480a4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49038/0 (socket says 172.21.15.189:49038) 2020-11-20T12:05:16.720 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.710+0000 7f3047fff700 1 -- 172.21.15.189:0/369199446 learned_addr learned my addr 172.21.15.189:0/369199446 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:16.721 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.710+0000 7f3047fff700 1 -- 172.21.15.189:0/369199446 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3048067b50 msgr2=0x7f30480a67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:16.721 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.710+0000 7f30477fe700 1 --2- 172.21.15.189:0/369199446 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f30480612b0 0x7f30480acaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:16.721 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.710+0000 7f3046ffd700 1 --2- 172.21.15.189:0/369199446 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3048067b50 0x7f30480a67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:16.721 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.710+0000 7f3047fff700 1 --2- 172.21.15.189:0/369199446 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3048067b50 0x7f30480a67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.721 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.710+0000 7f3047fff700 1 -- 172.21.15.189:0/369199446 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f30480612b0 msgr2=0x7f30480acaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:16.722 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.710+0000 7f3047fff700 1 --2- 172.21.15.189:0/369199446 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f30480612b0 0x7f30480acaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.722 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.710+0000 7f3047fff700 1 -- 172.21.15.189:0/369199446 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30480a6cc0 con 0x7f30481376c0 2020-11-20T12:05:16.722 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.710+0000 7f30477fe700 1 --2- 172.21.15.189:0/369199446 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f30480612b0 0x7f30480acaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:16.722 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.710+0000 7f3046ffd700 1 --2- 172.21.15.189:0/369199446 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3048067b50 0x7f30480a67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:16.722 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.710+0000 7f3047fff700 1 --2- 172.21.15.189:0/369199446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30481376c0 0x7f30480a4630 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f304000ec30 tx=0x7f30400091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:16.723 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.710+0000 7f30277fe700 1 -- 172.21.15.189:0/369199446 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304000ca60 con 0x7f30481376c0 2020-11-20T12:05:16.723 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.710+0000 7f30277fe700 1 -- 172.21.15.189:0/369199446 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f304000cbc0 con 0x7f30481376c0 2020-11-20T12:05:16.723 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.710+0000 7f304ce39700 1 -- 172.21.15.189:0/369199446 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30480a3ae0 con 0x7f30481376c0 2020-11-20T12:05:16.723 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.710+0000 7f30277fe700 1 -- 172.21.15.189:0/369199446 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30400123f0 con 0x7f30481376c0 2020-11-20T12:05:16.723 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.710+0000 7f304ce39700 1 -- 172.21.15.189:0/369199446 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30480a4d40 con 0x7f30481376c0 2020-11-20T12:05:16.724 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.710+0000 7f30277fe700 1 -- 172.21.15.189:0/369199446 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f3040022070 con 0x7f30481376c0 2020-11-20T12:05:16.725 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.710+0000 7f304ce39700 1 -- 172.21.15.189:0/369199446 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30480a3800 con 0x7f30481376c0 2020-11-20T12:05:16.725 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.710+0000 7f30277fe700 1 --2- 172.21.15.189:0/369199446 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f30200557c0 0x7f3020057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:16.725 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.710+0000 7f30277fe700 1 -- 172.21.15.189:0/369199446 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f304006edd0 con 0x7f30481376c0 2020-11-20T12:05:16.725 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.710+0000 7f30477fe700 1 --2- 172.21.15.189:0/369199446 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f30200557c0 0x7f3020057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:16.725 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.710+0000 7f30477fe700 1 --2- 172.21.15.189:0/369199446 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f30200557c0 0x7f3020057c80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f303800b1b0 tx=0x7f3038015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:16.726 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.718+0000 7f30277fe700 1 -- 172.21.15.189:0/369199446 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3040044c70 con 0x7f30481376c0 2020-11-20T12:05:16.875 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.866+0000 7f304ce39700 1 -- 172.21.15.189:0/369199446 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30480446d0 con 0x7f30481376c0 2020-11-20T12:05:16.876 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.870+0000 7f30277fe700 1 -- 172.21.15.189:0/369199446 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16826 (secure 0 0 0) 0x7f3040016070 con 0x7f30481376c0 2020-11-20T12:05:16.879 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:05:16.879 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":22,"fsid":"cda18d0e-64b3-41bb-964a-084069c3e648","created":"2020-11-20T12:04:37.538706+0000","modified":"2020-11-20T12:04:55.015818+0000","last_up_change":"2020-11-20T12:04:43.006631+0000","last_in_change":"2020-11-20T12:04:43.006631+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:41.395395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:48.550454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:52.101837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:53.018196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c81214e1-4f84-4aaa-8ceb-99c398b2a89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6811","nonce":11760},{"type":"v1","addr":"172.21.15.189:6814","nonce":11760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":11760},{"type":"v1","addr":"172.21.15.189:6818","nonce":11760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":11760},{"type":"v1","addr":"172.21.15.189:6828","nonce":11760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6821","nonce":11760},{"type":"v1","addr":"172.21.15.189:6824","nonce":11760}]},"public_addr":"172.21.15.189:6814/11760","cluster_addr":"172.21.15.189:6818/11760","heartbeat_back_addr":"172.21.15.189:6828/11760","heartbeat_front_addr":"172.21.15.189:6824/11760","state":["exists","up"]},{"osd":1,"uuid":"d2f3ec7d-d403-484f-b5ed-b2e6b079d63d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6805","nonce":11761},{"type":"v1","addr":"172.21.15.189:6807","nonce":11761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":11761},{"type":"v1","addr":"172.21.15.189:6812","nonce":11761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6817","nonce":11761},{"type":"v1","addr":"172.21.15.189:6820","nonce":11761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6813","nonce":11761},{"type":"v1","addr":"172.21.15.189:6815","nonce":11761}]},"public_addr":"172.21.15.189:6807/11761","cluster_addr":"172.21.15.189:6812/11761","heartbeat_back_addr":"172.21.15.189:6820/11761","heartbeat_front_addr":"172.21.15.189:6815/11761","state":["exists","up"]},{"osd":2,"uuid":"e6cf4790-000c-43eb-8844-175f04c60503","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6819","nonce":11751},{"type":"v1","addr":"172.21.15.189:6822","nonce":11751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":11751},{"type":"v1","addr":"172.21.15.189:6825","nonce":11751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":11751},{"type":"v1","addr":"172.21.15.189:6831","nonce":11751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6827","nonce":11751},{"type":"v1","addr":"172.21.15.189:6829","nonce":11751}]},"public_addr":"172.21.15.189:6822/11751","cluster_addr":"172.21.15.189:6825/11751","heartbeat_back_addr":"172.21.15.189:6831/11751","heartbeat_front_addr":"172.21.15.189:6829/11751","state":["exists","up"]},{"osd":3,"uuid":"ba49dfac-f2b9-476e-853d-42f05e21a77a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":11762},{"type":"v1","addr":"172.21.15.189:6801","nonce":11762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":11762},{"type":"v1","addr":"172.21.15.189:6803","nonce":11762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":11762},{"type":"v1","addr":"172.21.15.189:6809","nonce":11762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":11762},{"type":"v1","addr":"172.21.15.189:6806","nonce":11762}]},"public_addr":"172.21.15.189:6801/11762","cluster_addr":"172.21.15.189:6803/11762","heartbeat_back_addr":"172.21.15.189:6809/11762","heartbeat_front_addr":"172.21.15.189:6806/11762","state":["exists","up"]},{"osd":4,"uuid":"3ca7b438-38b2-4689-8b18-fb5a8ddf6aee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":12114},{"type":"v1","addr":"172.21.15.47:6825","nonce":12114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":12114},{"type":"v1","addr":"172.21.15.47:6827","nonce":12114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":12114},{"type":"v1","addr":"172.21.15.47:6831","nonce":12114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":12114},{"type":"v1","addr":"172.21.15.47:6829","nonce":12114}]},"public_addr":"172.21.15.47:6825/12114","cluster_addr":"172.21.15.47:6827/12114","heartbeat_back_addr":"172.21.15.47:6831/12114","heartbeat_front_addr":"172.21.15.47:6829/12114","state":["exists","up"]},{"osd":5,"uuid":"6a45fca7-83d3-44fd-a1d4-b01e4201a101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":12111},{"type":"v1","addr":"172.21.15.47:6801","nonce":12111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":12111},{"type":"v1","addr":"172.21.15.47:6803","nonce":12111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":12111},{"type":"v1","addr":"172.21.15.47:6807","nonce":12111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":12111},{"type":"v1","addr":"172.21.15.47:6805","nonce":12111}]},"public_addr":"172.21.15.47:6801/12111","cluster_addr":"172.21.15.47:6803/12111","heartbeat_back_addr":"172.21.15.47:6807/12111","heartbeat_front_addr":"172.21.15.47:6805/12111","state":["exists","up"]},{"osd":6,"uuid":"c4b1b9ed-61f8-4677-9755-dff59a30e447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":12113},{"type":"v1","addr":"172.21.15.47:6817","nonce":12113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":12113},{"type":"v1","addr":"172.21.15.47:6819","nonce":12113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":12113},{"type":"v1","addr":"172.21.15.47:6823","nonce":12113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":12113},{"type":"v1","addr":"172.21.15.47:6821","nonce":12113}]},"public_addr":"172.21.15.47:6817/12113","cluster_addr":"172.21.15.47:6819/12113","heartbeat_back_addr":"172.21.15.47:6823/12113","heartbeat_front_addr":"172.21.15.47:6821/12113","state":["exists","up"]},{"osd":7,"uuid":"4608e4ed-cc40-4c5c-ba8b-3779bbc62f65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":12112},{"type":"v1","addr":"172.21.15.47:6809","nonce":12112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":12112},{"type":"v1","addr":"172.21.15.47:6811","nonce":12112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":12112},{"type":"v1","addr":"172.21.15.47:6815","nonce":12112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":12112},{"type":"v1","addr":"172.21.15.47:6813","nonce":12112}]},"public_addr":"172.21.15.47:6809/12112","cluster_addr":"172.21.15.47:6811/12112","heartbeat_back_addr":"172.21.15.47:6815/12112","heartbeat_front_addr":"172.21.15.47:6813/12112","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.435265+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.472776+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.388370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.417366+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.102392+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.116529+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.167430+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.148139+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:05:16.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.870+0000 7f304ce39700 1 -- 172.21.15.189:0/369199446 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f30200557c0 msgr2=0x7f3020057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:16.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.870+0000 7f304ce39700 1 --2- 172.21.15.189:0/369199446 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f30200557c0 0x7f3020057c80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f303800b1b0 tx=0x7f3038015000).stop 2020-11-20T12:05:16.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.870+0000 7f304ce39700 1 -- 172.21.15.189:0/369199446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30481376c0 msgr2=0x7f30480a4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:16.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.870+0000 7f304ce39700 1 --2- 172.21.15.189:0/369199446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30481376c0 0x7f30480a4630 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f304000ec30 tx=0x7f30400091d0).stop 2020-11-20T12:05:16.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.870+0000 7f304ce39700 1 -- 172.21.15.189:0/369199446 shutdown_connections 2020-11-20T12:05:16.881 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.870+0000 7f304ce39700 1 --2- 172.21.15.189:0/369199446 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f30200557c0 0x7f3020057c80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.881 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.870+0000 7f304ce39700 1 --2- 172.21.15.189:0/369199446 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f30480612b0 0x7f30480acaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.881 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.870+0000 7f304ce39700 1 --2- 172.21.15.189:0/369199446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30481376c0 0x7f30480a4630 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.881 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.870+0000 7f304ce39700 1 --2- 172.21.15.189:0/369199446 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3048067b50 0x7f30480a67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:16.881 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.870+0000 7f304ce39700 1 -- 172.21.15.189:0/369199446 >> 172.21.15.189:0/369199446 conn(0x7f3048063210 msgr2=0x7f30480a9e90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.870+0000 7f304ce39700 1 -- 172.21.15.189:0/369199446 shutdown_connections 2020-11-20T12:05:16.882 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:16.870+0000 7f304ce39700 1 -- 172.21.15.189:0/369199446 wait complete. 2020-11-20T12:05:16.891 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:41.395395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:48.550454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:04:52.101837+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:04:53.018196+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:05:16.892 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:05:17.024 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.018+0000 7fa556f77700 1 Processor -- start 2020-11-20T12:05:17.025 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.018+0000 7fa556f77700 1 -- start start 2020-11-20T12:05:17.026 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.018+0000 7fa556f77700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa5500b19e0 0x7fa55013a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:17.026 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.018+0000 7fa556f77700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa550066fb0 con 0x7fa550066a30 2020-11-20T12:05:17.026 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.018+0000 7fa556f77700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa550067250 con 0x7fa550062930 2020-11-20T12:05:17.027 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.018+0000 7fa556f77700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5500666b0 con 0x7fa5500b19e0 2020-11-20T12:05:17.027 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.018+0000 7fa555f75700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa550062930 0x7fa5500b12d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50666/0 (socket says 172.21.15.189:50666) 2020-11-20T12:05:17.027 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.018+0000 7fa555f75700 1 -- 172.21.15.189:0/2134811909 learned_addr learned my addr 172.21.15.189:0/2134811909 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:17.027 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.018+0000 7fa555774700 1 --2- 172.21.15.189:0/2134811909 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa5500b19e0 0x7fa55013a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:17.027 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.018+0000 7fa554f73700 1 -- 172.21.15.189:0/2134811909 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 718769557 0 0) 0x7fa550067250 con 0x7fa550062930 2020-11-20T12:05:17.028 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.018+0000 7fa554f73700 1 -- 172.21.15.189:0/2134811909 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa538003240 con 0x7fa550062930 2020-11-20T12:05:17.032 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.018+0000 7fa554f73700 1 -- 172.21.15.189:0/2134811909 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1889671094 0 0) 0x7fa538003240 con 0x7fa550062930 2020-11-20T12:05:17.032 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.018+0000 7fa554f73700 1 -- 172.21.15.189:0/2134811909 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa5500b19e0 msgr2=0x7fa55013a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:17.032 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.018+0000 7fa554f73700 1 --2- 172.21.15.189:0/2134811909 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa5500b19e0 0x7fa55013a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.033 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.018+0000 7fa554f73700 1 -- 172.21.15.189:0/2134811909 >> v1:172.21.15.47:6789/0 conn(0x7fa550066a30 legacy=0x7fa55013ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:17.033 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.018+0000 7fa554f73700 1 -- 172.21.15.189:0/2134811909 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5500b0e30 con 0x7fa550062930 2020-11-20T12:05:17.033 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.018+0000 7fa554f73700 1 -- 172.21.15.189:0/2134811909 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7fa54c0014e0 con 0x7fa550062930 2020-11-20T12:05:17.033 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa554f73700 1 -- 172.21.15.189:0/2134811909 >> v1:172.21.15.189:6789/0 conn(0x7fa550062930 legacy=0x7fa5500b12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:17.034 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa554f73700 1 --2- 172.21.15.189:0/2134811909 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa538003730 0x7fa538003b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:17.034 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa554f73700 1 --2- 172.21.15.189:0/2134811909 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa538004070 0x7fa538004540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:17.034 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa554f73700 1 --2- 172.21.15.189:0/2134811909 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa5500b19e0 0x7fa538004ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:17.034 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa554f73700 1 -- 172.21.15.189:0/2134811909 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5380050b0 con 0x7fa538003730 2020-11-20T12:05:17.034 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa554f73700 1 -- 172.21.15.189:0/2134811909 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa538005220 con 0x7fa538004070 2020-11-20T12:05:17.035 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa554f73700 1 -- 172.21.15.189:0/2134811909 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa53800b730 con 0x7fa5500b19e0 2020-11-20T12:05:17.035 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa555774700 1 --2- 172.21.15.189:0/2134811909 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa538004070 0x7fa538004540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:17.035 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa555f75700 1 --2- 172.21.15.189:0/2134811909 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa538003730 0x7fa538003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:17.035 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa556776700 1 --2- 172.21.15.189:0/2134811909 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa5500b19e0 0x7fa538004ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:17.035 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa555774700 1 -- 172.21.15.189:0/2134811909 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa5500b19e0 msgr2=0x7fa538004ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:17.036 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa555774700 1 --2- 172.21.15.189:0/2134811909 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa5500b19e0 0x7fa538004ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.036 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa555774700 1 -- 172.21.15.189:0/2134811909 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa538003730 msgr2=0x7fa538003b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:17.036 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa555774700 1 --2- 172.21.15.189:0/2134811909 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa538003730 0x7fa538003b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.036 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa555774700 1 -- 172.21.15.189:0/2134811909 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa538003240 con 0x7fa538004070 2020-11-20T12:05:17.036 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa555f75700 1 --2- 172.21.15.189:0/2134811909 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa538003730 0x7fa538003b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:17.037 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa555774700 1 --2- 172.21.15.189:0/2134811909 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa538004070 0x7fa538004540 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fa540006fd0 tx=0x7fa540006df0).ready entity=mon.1 client_cookie=460486250df5041d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:17.037 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa554f73700 1 -- 172.21.15.189:0/2134811909 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54000b310 con 0x7fa538004070 2020-11-20T12:05:17.037 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa554f73700 1 -- 172.21.15.189:0/2134811909 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa54000eae0 con 0x7fa538004070 2020-11-20T12:05:17.037 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa554f73700 1 -- 172.21.15.189:0/2134811909 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54000ee60 con 0x7fa538004070 2020-11-20T12:05:17.037 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa556f77700 1 -- 172.21.15.189:0/2134811909 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa538004070 msgr2=0x7fa538004540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:17.038 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa556f77700 1 --2- 172.21.15.189:0/2134811909 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa538004070 0x7fa538004540 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fa540006fd0 tx=0x7fa540006df0).stop 2020-11-20T12:05:17.038 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa556776700 1 -- 172.21.15.189:0/2134811909 reap_dead start 2020-11-20T12:05:17.038 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa556f77700 1 -- 172.21.15.189:0/2134811909 shutdown_connections 2020-11-20T12:05:17.038 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa556f77700 1 -- 172.21.15.189:0/2134811909 >> 172.21.15.189:0/2134811909 conn(0x7fa550065080 msgr2=0x7fa550062080 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:17.038 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa556f77700 1 -- 172.21.15.189:0/2134811909 shutdown_connections 2020-11-20T12:05:17.039 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa556f77700 1 -- 172.21.15.189:0/2134811909 wait complete. 2020-11-20T12:05:17.039 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa556f77700 1 Processor -- start 2020-11-20T12:05:17.039 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa556f77700 1 -- start start 2020-11-20T12:05:17.039 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa556f77700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa550133fc0 0x7fa550139d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:17.040 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa556f77700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa550138930 0x7fa550138db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:17.040 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa556f77700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa55014d2d0 0x7fa55014d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:17.040 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa556f77700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55013a1f0 con 0x7fa550138930 2020-11-20T12:05:17.040 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa556f77700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5501392a0 con 0x7fa55014d2d0 2020-11-20T12:05:17.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa556f77700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa55013d000 con 0x7fa550133fc0 2020-11-20T12:05:17.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa556776700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa55014d2d0 0x7fa55014d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:17.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa556776700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa55014d2d0 0x7fa55014d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49058/0 (socket says 172.21.15.189:49058) 2020-11-20T12:05:17.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa556776700 1 -- 172.21.15.189:0/3896669875 learned_addr learned my addr 172.21.15.189:0/3896669875 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:17.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa555f75700 1 --2- 172.21.15.189:0/3896669875 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa550133fc0 0x7fa550139d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:17.042 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.022+0000 7fa555774700 1 --2- 172.21.15.189:0/3896669875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa550138930 0x7fa550138db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:17.042 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.026+0000 7fa556776700 1 -- 172.21.15.189:0/3896669875 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa550133fc0 msgr2=0x7fa550139d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:17.042 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.026+0000 7fa556776700 1 --2- 172.21.15.189:0/3896669875 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa550133fc0 0x7fa550139d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.042 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.026+0000 7fa556776700 1 -- 172.21.15.189:0/3896669875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa550138930 msgr2=0x7fa550138db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:17.042 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.026+0000 7fa556776700 1 --2- 172.21.15.189:0/3896669875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa550138930 0x7fa550138db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.026+0000 7fa556776700 1 -- 172.21.15.189:0/3896669875 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa55013d2a0 con 0x7fa55014d2d0 2020-11-20T12:05:17.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.026+0000 7fa556776700 1 --2- 172.21.15.189:0/3896669875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa55014d2d0 0x7fa55014d710 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fa55006de20 tx=0x7fa548009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:17.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.026+0000 7fa53dffb700 1 -- 172.21.15.189:0/3896669875 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa548003320 con 0x7fa55014d2d0 2020-11-20T12:05:17.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.026+0000 7fa53dffb700 1 -- 172.21.15.189:0/3896669875 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa548003480 con 0x7fa55014d2d0 2020-11-20T12:05:17.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.026+0000 7fa556f77700 1 -- 172.21.15.189:0/3896669875 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa55013d540 con 0x7fa55014d2d0 2020-11-20T12:05:17.044 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.026+0000 7fa53dffb700 1 -- 172.21.15.189:0/3896669875 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa548001810 con 0x7fa55014d2d0 2020-11-20T12:05:17.044 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.026+0000 7fa556f77700 1 -- 172.21.15.189:0/3896669875 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5501343a0 con 0x7fa55014d2d0 2020-11-20T12:05:17.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.026+0000 7fa53dffb700 1 -- 172.21.15.189:0/3896669875 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fa548001bc0 con 0x7fa55014d2d0 2020-11-20T12:05:17.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.026+0000 7fa556f77700 1 -- 172.21.15.189:0/3896669875 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5500ab1c0 con 0x7fa55014d2d0 2020-11-20T12:05:17.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.026+0000 7fa53dffb700 1 --2- 172.21.15.189:0/3896669875 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa5280557c0 0x7fa528057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:17.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.026+0000 7fa53dffb700 1 -- 172.21.15.189:0/3896669875 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa54806e140 con 0x7fa55014d2d0 2020-11-20T12:05:17.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.026+0000 7fa555f75700 1 --2- 172.21.15.189:0/3896669875 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa5280557c0 0x7fa528057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:17.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.026+0000 7fa555f75700 1 --2- 172.21.15.189:0/3896669875 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa5280557c0 0x7fa528057c80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fa54c000dd0 tx=0x7fa54c000cb0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:17.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.030+0000 7fa53dffb700 1 -- 172.21.15.189:0/3896669875 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa548043fe0 con 0x7fa55014d2d0 2020-11-20T12:05:17.183 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.178+0000 7fa556f77700 1 -- 172.21.15.189:0/3896669875 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fa5500446d0 con 0x7fa55014d2d0 2020-11-20T12:05:17.184 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.178+0000 7fa53dffb700 1 -- 172.21.15.189:0/3896669875 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v22) v1 ==== 110+0+10 (secure 0 0 0) 0x7fa548047600 con 0x7fa55014d2d0 2020-11-20T12:05:17.184 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2020-11-20T12:05:17.185 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.178+0000 7fa556f77700 1 -- 172.21.15.189:0/3896669875 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa5280557c0 msgr2=0x7fa528057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:17.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.178+0000 7fa556f77700 1 --2- 172.21.15.189:0/3896669875 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa5280557c0 0x7fa528057c80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fa54c000dd0 tx=0x7fa54c000cb0).stop 2020-11-20T12:05:17.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.178+0000 7fa556f77700 1 -- 172.21.15.189:0/3896669875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa55014d2d0 msgr2=0x7fa55014d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:17.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.178+0000 7fa556f77700 1 --2- 172.21.15.189:0/3896669875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa55014d2d0 0x7fa55014d710 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fa55006de20 tx=0x7fa548009040).stop 2020-11-20T12:05:17.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.178+0000 7fa556f77700 1 -- 172.21.15.189:0/3896669875 shutdown_connections 2020-11-20T12:05:17.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.178+0000 7fa556f77700 1 --2- 172.21.15.189:0/3896669875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa55014d2d0 0x7fa55014d710 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.178+0000 7fa556f77700 1 --2- 172.21.15.189:0/3896669875 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa550133fc0 0x7fa550139d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.178+0000 7fa556f77700 1 --2- 172.21.15.189:0/3896669875 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa5280557c0 0x7fa528057c80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.178+0000 7fa556f77700 1 --2- 172.21.15.189:0/3896669875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa550138930 0x7fa550138db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.178+0000 7fa556f77700 1 -- 172.21.15.189:0/3896669875 >> 172.21.15.189:0/3896669875 conn(0x7fa550065080 msgr2=0x7fa5500acee0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:17.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.178+0000 7fa556f77700 1 -- 172.21.15.189:0/3896669875 shutdown_connections 2020-11-20T12:05:17.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.178+0000 7fa556f77700 1 -- 172.21.15.189:0/3896669875 wait complete. 2020-11-20T12:05:17.198 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:05:17.322 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.314+0000 7fe152142700 1 Processor -- start 2020-11-20T12:05:17.322 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.314+0000 7fe152142700 1 -- start start 2020-11-20T12:05:17.324 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.314+0000 7fe152142700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe14c060310 0x7fe14c13a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:17.324 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.314+0000 7fe152142700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe14c066ba0 0x7fe14c0a0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:17.324 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.314+0000 7fe152142700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe14c0a8fc0 con 0x7fe14c0a0a10 2020-11-20T12:05:17.324 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.314+0000 7fe152142700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe14c0674b0 con 0x7fe14c066ba0 2020-11-20T12:05:17.324 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.314+0000 7fe152142700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe14c0a9260 con 0x7fe14c060310 2020-11-20T12:05:17.325 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe151941700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe14c0a0a10 0x7fe14c137360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50690/0 (socket says 172.21.15.189:50690) 2020-11-20T12:05:17.325 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe15093f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe14c066ba0 0x7fe14c0a0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:17.325 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe151941700 1 -- 172.21.15.189:0/1048292306 learned_addr learned my addr 172.21.15.189:0/1048292306 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:17.325 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe151140700 1 --2- 172.21.15.189:0/1048292306 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe14c060310 0x7fe14c13a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:17.325 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe13ffff700 1 -- 172.21.15.189:0/1048292306 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1526157909 0 0) 0x7fe14c0a8fc0 con 0x7fe14c0a0a10 2020-11-20T12:05:17.326 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe13ffff700 1 -- 172.21.15.189:0/1048292306 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe134003240 con 0x7fe14c0a0a10 2020-11-20T12:05:17.326 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe15093f700 1 -- 172.21.15.189:0/1048292306 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe14c060310 msgr2=0x7fe14c13a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:17.326 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe15093f700 1 --2- 172.21.15.189:0/1048292306 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe14c060310 0x7fe14c13a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.326 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe15093f700 1 -- 172.21.15.189:0/1048292306 >> v1:172.21.15.189:6789/0 conn(0x7fe14c0a0a10 legacy=0x7fe14c137360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:17.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe15093f700 1 -- 172.21.15.189:0/1048292306 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe14c068460 con 0x7fe14c066ba0 2020-11-20T12:05:17.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe151140700 1 --2- 172.21.15.189:0/1048292306 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe14c060310 0x7fe14c13a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:17.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe15093f700 1 --2- 172.21.15.189:0/1048292306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe14c066ba0 0x7fe14c0a0520 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fe14000b9a0 tx=0x7fe140006df0).ready entity=mon.1 client_cookie=5544d90fe0fb4cb0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:17.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe13ffff700 1 -- 172.21.15.189:0/1048292306 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe140009840 con 0x7fe14c066ba0 2020-11-20T12:05:17.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe13ffff700 1 -- 172.21.15.189:0/1048292306 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1400099a0 con 0x7fe14c066ba0 2020-11-20T12:05:17.328 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe13ffff700 1 -- 172.21.15.189:0/1048292306 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe140009d20 con 0x7fe14c066ba0 2020-11-20T12:05:17.331 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe152142700 1 -- 172.21.15.189:0/1048292306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe14c066ba0 msgr2=0x7fe14c0a0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:17.331 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe152142700 1 --2- 172.21.15.189:0/1048292306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe14c066ba0 0x7fe14c0a0520 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fe14000b9a0 tx=0x7fe140006df0).stop 2020-11-20T12:05:17.331 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe152142700 1 -- 172.21.15.189:0/1048292306 shutdown_connections 2020-11-20T12:05:17.331 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe152142700 1 --2- 172.21.15.189:0/1048292306 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe14c060310 0x7fe14c13a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe152142700 1 --2- 172.21.15.189:0/1048292306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe14c066ba0 0x7fe14c0a0520 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe152142700 1 -- 172.21.15.189:0/1048292306 >> 172.21.15.189:0/1048292306 conn(0x7fe14c065910 msgr2=0x7fe14c060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:17.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe152142700 1 -- 172.21.15.189:0/1048292306 shutdown_connections 2020-11-20T12:05:17.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe152142700 1 -- 172.21.15.189:0/1048292306 wait complete. 2020-11-20T12:05:17.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe152142700 1 Processor -- start 2020-11-20T12:05:17.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe152142700 1 -- start start 2020-11-20T12:05:17.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe152142700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe14c060310 0x7fe14c137ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:17.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe152142700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe14c066ba0 0x7fe14c0a2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:17.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe152142700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe14c0a0a10 0x7fe14c0acd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:17.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe152142700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14c0a34e0 con 0x7fe14c0a0a10 2020-11-20T12:05:17.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe152142700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14c0ad510 con 0x7fe14c060310 2020-11-20T12:05:17.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe152142700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe14c0ad820 con 0x7fe14c066ba0 2020-11-20T12:05:17.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe151140700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe14c060310 0x7fe14c137ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:17.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe151140700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe14c060310 0x7fe14c137ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49070/0 (socket says 172.21.15.189:49070) 2020-11-20T12:05:17.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe151140700 1 -- 172.21.15.189:0/2878631197 learned_addr learned my addr 172.21.15.189:0/2878631197 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:17.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe15093f700 1 --2- 172.21.15.189:0/2878631197 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe14c066ba0 0x7fe14c0a2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:17.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe151941700 1 --2- 172.21.15.189:0/2878631197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe14c0a0a10 0x7fe14c0acd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:17.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe151140700 1 -- 172.21.15.189:0/2878631197 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe14c066ba0 msgr2=0x7fe14c0a2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:17.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe151140700 1 --2- 172.21.15.189:0/2878631197 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe14c066ba0 0x7fe14c0a2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe151140700 1 -- 172.21.15.189:0/2878631197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe14c0a0a10 msgr2=0x7fe14c0acd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:17.336 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe151140700 1 --2- 172.21.15.189:0/2878631197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe14c0a0a10 0x7fe14c0acd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.336 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe151140700 1 -- 172.21.15.189:0/2878631197 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe14c0adac0 con 0x7fe14c060310 2020-11-20T12:05:17.336 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.318+0000 7fe151941700 1 --2- 172.21.15.189:0/2878631197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe14c0a0a10 0x7fe14c0acd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:17.336 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.322+0000 7fe151140700 1 --2- 172.21.15.189:0/2878631197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe14c060310 0x7fe14c137ce0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fe14800db60 tx=0x7fe148008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:17.336 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.322+0000 7fe13d7fa700 1 -- 172.21.15.189:0/2878631197 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe148009850 con 0x7fe14c060310 2020-11-20T12:05:17.337 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.322+0000 7fe13d7fa700 1 -- 172.21.15.189:0/2878631197 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1480099b0 con 0x7fe14c060310 2020-11-20T12:05:17.337 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.322+0000 7fe152142700 1 -- 172.21.15.189:0/2878631197 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe14c0ad210 con 0x7fe14c060310 2020-11-20T12:05:17.337 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.322+0000 7fe13d7fa700 1 -- 172.21.15.189:0/2878631197 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe148009d50 con 0x7fe14c060310 2020-11-20T12:05:17.337 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.322+0000 7fe152142700 1 -- 172.21.15.189:0/2878631197 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe14c0adf60 con 0x7fe14c060310 2020-11-20T12:05:17.337 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.322+0000 7fe152142700 1 -- 172.21.15.189:0/2878631197 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe14c1381d0 con 0x7fe14c060310 2020-11-20T12:05:17.338 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.322+0000 7fe13d7fa700 1 -- 172.21.15.189:0/2878631197 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fe148013690 con 0x7fe14c060310 2020-11-20T12:05:17.338 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.322+0000 7fe13d7fa700 1 --2- 172.21.15.189:0/2878631197 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe1240557c0 0x7fe124057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:17.338 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.322+0000 7fe13d7fa700 1 -- 172.21.15.189:0/2878631197 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe14806eaf0 con 0x7fe14c060310 2020-11-20T12:05:17.338 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.322+0000 7fe15093f700 1 --2- 172.21.15.189:0/2878631197 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe1240557c0 0x7fe124057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:17.339 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.322+0000 7fe15093f700 1 --2- 172.21.15.189:0/2878631197 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe1240557c0 0x7fe124057c80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fe14000b9a0 tx=0x7fe140015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:17.339 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.330+0000 7fe13d7fa700 1 -- 172.21.15.189:0/2878631197 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe148044930 con 0x7fe14c060310 2020-11-20T12:05:17.489 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.482+0000 7fe152142700 1 -- 172.21.15.189:0/2878631197 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fe14c0a4420 con 0x7fe14c060310 2020-11-20T12:05:17.490 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.482+0000 7fe13d7fa700 1 -- 172.21.15.189:0/2878631197 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v22) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe148047f50 con 0x7fe14c060310 2020-11-20T12:05:17.490 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 8 2020-11-20T12:05:17.493 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.486+0000 7fe152142700 1 -- 172.21.15.189:0/2878631197 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe1240557c0 msgr2=0x7fe124057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:17.493 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.486+0000 7fe152142700 1 --2- 172.21.15.189:0/2878631197 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe1240557c0 0x7fe124057c80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fe14000b9a0 tx=0x7fe140015000).stop 2020-11-20T12:05:17.493 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.486+0000 7fe152142700 1 -- 172.21.15.189:0/2878631197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe14c060310 msgr2=0x7fe14c137ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:17.494 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.486+0000 7fe152142700 1 --2- 172.21.15.189:0/2878631197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe14c060310 0x7fe14c137ce0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fe14800db60 tx=0x7fe148008040).stop 2020-11-20T12:05:17.494 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.486+0000 7fe152142700 1 -- 172.21.15.189:0/2878631197 shutdown_connections 2020-11-20T12:05:17.494 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.486+0000 7fe152142700 1 --2- 172.21.15.189:0/2878631197 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe1240557c0 0x7fe124057c80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.494 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.486+0000 7fe152142700 1 --2- 172.21.15.189:0/2878631197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe14c0a0a10 0x7fe14c0acd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.495 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.486+0000 7fe152142700 1 --2- 172.21.15.189:0/2878631197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe14c060310 0x7fe14c137ce0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.495 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.486+0000 7fe152142700 1 --2- 172.21.15.189:0/2878631197 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe14c066ba0 0x7fe14c0a2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.495 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.486+0000 7fe152142700 1 -- 172.21.15.189:0/2878631197 >> 172.21.15.189:0/2878631197 conn(0x7fe14c065910 msgr2=0x7fe14c13a340 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:17.495 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.486+0000 7fe152142700 1 -- 172.21.15.189:0/2878631197 shutdown_connections 2020-11-20T12:05:17.495 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.486+0000 7fe152142700 1 -- 172.21.15.189:0/2878631197 wait complete. 2020-11-20T12:05:17.510 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:05:17.630 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.622+0000 7fa451c5b700 1 Processor -- start 2020-11-20T12:05:17.631 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.622+0000 7fa451c5b700 1 -- start start 2020-11-20T12:05:17.632 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.626+0000 7fa451c5b700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa44c066a30 0x7fa44c13af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:17.632 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.626+0000 7fa451c5b700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa44c066fb0 con 0x7fa44c0b19e0 2020-11-20T12:05:17.632 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.626+0000 7fa451c5b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa44c067250 con 0x7fa44c062930 2020-11-20T12:05:17.633 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.626+0000 7fa451c5b700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa44c0666b0 con 0x7fa44c066a30 2020-11-20T12:05:17.633 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.626+0000 7fa450c59700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa44c062930 0x7fa44c0b12d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50700/0 (socket says 172.21.15.189:50700) 2020-11-20T12:05:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.626+0000 7fa450c59700 1 -- 172.21.15.189:0/3185117022 learned_addr learned my addr 172.21.15.189:0/3185117022 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:17.634 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.626+0000 7fa45145a700 1 --2- 172.21.15.189:0/3185117022 >> v2:172.21.15.47:3300/0 conn(0x7fa44c066a30 0x7fa44c13af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:17.636 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.626+0000 7fa44b7fe700 1 -- 172.21.15.189:0/3185117022 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1380083063 0 0) 0x7fa44c067250 con 0x7fa44c062930 2020-11-20T12:05:17.636 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.626+0000 7fa44b7fe700 1 -- 172.21.15.189:0/3185117022 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa42c003240 con 0x7fa44c062930 2020-11-20T12:05:17.636 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.626+0000 7fa45145a700 1 -- 172.21.15.189:0/3185117022 >> v1:172.21.15.189:6789/0 conn(0x7fa44c062930 legacy=0x7fa44c0b12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:17.636 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.626+0000 7fa45145a700 1 -- 172.21.15.189:0/3185117022 >> v1:172.21.15.47:6789/0 conn(0x7fa44c0b19e0 legacy=0x7fa44c13a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:17.637 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.626+0000 7fa45145a700 1 -- 172.21.15.189:0/3185117022 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44c0b0e30 con 0x7fa44c066a30 2020-11-20T12:05:17.637 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.626+0000 7fa45145a700 1 --2- 172.21.15.189:0/3185117022 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa44c066a30 0x7fa44c13af00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fa444009c90 tx=0x7fa4440091d0).ready entity=mon.0 client_cookie=794bd8b75fd45035 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:17.637 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.626+0000 7fa44b7fe700 1 -- 172.21.15.189:0/3185117022 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44400ba70 con 0x7fa44c066a30 2020-11-20T12:05:17.637 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.626+0000 7fa44b7fe700 1 -- 172.21.15.189:0/3185117022 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa44400bbd0 con 0x7fa44c066a30 2020-11-20T12:05:17.637 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.626+0000 7fa44b7fe700 1 -- 172.21.15.189:0/3185117022 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa444011400 con 0x7fa44c066a30 2020-11-20T12:05:17.638 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.626+0000 7fa451c5b700 1 -- 172.21.15.189:0/3185117022 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa44c066a30 msgr2=0x7fa44c13af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:17.638 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.626+0000 7fa451c5b700 1 --2- 172.21.15.189:0/3185117022 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa44c066a30 0x7fa44c13af00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fa444009c90 tx=0x7fa4440091d0).stop 2020-11-20T12:05:17.638 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.626+0000 7fa451c5b700 1 -- 172.21.15.189:0/3185117022 shutdown_connections 2020-11-20T12:05:17.638 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.626+0000 7fa451c5b700 1 --2- 172.21.15.189:0/3185117022 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa44c066a30 0x7fa44c13af00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.639 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.626+0000 7fa451c5b700 1 -- 172.21.15.189:0/3185117022 >> 172.21.15.189:0/3185117022 conn(0x7fa44c065080 msgr2=0x7fa44c062080 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:17.639 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.626+0000 7fa451c5b700 1 -- 172.21.15.189:0/3185117022 shutdown_connections 2020-11-20T12:05:17.639 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.626+0000 7fa451c5b700 1 -- 172.21.15.189:0/3185117022 wait complete. 2020-11-20T12:05:17.639 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.630+0000 7fa451c5b700 1 Processor -- start 2020-11-20T12:05:17.639 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.630+0000 7fa451c5b700 1 -- start start 2020-11-20T12:05:17.642 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.630+0000 7fa451c5b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa44c062930 0x7fa44c13d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:17.642 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.630+0000 7fa451c5b700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa44c066a30 0x7fa44c0a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:17.642 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.630+0000 7fa451c5b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa44c0b19e0 0x7fa44c14d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:17.643 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.630+0000 7fa451c5b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44c13c0f0 con 0x7fa44c0b19e0 2020-11-20T12:05:17.643 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.630+0000 7fa451c5b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44c13c320 con 0x7fa44c062930 2020-11-20T12:05:17.643 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.630+0000 7fa451c5b700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa44c13c5c0 con 0x7fa44c066a30 2020-11-20T12:05:17.643 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.630+0000 7fa450c59700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa44c062930 0x7fa44c13d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:17.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.630+0000 7fa450c59700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa44c062930 0x7fa44c13d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49086/0 (socket says 172.21.15.189:49086) 2020-11-20T12:05:17.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.630+0000 7fa450c59700 1 -- 172.21.15.189:0/747233008 learned_addr learned my addr 172.21.15.189:0/747233008 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:17.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.630+0000 7fa45145a700 1 --2- 172.21.15.189:0/747233008 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa44c0b19e0 0x7fa44c14d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:17.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.630+0000 7fa44bfff700 1 --2- 172.21.15.189:0/747233008 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa44c066a30 0x7fa44c0a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:17.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.630+0000 7fa450c59700 1 -- 172.21.15.189:0/747233008 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa44c066a30 msgr2=0x7fa44c0a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:17.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.630+0000 7fa450c59700 1 --2- 172.21.15.189:0/747233008 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa44c066a30 0x7fa44c0a0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.630+0000 7fa450c59700 1 -- 172.21.15.189:0/747233008 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa44c0b19e0 msgr2=0x7fa44c14d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:17.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.630+0000 7fa450c59700 1 --2- 172.21.15.189:0/747233008 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa44c0b19e0 0x7fa44c14d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.630+0000 7fa450c59700 1 -- 172.21.15.189:0/747233008 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44c13c860 con 0x7fa44c062930 2020-11-20T12:05:17.646 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.630+0000 7fa450c59700 1 --2- 172.21.15.189:0/747233008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa44c062930 0x7fa44c13d870 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fa43400c6f0 tx=0x7fa434007ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:17.646 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.630+0000 7fa448ff9700 1 -- 172.21.15.189:0/747233008 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4340072e0 con 0x7fa44c062930 2020-11-20T12:05:17.646 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.630+0000 7fa448ff9700 1 -- 172.21.15.189:0/747233008 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa434007440 con 0x7fa44c062930 2020-11-20T12:05:17.646 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.630+0000 7fa451c5b700 1 -- 172.21.15.189:0/747233008 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa44c13cd00 con 0x7fa44c062930 2020-11-20T12:05:17.647 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.630+0000 7fa448ff9700 1 -- 172.21.15.189:0/747233008 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa434005800 con 0x7fa44c062930 2020-11-20T12:05:17.647 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.630+0000 7fa451c5b700 1 -- 172.21.15.189:0/747233008 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa44c0a7ae0 con 0x7fa44c062930 2020-11-20T12:05:17.647 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.634+0000 7fa451c5b700 1 -- 172.21.15.189:0/747233008 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa44c13c9f0 con 0x7fa44c062930 2020-11-20T12:05:17.647 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.634+0000 7fa448ff9700 1 -- 172.21.15.189:0/747233008 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fa434010070 con 0x7fa44c062930 2020-11-20T12:05:17.647 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.634+0000 7fa448ff9700 1 --2- 172.21.15.189:0/747233008 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa4280557c0 0x7fa428057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:17.648 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.634+0000 7fa448ff9700 1 -- 172.21.15.189:0/747233008 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa43406deb0 con 0x7fa44c062930 2020-11-20T12:05:17.648 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.634+0000 7fa44bfff700 1 --2- 172.21.15.189:0/747233008 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa4280557c0 0x7fa428057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:17.648 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.634+0000 7fa44bfff700 1 --2- 172.21.15.189:0/747233008 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa4280557c0 0x7fa428057c80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fa44c13d810 tx=0x7fa43c007b70).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:17.649 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.642+0000 7fa448ff9700 1 -- 172.21.15.189:0/747233008 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa434043dd0 con 0x7fa44c062930 2020-11-20T12:05:17.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.794+0000 7fa451c5b700 1 -- 172.21.15.189:0/747233008 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa44c0b0ac0 con 0x7fa44c062930 2020-11-20T12:05:17.801 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.794+0000 7fa448ff9700 1 -- 172.21.15.189:0/747233008 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+10 (secure 0 0 0) 0x7fa4340473f0 con 0x7fa44c062930 2020-11-20T12:05:17.801 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 8 2020-11-20T12:05:17.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.794+0000 7fa451c5b700 1 -- 172.21.15.189:0/747233008 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa4280557c0 msgr2=0x7fa428057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:17.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.794+0000 7fa451c5b700 1 --2- 172.21.15.189:0/747233008 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa4280557c0 0x7fa428057c80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fa44c13d810 tx=0x7fa43c007b70).stop 2020-11-20T12:05:17.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.794+0000 7fa451c5b700 1 -- 172.21.15.189:0/747233008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa44c062930 msgr2=0x7fa44c13d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:17.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.794+0000 7fa451c5b700 1 --2- 172.21.15.189:0/747233008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa44c062930 0x7fa44c13d870 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fa43400c6f0 tx=0x7fa434007ad0).stop 2020-11-20T12:05:17.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.794+0000 7fa451c5b700 1 -- 172.21.15.189:0/747233008 shutdown_connections 2020-11-20T12:05:17.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.794+0000 7fa451c5b700 1 --2- 172.21.15.189:0/747233008 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa4280557c0 0x7fa428057c80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.794+0000 7fa451c5b700 1 --2- 172.21.15.189:0/747233008 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa44c0b19e0 0x7fa44c14d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.794+0000 7fa451c5b700 1 --2- 172.21.15.189:0/747233008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa44c062930 0x7fa44c13d870 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.794+0000 7fa451c5b700 1 --2- 172.21.15.189:0/747233008 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa44c066a30 0x7fa44c0a0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.794+0000 7fa451c5b700 1 -- 172.21.15.189:0/747233008 >> 172.21.15.189:0/747233008 conn(0x7fa44c065080 msgr2=0x7fa44c0b11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:17.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.794+0000 7fa451c5b700 1 -- 172.21.15.189:0/747233008 shutdown_connections 2020-11-20T12:05:17.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.794+0000 7fa451c5b700 1 -- 172.21.15.189:0/747233008 wait complete. 2020-11-20T12:05:17.815 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:05:17.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.934+0000 7f8d919be700 1 Processor -- start 2020-11-20T12:05:17.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.934+0000 7f8d919be700 1 -- start start 2020-11-20T12:05:17.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.938+0000 7f8d919be700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8d8c0a8a60 0x7f8d8c067400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:17.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.938+0000 7f8d919be700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8d8c13e460 0x7f8d8c137640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:17.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.938+0000 7f8d919be700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d8c066ad0 con 0x7f8d8c0678f0 2020-11-20T12:05:17.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.938+0000 7f8d919be700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d8c065f70 con 0x7f8d8c13e460 2020-11-20T12:05:17.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.938+0000 7f8d919be700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8c066d70 con 0x7f8d8c0a8a60 2020-11-20T12:05:17.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.938+0000 7f8d911bd700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8d8c13e460 0x7f8d8c137640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:17.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.938+0000 7f8d909bc700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8d8c0a8a60 0x7f8d8c067400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:17.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.938+0000 7f8d911bd700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8d8c13e460 0x7f8d8c137640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.189:55316/0 (socket says 172.21.15.189:55316) 2020-11-20T12:05:17.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.938+0000 7f8d911bd700 1 -- 172.21.15.189:0/2060349938 learned_addr learned my addr 172.21.15.189:0/2060349938 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:17.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.938+0000 7f8d8bfff700 1 --1- 172.21.15.189:0/2060349938 >> v1:172.21.15.47:6789/0 conn(0x7f8d8c0678f0 0x7f8d8c13dd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.189:45382/0 (socket says 172.21.15.189:45382) 2020-11-20T12:05:17.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.938+0000 7f8d911bd700 1 -- 172.21.15.189:0/2060349938 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8d8c0a8a60 msgr2=0x7f8d8c067400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:17.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.938+0000 7f8d911bd700 1 --2- 172.21.15.189:0/2060349938 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8d8c0a8a60 0x7f8d8c067400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.938+0000 7f8d911bd700 1 -- 172.21.15.189:0/2060349938 >> v1:172.21.15.47:6789/0 conn(0x7f8d8c0678f0 legacy=0x7f8d8c13dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:17.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.938+0000 7f8d911bd700 1 -- 172.21.15.189:0/2060349938 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8c067010 con 0x7f8d8c13e460 2020-11-20T12:05:17.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.938+0000 7f8d911bd700 1 --2- 172.21.15.189:0/2060349938 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d8c13e460 0x7f8d8c137640 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f8d84009d50 tx=0x7f8d84009290).ready entity=mon.0 client_cookie=fabd0e6d390c077c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:17.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.938+0000 7f8d8b7fe700 1 -- 172.21.15.189:0/2060349938 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8400bbc0 con 0x7f8d8c13e460 2020-11-20T12:05:17.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.938+0000 7f8d8b7fe700 1 -- 172.21.15.189:0/2060349938 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8400bd20 con 0x7f8d8c13e460 2020-11-20T12:05:17.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.938+0000 7f8d8b7fe700 1 -- 172.21.15.189:0/2060349938 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d84011610 con 0x7f8d8c13e460 2020-11-20T12:05:17.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.938+0000 7f8d919be700 1 -- 172.21.15.189:0/2060349938 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d8c13e460 msgr2=0x7f8d8c137640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:17.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.938+0000 7f8d919be700 1 --2- 172.21.15.189:0/2060349938 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d8c13e460 0x7f8d8c137640 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f8d84009d50 tx=0x7f8d84009290).stop 2020-11-20T12:05:17.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.938+0000 7f8d919be700 1 -- 172.21.15.189:0/2060349938 shutdown_connections 2020-11-20T12:05:17.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.938+0000 7f8d919be700 1 --2- 172.21.15.189:0/2060349938 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d8c13e460 0x7f8d8c137640 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.938+0000 7f8d919be700 1 --2- 172.21.15.189:0/2060349938 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8d8c0a8a60 0x7f8d8c067400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.938+0000 7f8d919be700 1 -- 172.21.15.189:0/2060349938 >> 172.21.15.189:0/2060349938 conn(0x7f8d8c065420 msgr2=0x7f8d8c062080 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:17.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.938+0000 7f8d919be700 1 -- 172.21.15.189:0/2060349938 shutdown_connections 2020-11-20T12:05:17.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.938+0000 7f8d919be700 1 -- 172.21.15.189:0/2060349938 wait complete. 2020-11-20T12:05:17.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.942+0000 7f8d919be700 1 Processor -- start 2020-11-20T12:05:17.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.942+0000 7f8d919be700 1 -- start start 2020-11-20T12:05:17.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.942+0000 7f8d919be700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d8c0678f0 0x7f8d8c0a3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:17.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.942+0000 7f8d919be700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c0a8a60 0x7f8d8c136960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:17.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.942+0000 7f8d919be700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8d8c13e460 0x7f8d8c0a4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:17.954 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.942+0000 7f8d919be700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8c149ba0 con 0x7f8d8c0678f0 2020-11-20T12:05:17.954 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.942+0000 7f8d919be700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8c0a43b0 con 0x7f8d8c0a8a60 2020-11-20T12:05:17.954 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.942+0000 7f8d919be700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8c0a4650 con 0x7f8d8c13e460 2020-11-20T12:05:17.954 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.942+0000 7f8d8bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c0a8a60 0x7f8d8c136960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:17.954 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.942+0000 7f8d8bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c0a8a60 0x7f8d8c136960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49098/0 (socket says 172.21.15.189:49098) 2020-11-20T12:05:17.955 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.942+0000 7f8d8bfff700 1 -- 172.21.15.189:0/3391060460 learned_addr learned my addr 172.21.15.189:0/3391060460 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:17.955 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.942+0000 7f8d911bd700 1 --2- 172.21.15.189:0/3391060460 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8d8c13e460 0x7f8d8c0a4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:17.955 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.942+0000 7f8d909bc700 1 --2- 172.21.15.189:0/3391060460 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d8c0678f0 0x7f8d8c0a3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:17.955 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.942+0000 7f8d8bfff700 1 -- 172.21.15.189:0/3391060460 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8d8c13e460 msgr2=0x7f8d8c0a4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:17.955 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.942+0000 7f8d8bfff700 1 --2- 172.21.15.189:0/3391060460 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8d8c13e460 0x7f8d8c0a4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.942+0000 7f8d8bfff700 1 -- 172.21.15.189:0/3391060460 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d8c0678f0 msgr2=0x7f8d8c0a3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:17.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.942+0000 7f8d8bfff700 1 --2- 172.21.15.189:0/3391060460 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d8c0678f0 0x7f8d8c0a3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:17.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.942+0000 7f8d8bfff700 1 -- 172.21.15.189:0/3391060460 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8c0a4ac0 con 0x7f8d8c0a8a60 2020-11-20T12:05:17.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.942+0000 7f8d8bfff700 1 --2- 172.21.15.189:0/3391060460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c0a8a60 0x7f8d8c136960 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f8d7c00c6e0 tx=0x7f8d7c007c10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:17.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.942+0000 7f8d88ff9700 1 -- 172.21.15.189:0/3391060460 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c0072e0 con 0x7f8d8c0a8a60 2020-11-20T12:05:17.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.942+0000 7f8d88ff9700 1 -- 172.21.15.189:0/3391060460 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d7c005530 con 0x7f8d8c0a8a60 2020-11-20T12:05:17.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.942+0000 7f8d919be700 1 -- 172.21.15.189:0/3391060460 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d8c0a6d10 con 0x7f8d8c0a8a60 2020-11-20T12:05:17.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.942+0000 7f8d88ff9700 1 -- 172.21.15.189:0/3391060460 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c0058b0 con 0x7f8d8c0a8a60 2020-11-20T12:05:17.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.942+0000 7f8d919be700 1 -- 172.21.15.189:0/3391060460 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d8c0a5450 con 0x7f8d8c0a8a60 2020-11-20T12:05:17.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.946+0000 7f8d919be700 1 -- 172.21.15.189:0/3391060460 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d8c0a4c50 con 0x7f8d8c0a8a60 2020-11-20T12:05:17.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.946+0000 7f8d88ff9700 1 -- 172.21.15.189:0/3391060460 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f8d7c010070 con 0x7f8d8c0a8a60 2020-11-20T12:05:17.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.946+0000 7f8d88ff9700 1 --2- 172.21.15.189:0/3391060460 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f8d680557f0 0x7f8d68057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:17.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.946+0000 7f8d909bc700 1 --2- 172.21.15.189:0/3391060460 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f8d680557f0 0x7f8d68057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:17.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.946+0000 7f8d88ff9700 1 -- 172.21.15.189:0/3391060460 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8d7c06dd50 con 0x7f8d8c0a8a60 2020-11-20T12:05:17.960 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.946+0000 7f8d909bc700 1 --2- 172.21.15.189:0/3391060460 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f8d680557f0 0x7f8d68057cb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f8d74006fd0 tx=0x7f8d74006df0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:17.962 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:17.954+0000 7f8d88ff9700 1 -- 172.21.15.189:0/3391060460 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8d7c043bf0 con 0x7f8d8c0a8a60 2020-11-20T12:05:18.116 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.110+0000 7f8d919be700 1 -- 172.21.15.189:0/3391060460 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8d8c044f60 con 0x7f8d8c0a8a60 2020-11-20T12:05:18.117 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.110+0000 7f8d88ff9700 1 -- 172.21.15.189:0/3391060460 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+10 (secure 0 0 0) 0x7f8d7c047210 con 0x7f8d8c0a8a60 2020-11-20T12:05:18.117 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 8 2020-11-20T12:05:18.119 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.110+0000 7f8d919be700 1 -- 172.21.15.189:0/3391060460 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f8d680557f0 msgr2=0x7f8d68057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:18.119 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.110+0000 7f8d919be700 1 --2- 172.21.15.189:0/3391060460 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f8d680557f0 0x7f8d68057cb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f8d74006fd0 tx=0x7f8d74006df0).stop 2020-11-20T12:05:18.119 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.110+0000 7f8d919be700 1 -- 172.21.15.189:0/3391060460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c0a8a60 msgr2=0x7f8d8c136960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:18.119 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.110+0000 7f8d919be700 1 --2- 172.21.15.189:0/3391060460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c0a8a60 0x7f8d8c136960 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f8d7c00c6e0 tx=0x7f8d7c007c10).stop 2020-11-20T12:05:18.120 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.110+0000 7f8d919be700 1 -- 172.21.15.189:0/3391060460 shutdown_connections 2020-11-20T12:05:18.120 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.110+0000 7f8d919be700 1 --2- 172.21.15.189:0/3391060460 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f8d680557f0 0x7f8d68057cb0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:18.120 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.110+0000 7f8d919be700 1 --2- 172.21.15.189:0/3391060460 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8d8c0678f0 0x7f8d8c0a3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:18.121 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.110+0000 7f8d919be700 1 --2- 172.21.15.189:0/3391060460 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8d8c13e460 0x7f8d8c0a4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:18.121 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.110+0000 7f8d919be700 1 --2- 172.21.15.189:0/3391060460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c0a8a60 0x7f8d8c136960 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:18.121 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.110+0000 7f8d919be700 1 -- 172.21.15.189:0/3391060460 >> 172.21.15.189:0/3391060460 conn(0x7f8d8c065420 msgr2=0x7f8d8c05ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:18.121 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.110+0000 7f8d919be700 1 -- 172.21.15.189:0/3391060460 shutdown_connections 2020-11-20T12:05:18.121 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.110+0000 7f8d919be700 1 -- 172.21.15.189:0/3391060460 wait complete. 2020-11-20T12:05:18.131 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2020-11-20T12:05:18.132 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:05:18.250 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.242+0000 7f59c23e3700 1 Processor -- start 2020-11-20T12:05:18.251 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.242+0000 7f59c23e3700 1 -- start start 2020-11-20T12:05:18.253 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59c23e3700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59bc062220 0x7f59bc13a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:18.253 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59c23e3700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f59bc066fd0 0x7f59bc060a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:18.253 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59c23e3700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59bc0a9bb0 con 0x7f59bc062600 2020-11-20T12:05:18.254 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59c23e3700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59bc13ab40 con 0x7f59bc066fd0 2020-11-20T12:05:18.254 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59c23e3700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59bc13ade0 con 0x7f59bc062220 2020-11-20T12:05:18.254 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59c0be0700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f59bc062600 0x7f59bc137750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50728/0 (socket says 172.21.15.189:50728) 2020-11-20T12:05:18.254 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59c0be0700 1 -- 172.21.15.189:0/86796316 learned_addr learned my addr 172.21.15.189:0/86796316 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:18.255 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59c13e1700 1 --2- 172.21.15.189:0/86796316 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59bc062220 0x7f59bc13a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:18.255 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59c1be2700 1 --2- 172.21.15.189:0/86796316 >> v2:172.21.15.47:3300/0 conn(0x7f59bc066fd0 0x7f59bc060a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:18.255 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59b3fff700 1 -- 172.21.15.189:0/86796316 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7552870 0 0) 0x7f59ac0010d0 con 0x7f59bc062600 2020-11-20T12:05:18.255 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59b3fff700 1 -- 172.21.15.189:0/86796316 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59a0003240 con 0x7f59bc062600 2020-11-20T12:05:18.256 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59c13e1700 1 -- 172.21.15.189:0/86796316 >> v2:172.21.15.47:3300/0 conn(0x7f59bc066fd0 msgr2=0x7f59bc060a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:18.256 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59c13e1700 1 --2- 172.21.15.189:0/86796316 >> v2:172.21.15.47:3300/0 conn(0x7f59bc066fd0 0x7f59bc060a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:18.256 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59c13e1700 1 -- 172.21.15.189:0/86796316 >> v1:172.21.15.189:6789/0 conn(0x7f59bc062600 legacy=0x7f59bc137750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:18.256 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59b3fff700 1 -- 172.21.15.189:0/86796316 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 86971530 0 0) 0x7f59a0003240 con 0x7f59bc062600 2020-11-20T12:05:18.257 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59c13e1700 1 -- 172.21.15.189:0/86796316 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59bc13b080 con 0x7f59bc062220 2020-11-20T12:05:18.257 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59c13e1700 1 --2- 172.21.15.189:0/86796316 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59bc062220 0x7f59bc13a4a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f59b800b9a0 tx=0x7f59b8006df0).ready entity=mon.2 client_cookie=fbc3d6d07d8c9e27 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:18.257 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59b3fff700 1 -- 172.21.15.189:0/86796316 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b8009710 con 0x7f59bc062220 2020-11-20T12:05:18.257 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59b3fff700 1 -- 172.21.15.189:0/86796316 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b8009870 con 0x7f59bc062220 2020-11-20T12:05:18.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59b3fff700 1 -- 172.21.15.189:0/86796316 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b8009bf0 con 0x7f59bc062220 2020-11-20T12:05:18.261 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59c23e3700 1 -- 172.21.15.189:0/86796316 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59bc062220 msgr2=0x7f59bc13a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:18.261 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59c23e3700 1 --2- 172.21.15.189:0/86796316 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59bc062220 0x7f59bc13a4a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f59b800b9a0 tx=0x7f59b8006df0).stop 2020-11-20T12:05:18.261 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59c23e3700 1 -- 172.21.15.189:0/86796316 shutdown_connections 2020-11-20T12:05:18.262 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59c23e3700 1 --2- 172.21.15.189:0/86796316 >> v2:172.21.15.47:3300/0 conn(0x7f59bc066fd0 0x7f59bc060a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:18.262 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59c23e3700 1 --2- 172.21.15.189:0/86796316 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59bc062220 0x7f59bc13a4a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:18.262 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59c23e3700 1 -- 172.21.15.189:0/86796316 >> 172.21.15.189:0/86796316 conn(0x7f59bc065fe0 msgr2=0x7f59bc061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:18.262 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59c23e3700 1 -- 172.21.15.189:0/86796316 shutdown_connections 2020-11-20T12:05:18.262 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59c23e3700 1 -- 172.21.15.189:0/86796316 wait complete. 2020-11-20T12:05:18.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.246+0000 7f59c23e3700 1 Processor -- start 2020-11-20T12:05:18.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.250+0000 7f59c23e3700 1 -- start start 2020-11-20T12:05:18.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.250+0000 7f59c23e3700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59bc062220 0x7f59bc131df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:18.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.250+0000 7f59c23e3700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59bc062600 0x7f59bc13cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:18.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.250+0000 7f59c23e3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59bc066fd0 0x7f59bc149d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:18.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.250+0000 7f59c23e3700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59bc13d050 con 0x7f59bc062600 2020-11-20T12:05:18.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.250+0000 7f59c23e3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59bc13b9b0 con 0x7f59bc066fd0 2020-11-20T12:05:18.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.250+0000 7f59c23e3700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59bc13bc50 con 0x7f59bc062220 2020-11-20T12:05:18.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.250+0000 7f59c1be2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59bc066fd0 0x7f59bc149d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:18.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.250+0000 7f59c1be2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59bc066fd0 0x7f59bc149d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49114/0 (socket says 172.21.15.189:49114) 2020-11-20T12:05:18.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.250+0000 7f59c1be2700 1 -- 172.21.15.189:0/3255744168 learned_addr learned my addr 172.21.15.189:0/3255744168 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:18.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.250+0000 7f59c13e1700 1 --2- 172.21.15.189:0/3255744168 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59bc062220 0x7f59bc131df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:18.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.250+0000 7f59c0be0700 1 --2- 172.21.15.189:0/3255744168 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59bc062600 0x7f59bc13cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:18.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.250+0000 7f59c1be2700 1 -- 172.21.15.189:0/3255744168 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59bc062220 msgr2=0x7f59bc131df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:18.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.250+0000 7f59c1be2700 1 --2- 172.21.15.189:0/3255744168 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59bc062220 0x7f59bc131df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:18.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.250+0000 7f59c1be2700 1 -- 172.21.15.189:0/3255744168 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59bc062600 msgr2=0x7f59bc13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:18.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.250+0000 7f59c1be2700 1 --2- 172.21.15.189:0/3255744168 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59bc062600 0x7f59bc13cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:18.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.250+0000 7f59c1be2700 1 -- 172.21.15.189:0/3255744168 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59bc13bef0 con 0x7f59bc066fd0 2020-11-20T12:05:18.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.250+0000 7f59c1be2700 1 --2- 172.21.15.189:0/3255744168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59bc066fd0 0x7f59bc149d20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f59b400ed50 tx=0x7f59b40091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:18.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.250+0000 7f59b17fa700 1 -- 172.21.15.189:0/3255744168 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b400cb00 con 0x7f59bc066fd0 2020-11-20T12:05:18.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.250+0000 7f59b17fa700 1 -- 172.21.15.189:0/3255744168 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b400cc60 con 0x7f59bc066fd0 2020-11-20T12:05:18.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.250+0000 7f59b17fa700 1 -- 172.21.15.189:0/3255744168 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b40123f0 con 0x7f59bc066fd0 2020-11-20T12:05:18.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.250+0000 7f59c23e3700 1 -- 172.21.15.189:0/3255744168 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59bc13c390 con 0x7f59bc066fd0 2020-11-20T12:05:18.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.250+0000 7f59c23e3700 1 -- 172.21.15.189:0/3255744168 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59bc0a1730 con 0x7f59bc066fd0 2020-11-20T12:05:18.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.250+0000 7f59c23e3700 1 -- 172.21.15.189:0/3255744168 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59bc0a1980 con 0x7f59bc066fd0 2020-11-20T12:05:18.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.254+0000 7f59b17fa700 1 -- 172.21.15.189:0/3255744168 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f59b4022070 con 0x7f59bc066fd0 2020-11-20T12:05:18.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.254+0000 7f59b17fa700 1 --2- 172.21.15.189:0/3255744168 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f59940557c0 0x7f5994057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:18.269 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.254+0000 7f59b17fa700 1 -- 172.21.15.189:0/3255744168 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f59b406ee10 con 0x7f59bc066fd0 2020-11-20T12:05:18.269 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.254+0000 7f59c13e1700 1 --2- 172.21.15.189:0/3255744168 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f59940557c0 0x7f5994057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:18.269 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.254+0000 7f59c13e1700 1 --2- 172.21.15.189:0/3255744168 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f59940557c0 0x7f5994057c80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f59b8009fd0 tx=0x7f59b8015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:18.270 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.258+0000 7f59b17fa700 1 -- 172.21.15.189:0/3255744168 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f59bc0a1980 con 0x7f59bc066fd0 2020-11-20T12:05:18.469 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.462+0000 7f59c23e3700 1 -- 172.21.15.189:0/3255744168 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59bc0a1980 con 0x7f59bc066fd0 2020-11-20T12:05:18.470 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.462+0000 7f59b17fa700 1 -- 172.21.15.189:0/3255744168 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2134 (secure 0 0 0) 0x7f59bc0a1980 con 0x7f59bc066fd0 2020-11-20T12:05:18.470 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:05:18.470 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2660417436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2660417436},{"type":"v1","addr":"172.21.15.47:6833","nonce":2660417436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:04:56.025055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.189:6835/1042140684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1042140684},{"type":"v1","addr":"172.21.15.189:6835","nonce":1042140684}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:05:18.472 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.466+0000 7f59c23e3700 1 -- 172.21.15.189:0/3255744168 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f59940557c0 msgr2=0x7f5994057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:18.472 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.466+0000 7f59c23e3700 1 --2- 172.21.15.189:0/3255744168 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f59940557c0 0x7f5994057c80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f59b8009fd0 tx=0x7f59b8015000).stop 2020-11-20T12:05:18.472 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.466+0000 7f59c23e3700 1 -- 172.21.15.189:0/3255744168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59bc066fd0 msgr2=0x7f59bc149d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:18.473 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.466+0000 7f59c23e3700 1 --2- 172.21.15.189:0/3255744168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59bc066fd0 0x7f59bc149d20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f59b400ed50 tx=0x7f59b40091d0).stop 2020-11-20T12:05:18.473 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.466+0000 7f59c23e3700 1 -- 172.21.15.189:0/3255744168 shutdown_connections 2020-11-20T12:05:18.473 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.466+0000 7f59c23e3700 1 --2- 172.21.15.189:0/3255744168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59bc066fd0 0x7f59bc149d20 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:18.473 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.466+0000 7f59c23e3700 1 --2- 172.21.15.189:0/3255744168 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59bc062220 0x7f59bc131df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:18.473 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.466+0000 7f59c23e3700 1 --2- 172.21.15.189:0/3255744168 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f59940557c0 0x7f5994057c80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:18.474 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.466+0000 7f59c23e3700 1 --2- 172.21.15.189:0/3255744168 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59bc062600 0x7f59bc13cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:18.474 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.466+0000 7f59c23e3700 1 -- 172.21.15.189:0/3255744168 >> 172.21.15.189:0/3255744168 conn(0x7f59bc065fe0 msgr2=0x7f59bc060f70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:18.474 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.466+0000 7f59c23e3700 1 -- 172.21.15.189:0/3255744168 shutdown_connections 2020-11-20T12:05:18.474 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.466+0000 7f59c23e3700 1 -- 172.21.15.189:0/3255744168 wait complete. 2020-11-20T12:05:18.475 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 6 2020-11-20T12:05:18.484 INFO:tasks.mds_thrash:Ready to start thrashing 2020-11-20T12:05:18.484 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2020-11-20T12:05:18.485 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-20T12:05:18.609 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.602+0000 7f1c05f94700 1 Processor -- start 2020-11-20T12:05:18.610 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.602+0000 7f1c05f94700 1 -- start start 2020-11-20T12:05:18.612 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1c05f94700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1c000612b0 0x7f1c000a0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:18.612 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1c05f94700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1c001376c0 0x7f1c00133a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:18.612 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1c05f94700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c00068020 con 0x7f1c00067b50 2020-11-20T12:05:18.612 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1c05f94700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c000682c0 con 0x7f1c001376c0 2020-11-20T12:05:18.612 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1c05f94700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c00061690 con 0x7f1c000612b0 2020-11-20T12:05:18.613 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1bf7fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1c00067b50 0x7f1c00136fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50744/0 (socket says 172.21.15.189:50744) 2020-11-20T12:05:18.613 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1bf7fff700 1 -- 172.21.15.189:0/1889319340 learned_addr learned my addr 172.21.15.189:0/1889319340 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:18.613 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1c04f92700 1 --2- 172.21.15.189:0/1889319340 >> v2:172.21.15.189:3300/0 conn(0x7f1c000612b0 0x7f1c000a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:18.613 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1c05793700 1 --2- 172.21.15.189:0/1889319340 >> v2:172.21.15.47:3300/0 conn(0x7f1c001376c0 0x7f1c00133a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:18.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1bf77fe700 1 -- 172.21.15.189:0/1889319340 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201605740 0 0) 0x7f1c00068020 con 0x7f1c00067b50 2020-11-20T12:05:18.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1bf77fe700 1 -- 172.21.15.189:0/1889319340 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bec003240 con 0x7f1c00067b50 2020-11-20T12:05:18.617 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1c04f92700 1 -- 172.21.15.189:0/1889319340 >> v2:172.21.15.47:3300/0 conn(0x7f1c001376c0 msgr2=0x7f1c00133a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:18.618 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1bf77fe700 1 -- 172.21.15.189:0/1889319340 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 190748671 0 0) 0x7f1bec003240 con 0x7f1c00067b50 2020-11-20T12:05:18.618 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1c04f92700 1 --2- 172.21.15.189:0/1889319340 >> v2:172.21.15.47:3300/0 conn(0x7f1c001376c0 0x7f1c00133a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:18.618 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1c04f92700 1 -- 172.21.15.189:0/1889319340 >> v1:172.21.15.189:6789/0 conn(0x7f1c00067b50 legacy=0x7f1c00136fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:18.618 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1c04f92700 1 -- 172.21.15.189:0/1889319340 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c00136aa0 con 0x7f1c000612b0 2020-11-20T12:05:18.619 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1c04f92700 1 --2- 172.21.15.189:0/1889319340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c000612b0 0x7f1c000a0960 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f1c000ae330 tx=0x7f1bfc006df0).ready entity=mon.1 client_cookie=6f7b5697d6f88a89 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:18.619 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1bf77fe700 1 -- 172.21.15.189:0/1889319340 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf00014e0 con 0x7f1c000612b0 2020-11-20T12:05:18.619 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1bf77fe700 1 -- 172.21.15.189:0/1889319340 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bfc011400 con 0x7f1c000612b0 2020-11-20T12:05:18.619 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1bf77fe700 1 -- 172.21.15.189:0/1889319340 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc011780 con 0x7f1c000612b0 2020-11-20T12:05:18.620 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1c05f94700 1 -- 172.21.15.189:0/1889319340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c000612b0 msgr2=0x7f1c000a0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:18.620 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1c05f94700 1 --2- 172.21.15.189:0/1889319340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c000612b0 0x7f1c000a0960 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f1c000ae330 tx=0x7f1bfc006df0).stop 2020-11-20T12:05:18.620 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1c05f94700 1 -- 172.21.15.189:0/1889319340 shutdown_connections 2020-11-20T12:05:18.620 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1c05f94700 1 --2- 172.21.15.189:0/1889319340 >> v2:172.21.15.47:3300/0 conn(0x7f1c001376c0 0x7f1c00133a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:18.621 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1c05f94700 1 --2- 172.21.15.189:0/1889319340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c000612b0 0x7f1c000a0960 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:18.621 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1c05f94700 1 -- 172.21.15.189:0/1889319340 >> 172.21.15.189:0/1889319340 conn(0x7f1c00063210 msgr2=0x7f1c00060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:18.621 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1c05f94700 1 -- 172.21.15.189:0/1889319340 shutdown_connections 2020-11-20T12:05:18.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.606+0000 7f1c05f94700 1 -- 172.21.15.189:0/1889319340 wait complete. 2020-11-20T12:05:18.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.610+0000 7f1c05f94700 1 Processor -- start 2020-11-20T12:05:18.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.610+0000 7f1c05f94700 1 -- start start 2020-11-20T12:05:18.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.610+0000 7f1c05f94700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1c000612b0 0x7f1c00130550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:18.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.610+0000 7f1c05f94700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c00067b50 0x7f1c0014cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:18.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.610+0000 7f1c05f94700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c001376c0 0x7f1c00138ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:18.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.610+0000 7f1c05f94700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0014d560 con 0x7f1c001376c0 2020-11-20T12:05:18.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.610+0000 7f1c05f94700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0014d800 con 0x7f1c00067b50 2020-11-20T12:05:18.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.610+0000 7f1c05f94700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0014daa0 con 0x7f1c000612b0 2020-11-20T12:05:18.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.610+0000 7f1bf7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c00067b50 0x7f1c0014cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:18.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.610+0000 7f1bf7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c00067b50 0x7f1c0014cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49126/0 (socket says 172.21.15.189:49126) 2020-11-20T12:05:18.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.610+0000 7f1bf7fff700 1 -- 172.21.15.189:0/2831132895 learned_addr learned my addr 172.21.15.189:0/2831132895 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:18.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.610+0000 7f1c05793700 1 --2- 172.21.15.189:0/2831132895 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c001376c0 0x7f1c00138ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:18.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.610+0000 7f1c04f92700 1 --2- 172.21.15.189:0/2831132895 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1c000612b0 0x7f1c00130550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:18.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.610+0000 7f1bf7fff700 1 -- 172.21.15.189:0/2831132895 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1c000612b0 msgr2=0x7f1c00130550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:18.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.610+0000 7f1bf7fff700 1 --2- 172.21.15.189:0/2831132895 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1c000612b0 0x7f1c00130550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:18.626 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.610+0000 7f1bf7fff700 1 -- 172.21.15.189:0/2831132895 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c001376c0 msgr2=0x7f1c00138ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:18.626 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.610+0000 7f1bf7fff700 1 --2- 172.21.15.189:0/2831132895 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c001376c0 0x7f1c00138ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:18.626 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.610+0000 7f1bf7fff700 1 -- 172.21.15.189:0/2831132895 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c001393b0 con 0x7f1c00067b50 2020-11-20T12:05:18.626 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.610+0000 7f1c04f92700 1 --2- 172.21.15.189:0/2831132895 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1c000612b0 0x7f1c00130550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:18.627 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.610+0000 7f1bf7fff700 1 --2- 172.21.15.189:0/2831132895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c00067b50 0x7f1c0014cec0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f1bf000ca60 tx=0x7f1bf00011f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:18.627 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.610+0000 7f1bf4ff9700 1 -- 172.21.15.189:0/2831132895 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf0007320 con 0x7f1c00067b50 2020-11-20T12:05:18.627 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.610+0000 7f1bf4ff9700 1 -- 172.21.15.189:0/2831132895 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf0007480 con 0x7f1c00067b50 2020-11-20T12:05:18.627 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.610+0000 7f1bf4ff9700 1 -- 172.21.15.189:0/2831132895 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf0005800 con 0x7f1c00067b50 2020-11-20T12:05:18.627 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.610+0000 7f1c05f94700 1 -- 172.21.15.189:0/2831132895 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c000ac0e0 con 0x7f1c00067b50 2020-11-20T12:05:18.628 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.610+0000 7f1c05f94700 1 -- 172.21.15.189:0/2831132895 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c000ac5e0 con 0x7f1c00067b50 2020-11-20T12:05:18.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.610+0000 7f1c05f94700 1 -- 172.21.15.189:0/2831132895 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c0014d3b0 con 0x7f1c00067b50 2020-11-20T12:05:18.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.614+0000 7f1bf4ff9700 1 -- 172.21.15.189:0/2831132895 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f1bf0010030 con 0x7f1c00067b50 2020-11-20T12:05:18.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.614+0000 7f1bf4ff9700 1 --2- 172.21.15.189:0/2831132895 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1bd80557c0 0x7f1bd8057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:18.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.614+0000 7f1bf4ff9700 1 -- 172.21.15.189:0/2831132895 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1bf006e790 con 0x7f1c00067b50 2020-11-20T12:05:18.630 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.614+0000 7f1c04f92700 1 --2- 172.21.15.189:0/2831132895 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1bd80557c0 0x7f1bd8057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:18.630 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.614+0000 7f1c04f92700 1 --2- 172.21.15.189:0/2831132895 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1bd80557c0 0x7f1bd8057c80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f1bfc00d260 tx=0x7f1bfc015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:18.630 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.618+0000 7f1bf4ff9700 1 -- 172.21.15.189:0/2831132895 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1bf0044630 con 0x7f1c00067b50 2020-11-20T12:05:18.777 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.770+0000 7f1c05f94700 1 -- 172.21.15.189:0/2831132895 --> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1c000446d0 con 0x7f1bd80557c0 2020-11-20T12:05:18.785 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.778+0000 7f1bf4ff9700 1 -- 172.21.15.189:0/2831132895 <== mgr.4105 v2:172.21.15.189:6832/11671 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+82253 (secure 0 0 0) 0x7f1c000446d0 con 0x7f1bd80557c0 2020-11-20T12:05:18.786 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:05:18.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.782+0000 7f1c05f94700 1 -- 172.21.15.189:0/2831132895 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1bd80557c0 msgr2=0x7f1bd8057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:18.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.782+0000 7f1c05f94700 1 --2- 172.21.15.189:0/2831132895 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1bd80557c0 0x7f1bd8057c80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f1bfc00d260 tx=0x7f1bfc015000).stop 2020-11-20T12:05:18.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.782+0000 7f1c05f94700 1 -- 172.21.15.189:0/2831132895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c00067b50 msgr2=0x7f1c0014cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:18.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.782+0000 7f1c05f94700 1 --2- 172.21.15.189:0/2831132895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c00067b50 0x7f1c0014cec0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f1bf000ca60 tx=0x7f1bf00011f0).stop 2020-11-20T12:05:18.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.782+0000 7f1c05f94700 1 -- 172.21.15.189:0/2831132895 shutdown_connections 2020-11-20T12:05:18.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.782+0000 7f1c05f94700 1 --2- 172.21.15.189:0/2831132895 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1bd80557c0 0x7f1bd8057c80 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:18.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.782+0000 7f1c05f94700 1 --2- 172.21.15.189:0/2831132895 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c001376c0 0x7f1c00138ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:18.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.782+0000 7f1c05f94700 1 --2- 172.21.15.189:0/2831132895 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1c000612b0 0x7f1c00130550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:18.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.782+0000 7f1c05f94700 1 --2- 172.21.15.189:0/2831132895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c00067b50 0x7f1c0014cec0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:18.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.782+0000 7f1c05f94700 1 -- 172.21.15.189:0/2831132895 >> 172.21.15.189:0/2831132895 conn(0x7f1c00063210 msgr2=0x7f1c000adae0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:18.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.782+0000 7f1c05f94700 1 -- 172.21.15.189:0/2831132895 shutdown_connections 2020-11-20T12:05:18.790 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.782+0000 7f1c05f94700 1 -- 172.21.15.189:0/2831132895 wait complete. 2020-11-20T12:05:18.790 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2020-11-20T12:05:18.802 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2020-11-20T12:05:17.342765+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2747,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":1887024,"kb_used_data":747708032,"kb_used_omap":602,"kb_used_meta":0,"kb_avail":747708016,"statfs":{"total":767585320960,"available":765653008384,"internally_reserved":16384,"allocated":765653024768,"data_stored":765653024768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":617346,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":443,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003773"},"pg_stats":[{"pgid":"4.2","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.025896+0000","last_change":"2020-11-20T12:04:54.024234+0000","last_active":"2020-11-20T12:04:54.025896+0000","last_peered":"2020-11-20T12:04:54.025896+0000","last_clean":"2020-11-20T12:04:54.025896+0000","last_became_active":"2020-11-20T12:04:54.023862+0000","last_became_peered":"2020-11-20T12:04:54.023862+0000","last_unstale":"2020-11-20T12:04:54.025896+0000","last_undegraded":"2020-11-20T12:04:54.025896+0000","last_fullsized":"2020-11-20T12:04:54.025896+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.025135+0000","last_change":"2020-11-20T12:04:51.104716+0000","last_active":"2020-11-20T12:04:54.025135+0000","last_peered":"2020-11-20T12:04:54.025135+0000","last_clean":"2020-11-20T12:04:54.025135+0000","last_became_active":"2020-11-20T12:04:51.104301+0000","last_became_peered":"2020-11-20T12:04:51.104301+0000","last_unstale":"2020-11-20T12:04:54.025135+0000","last_undegraded":"2020-11-20T12:04:54.025135+0000","last_fullsized":"2020-11-20T12:04:54.025135+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"21'6","reported_seq":"16","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.039919+0000","last_change":"2020-11-20T12:04:53.664474+0000","last_active":"2020-11-20T12:04:54.039919+0000","last_peered":"2020-11-20T12:04:54.039919+0000","last_clean":"2020-11-20T12:04:54.039919+0000","last_became_active":"2020-11-20T12:04:53.664073+0000","last_became_peered":"2020-11-20T12:04:53.664073+0000","last_unstale":"2020-11-20T12:04:54.039919+0000","last_undegraded":"2020-11-20T12:04:54.039919+0000","last_fullsized":"2020-11-20T12:04:54.039919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.037879+0000","last_change":"2020-11-20T12:04:54.033863+0000","last_active":"2020-11-20T12:04:54.037879+0000","last_peered":"2020-11-20T12:04:54.037879+0000","last_clean":"2020-11-20T12:04:54.037879+0000","last_became_active":"2020-11-20T12:04:54.032723+0000","last_became_peered":"2020-11-20T12:04:54.032723+0000","last_unstale":"2020-11-20T12:04:54.037879+0000","last_undegraded":"2020-11-20T12:04:54.037879+0000","last_fullsized":"2020-11-20T12:04:54.037879+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.018806+0000","last_change":"2020-11-20T12:04:51.107483+0000","last_active":"2020-11-20T12:04:54.018806+0000","last_peered":"2020-11-20T12:04:54.018806+0000","last_clean":"2020-11-20T12:04:54.018806+0000","last_became_active":"2020-11-20T12:04:51.106350+0000","last_became_peered":"2020-11-20T12:04:51.106350+0000","last_unstale":"2020-11-20T12:04:54.018806+0000","last_undegraded":"2020-11-20T12:04:54.018806+0000","last_fullsized":"2020-11-20T12:04:54.018806+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"21'5","reported_seq":"15","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.034249+0000","last_change":"2020-11-20T12:04:53.660754+0000","last_active":"2020-11-20T12:04:54.034249+0000","last_peered":"2020-11-20T12:04:54.034249+0000","last_clean":"2020-11-20T12:04:54.034249+0000","last_became_active":"2020-11-20T12:04:53.660354+0000","last_became_peered":"2020-11-20T12:04:53.660354+0000","last_unstale":"2020-11-20T12:04:54.034249+0000","last_undegraded":"2020-11-20T12:04:54.034249+0000","last_fullsized":"2020-11-20T12:04:54.034249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.036255+0000","last_change":"2020-11-20T12:04:54.036030+0000","last_active":"2020-11-20T12:04:54.036255+0000","last_peered":"2020-11-20T12:04:54.036255+0000","last_clean":"2020-11-20T12:04:54.036255+0000","last_became_active":"2020-11-20T12:04:54.035638+0000","last_became_peered":"2020-11-20T12:04:54.035638+0000","last_unstale":"2020-11-20T12:04:54.036255+0000","last_undegraded":"2020-11-20T12:04:54.036255+0000","last_fullsized":"2020-11-20T12:04:54.036255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.028309+0000","last_change":"2020-11-20T12:04:50.499140+0000","last_active":"2020-11-20T12:04:54.028309+0000","last_peered":"2020-11-20T12:04:54.028309+0000","last_clean":"2020-11-20T12:04:54.028309+0000","last_became_active":"2020-11-20T12:04:50.498705+0000","last_became_peered":"2020-11-20T12:04:50.498705+0000","last_unstale":"2020-11-20T12:04:54.028309+0000","last_undegraded":"2020-11-20T12:04:54.028309+0000","last_fullsized":"2020-11-20T12:04:54.028309+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"21'8","reported_seq":"18","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.040833+0000","last_change":"2020-11-20T12:04:53.657720+0000","last_active":"2020-11-20T12:04:54.040833+0000","last_peered":"2020-11-20T12:04:54.040833+0000","last_clean":"2020-11-20T12:04:54.040833+0000","last_became_active":"2020-11-20T12:04:53.657238+0000","last_became_peered":"2020-11-20T12:04:53.657238+0000","last_unstale":"2020-11-20T12:04:54.040833+0000","last_undegraded":"2020-11-20T12:04:54.040833+0000","last_fullsized":"2020-11-20T12:04:54.040833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.345488+0000","last_change":"2020-11-20T12:04:54.345320+0000","last_active":"2020-11-20T12:04:54.345488+0000","last_peered":"2020-11-20T12:04:54.345488+0000","last_clean":"2020-11-20T12:04:54.345488+0000","last_became_active":"2020-11-20T12:04:54.344995+0000","last_became_peered":"2020-11-20T12:04:54.344995+0000","last_unstale":"2020-11-20T12:04:54.345488+0000","last_undegraded":"2020-11-20T12:04:54.345488+0000","last_fullsized":"2020-11-20T12:04:54.345488+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.019834+0000","last_change":"2020-11-20T12:04:51.425413+0000","last_active":"2020-11-20T12:04:54.019834+0000","last_peered":"2020-11-20T12:04:54.019834+0000","last_clean":"2020-11-20T12:04:54.019834+0000","last_became_active":"2020-11-20T12:04:51.424595+0000","last_became_peered":"2020-11-20T12:04:51.424595+0000","last_unstale":"2020-11-20T12:04:54.019834+0000","last_undegraded":"2020-11-20T12:04:54.019834+0000","last_fullsized":"2020-11-20T12:04:54.019834+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"14'2","reported_seq":"18","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.344529+0000","last_change":"2020-11-20T12:04:44.090086+0000","last_active":"2020-11-20T12:04:54.344529+0000","last_peered":"2020-11-20T12:04:54.344529+0000","last_clean":"2020-11-20T12:04:54.344529+0000","last_became_active":"2020-11-20T12:04:44.089120+0000","last_became_peered":"2020-11-20T12:04:44.089120+0000","last_unstale":"2020-11-20T12:04:54.344529+0000","last_undegraded":"2020-11-20T12:04:54.344529+0000","last_fullsized":"2020-11-20T12:04:54.344529+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:42.003758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:42.003758+0000","last_clean_scrub_stamp":"2020-11-20T12:04:42.003758+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"21'1","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.034984+0000","last_change":"2020-11-20T12:04:53.660727+0000","last_active":"2020-11-20T12:04:54.034984+0000","last_peered":"2020-11-20T12:04:54.034984+0000","last_clean":"2020-11-20T12:04:54.034984+0000","last_became_active":"2020-11-20T12:04:53.660424+0000","last_became_peered":"2020-11-20T12:04:53.660424+0000","last_unstale":"2020-11-20T12:04:54.034984+0000","last_undegraded":"2020-11-20T12:04:54.034984+0000","last_fullsized":"2020-11-20T12:04:54.034984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.038696+0000","last_change":"2020-11-20T12:04:54.038479+0000","last_active":"2020-11-20T12:04:54.038696+0000","last_peered":"2020-11-20T12:04:54.038696+0000","last_clean":"2020-11-20T12:04:54.038696+0000","last_became_active":"2020-11-20T12:04:54.037927+0000","last_became_peered":"2020-11-20T12:04:54.037927+0000","last_unstale":"2020-11-20T12:04:54.038696+0000","last_undegraded":"2020-11-20T12:04:54.038696+0000","last_fullsized":"2020-11-20T12:04:54.038696+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.020094+0000","last_change":"2020-11-20T12:04:50.499789+0000","last_active":"2020-11-20T12:04:54.020094+0000","last_peered":"2020-11-20T12:04:54.020094+0000","last_clean":"2020-11-20T12:04:54.020094+0000","last_became_active":"2020-11-20T12:04:50.499414+0000","last_became_peered":"2020-11-20T12:04:50.499414+0000","last_unstale":"2020-11-20T12:04:54.020094+0000","last_undegraded":"2020-11-20T12:04:54.020094+0000","last_fullsized":"2020-11-20T12:04:54.020094+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"10","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.019676+0000","last_change":"2020-11-20T12:04:53.665538+0000","last_active":"2020-11-20T12:04:54.019676+0000","last_peered":"2020-11-20T12:04:54.019676+0000","last_clean":"2020-11-20T12:04:54.019676+0000","last_became_active":"2020-11-20T12:04:53.664812+0000","last_became_peered":"2020-11-20T12:04:53.664812+0000","last_unstale":"2020-11-20T12:04:54.019676+0000","last_undegraded":"2020-11-20T12:04:54.019676+0000","last_fullsized":"2020-11-20T12:04:54.019676+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.039072+0000","last_change":"2020-11-20T12:04:54.038559+0000","last_active":"2020-11-20T12:04:54.039072+0000","last_peered":"2020-11-20T12:04:54.039072+0000","last_clean":"2020-11-20T12:04:54.039072+0000","last_became_active":"2020-11-20T12:04:54.038134+0000","last_became_peered":"2020-11-20T12:04:54.038134+0000","last_unstale":"2020-11-20T12:04:54.039072+0000","last_undegraded":"2020-11-20T12:04:54.039072+0000","last_fullsized":"2020-11-20T12:04:54.039072+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.019201+0000","last_change":"2020-11-20T12:04:50.500191+0000","last_active":"2020-11-20T12:04:54.019201+0000","last_peered":"2020-11-20T12:04:54.019201+0000","last_clean":"2020-11-20T12:04:54.019201+0000","last_became_active":"2020-11-20T12:04:50.499854+0000","last_became_peered":"2020-11-20T12:04:50.499854+0000","last_unstale":"2020-11-20T12:04:54.019201+0000","last_undegraded":"2020-11-20T12:04:54.019201+0000","last_fullsized":"2020-11-20T12:04:54.019201+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"22'10","reported_seq":"20","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:05:11.751763+0000","last_change":"2020-11-20T12:04:53.665307+0000","last_active":"2020-11-20T12:05:11.751763+0000","last_peered":"2020-11-20T12:05:11.751763+0000","last_clean":"2020-11-20T12:05:11.751763+0000","last_became_active":"2020-11-20T12:04:53.664857+0000","last_became_peered":"2020-11-20T12:04:53.664857+0000","last_unstale":"2020-11-20T12:05:11.751763+0000","last_undegraded":"2020-11-20T12:05:11.751763+0000","last_fullsized":"2020-11-20T12:05:11.751763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1978,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.346166+0000","last_change":"2020-11-20T12:04:54.345851+0000","last_active":"2020-11-20T12:04:54.346166+0000","last_peered":"2020-11-20T12:04:54.346166+0000","last_clean":"2020-11-20T12:04:54.346166+0000","last_became_active":"2020-11-20T12:04:54.345040+0000","last_became_peered":"2020-11-20T12:04:54.345040+0000","last_unstale":"2020-11-20T12:04:54.346166+0000","last_undegraded":"2020-11-20T12:04:54.346166+0000","last_fullsized":"2020-11-20T12:04:54.346166+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"21'6","reported_seq":"16","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.039735+0000","last_change":"2020-11-20T12:04:53.663035+0000","last_active":"2020-11-20T12:04:54.039735+0000","last_peered":"2020-11-20T12:04:54.039735+0000","last_clean":"2020-11-20T12:04:54.039735+0000","last_became_active":"2020-11-20T12:04:53.662666+0000","last_became_peered":"2020-11-20T12:04:53.662666+0000","last_unstale":"2020-11-20T12:04:54.039735+0000","last_undegraded":"2020-11-20T12:04:54.039735+0000","last_fullsized":"2020-11-20T12:04:54.039735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.025490+0000","last_change":"2020-11-20T12:04:51.447405+0000","last_active":"2020-11-20T12:04:54.025490+0000","last_peered":"2020-11-20T12:04:54.025490+0000","last_clean":"2020-11-20T12:04:54.025490+0000","last_became_active":"2020-11-20T12:04:51.446352+0000","last_became_peered":"2020-11-20T12:04:51.446352+0000","last_unstale":"2020-11-20T12:04:54.025490+0000","last_undegraded":"2020-11-20T12:04:54.025490+0000","last_fullsized":"2020-11-20T12:04:54.025490+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.024608+0000","last_change":"2020-11-20T12:04:54.024326+0000","last_active":"2020-11-20T12:04:54.024608+0000","last_peered":"2020-11-20T12:04:54.024608+0000","last_clean":"2020-11-20T12:04:54.024608+0000","last_became_active":"2020-11-20T12:04:54.023946+0000","last_became_peered":"2020-11-20T12:04:54.023946+0000","last_unstale":"2020-11-20T12:04:54.024608+0000","last_undegraded":"2020-11-20T12:04:54.024608+0000","last_fullsized":"2020-11-20T12:04:54.024608+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.019334+0000","last_change":"2020-11-20T12:04:51.447214+0000","last_active":"2020-11-20T12:04:54.019334+0000","last_peered":"2020-11-20T12:04:54.019334+0000","last_clean":"2020-11-20T12:04:54.019334+0000","last_became_active":"2020-11-20T12:04:51.446767+0000","last_became_peered":"2020-11-20T12:04:51.446767+0000","last_unstale":"2020-11-20T12:04:54.019334+0000","last_undegraded":"2020-11-20T12:04:54.019334+0000","last_fullsized":"2020-11-20T12:04:54.019334+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"22'4","reported_seq":"14","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:55.029034+0000","last_change":"2020-11-20T12:04:53.658055+0000","last_active":"2020-11-20T12:04:55.029034+0000","last_peered":"2020-11-20T12:04:55.029034+0000","last_clean":"2020-11-20T12:04:55.029034+0000","last_became_active":"2020-11-20T12:04:53.657639+0000","last_became_peered":"2020-11-20T12:04:53.657639+0000","last_unstale":"2020-11-20T12:04:55.029034+0000","last_undegraded":"2020-11-20T12:04:55.029034+0000","last_fullsized":"2020-11-20T12:04:55.029034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":2747,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40,"ondisk_log_size":40,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235868,"kb_used_data":93463512,"kb_used_omap":71,"kb_used_meta":0,"kb_avail":93463512,"statfs":{"total":95948165120,"available":95706636288,"internally_reserved":0,"allocated":95706636288,"data_stored":95706636288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":73586,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6779999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2869999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0790000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.79}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4529999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6139999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0049999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":10,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235904,"kb_used_data":93463484,"kb_used_omap":98,"kb_used_meta":0,"kb_avail":93463476,"statfs":{"total":95948165120,"available":95706599424,"internally_reserved":8192,"allocated":95706607616,"data_stored":95706607616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101359,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.984}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8450000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1909999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9510000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.8559999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.6259999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.4550000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235868,"kb_used_data":93463512,"kb_used_omap":52,"kb_used_meta":0,"kb_avail":93463512,"statfs":{"total":95948165120,"available":95706636288,"internally_reserved":0,"allocated":95706636288,"data_stored":95706636288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53960,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4850000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.218}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.621}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7639999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.915}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5830000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.484}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235872,"kb_used_data":93463508,"kb_used_omap":68,"kb_used_meta":0,"kb_avail":93463508,"statfs":{"total":95948165120,"available":95706632192,"internally_reserved":0,"allocated":95706632192,"data_stored":95706632192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":70475,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7949999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5910000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.944}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7389999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.4649999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.3959999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2549999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235880,"kb_used_data":93463500,"kb_used_omap":76,"kb_used_meta":0,"kb_avail":93463500,"statfs":{"total":95948165120,"available":95706624000,"internally_reserved":0,"allocated":95706624000,"data_stored":95706624000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78075,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1280000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1520000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3660000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6479999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8849999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5209999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4590000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235868,"kb_used_data":93463512,"kb_used_omap":68,"kb_used_meta":0,"kb_avail":93463512,"statfs":{"total":95948165120,"available":95706636288,"internally_reserved":0,"allocated":95706636288,"data_stored":95706636288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":70478,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.554}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.48}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2120000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.395}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5939999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1019999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.286}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235876,"kb_used_data":93463504,"kb_used_omap":75,"kb_used_meta":0,"kb_avail":93463504,"statfs":{"total":95948165120,"available":95706628096,"internally_reserved":0,"allocated":95706628096,"data_stored":95706628096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77210,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.431}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.478}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2440000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5910000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.399}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1259999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2170000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":7,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235888,"kb_used_data":93463500,"kb_used_omap":90,"kb_used_meta":0,"kb_avail":93463492,"statfs":{"total":95948165120,"available":95706615808,"internally_reserved":8192,"allocated":95706624000,"data_stored":95706624000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92203,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4180000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9410000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.605}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9729999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8519999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5720000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.6259999999999999}]}]}]}} 2020-11-20T12:05:18.805 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-20T12:05:18.928 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.922+0000 7fd8db84c700 1 Processor -- start 2020-11-20T12:05:18.928 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.922+0000 7fd8db84c700 1 -- start start 2020-11-20T12:05:18.930 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.922+0000 7fd8db84c700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd8d40a9290 0x7fd8d4134100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:18.930 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.922+0000 7fd8db84c700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8d40663e0 con 0x7fd8d40a9e20 2020-11-20T12:05:18.930 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.922+0000 7fd8db84c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8d4067b80 con 0x7fd8d4068020 2020-11-20T12:05:18.931 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.922+0000 7fd8db84c700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8d40a9970 con 0x7fd8d40a9290 2020-11-20T12:05:18.931 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.922+0000 7fd8d8de7700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd8d4068020 0x7fd8d41339f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50758/0 (socket says 172.21.15.189:50758) 2020-11-20T12:05:18.931 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.922+0000 7fd8d8de7700 1 -- 172.21.15.189:0/3163994747 learned_addr learned my addr 172.21.15.189:0/3163994747 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:18.931 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.922+0000 7fd8d9de9700 1 --2- 172.21.15.189:0/3163994747 >> v2:172.21.15.47:3300/0 conn(0x7fd8d40a9290 0x7fd8d4134100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:18.932 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.922+0000 7fd8d3fff700 1 -- 172.21.15.189:0/3163994747 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009710908 0 0) 0x7fd8d4067b80 con 0x7fd8d4068020 2020-11-20T12:05:18.933 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.922+0000 7fd8d3fff700 1 -- 172.21.15.189:0/3163994747 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8b8003240 con 0x7fd8d4068020 2020-11-20T12:05:18.933 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.922+0000 7fd8d3fff700 1 -- 172.21.15.189:0/3163994747 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4072641729 0 0) 0x7fd8d40663e0 con 0x7fd8d40a9e20 2020-11-20T12:05:18.933 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.926+0000 7fd8d3fff700 1 -- 172.21.15.189:0/3163994747 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8d4067b80 con 0x7fd8d40a9e20 2020-11-20T12:05:18.933 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.926+0000 7fd8d9de9700 1 -- 172.21.15.189:0/3163994747 >> v1:172.21.15.189:6789/0 conn(0x7fd8d4068020 legacy=0x7fd8d41339f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:18.933 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.926+0000 7fd8d9de9700 1 -- 172.21.15.189:0/3163994747 >> v1:172.21.15.47:6789/0 conn(0x7fd8d40a9e20 legacy=0x7fd8d40a83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:18.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.926+0000 7fd8d9de9700 1 -- 172.21.15.189:0/3163994747 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d4066680 con 0x7fd8d40a9290 2020-11-20T12:05:18.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.926+0000 7fd8d9de9700 1 --2- 172.21.15.189:0/3163994747 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8d40a9290 0x7fd8d4134100 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fd8cc00cb30 tx=0x7fd8cc006e80).ready entity=mon.0 client_cookie=adf0ad55f047dc78 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:18.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.926+0000 7fd8d3fff700 1 -- 172.21.15.189:0/3163994747 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc00a6b0 con 0x7fd8d40a9290 2020-11-20T12:05:18.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.926+0000 7fd8d3fff700 1 -- 172.21.15.189:0/3163994747 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8cc00a810 con 0x7fd8d40a9290 2020-11-20T12:05:18.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.926+0000 7fd8d3fff700 1 -- 172.21.15.189:0/3163994747 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc00ab90 con 0x7fd8d40a9290 2020-11-20T12:05:18.935 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.926+0000 7fd8db84c700 1 -- 172.21.15.189:0/3163994747 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8d40a9290 msgr2=0x7fd8d4134100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:18.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.926+0000 7fd8db84c700 1 --2- 172.21.15.189:0/3163994747 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8d40a9290 0x7fd8d4134100 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fd8cc00cb30 tx=0x7fd8cc006e80).stop 2020-11-20T12:05:18.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.926+0000 7fd8db84c700 1 -- 172.21.15.189:0/3163994747 shutdown_connections 2020-11-20T12:05:18.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.926+0000 7fd8db84c700 1 --2- 172.21.15.189:0/3163994747 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8d40a9290 0x7fd8d4134100 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:18.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.926+0000 7fd8db84c700 1 -- 172.21.15.189:0/3163994747 >> 172.21.15.189:0/3163994747 conn(0x7fd8d40627f0 msgr2=0x7fd8d4065870 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:18.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.926+0000 7fd8db84c700 1 -- 172.21.15.189:0/3163994747 shutdown_connections 2020-11-20T12:05:18.937 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.926+0000 7fd8db84c700 1 -- 172.21.15.189:0/3163994747 wait complete. 2020-11-20T12:05:18.937 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.930+0000 7fd8db84c700 1 Processor -- start 2020-11-20T12:05:18.937 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.930+0000 7fd8db84c700 1 -- start start 2020-11-20T12:05:18.939 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.930+0000 7fd8db84c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d4068020 0x7fd8d4139750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:18.939 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.930+0000 7fd8db84c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8d40a9290 0x7fd8d4134920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:18.939 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.930+0000 7fd8db84c700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd8d40a9e20 0x7fd8d41454e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:18.939 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.930+0000 7fd8db84c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d40ac830 con 0x7fd8d40a9290 2020-11-20T12:05:18.940 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.930+0000 7fd8db84c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d412c1d0 con 0x7fd8d4068020 2020-11-20T12:05:18.940 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.930+0000 7fd8db84c700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d4148fb0 con 0x7fd8d40a9e20 2020-11-20T12:05:18.940 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.930+0000 7fd8d95e8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d4068020 0x7fd8d4139750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:18.940 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.930+0000 7fd8d95e8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d4068020 0x7fd8d4139750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49142/0 (socket says 172.21.15.189:49142) 2020-11-20T12:05:18.940 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.930+0000 7fd8d95e8700 1 -- 172.21.15.189:0/1471811596 learned_addr learned my addr 172.21.15.189:0/1471811596 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:18.941 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.930+0000 7fd8d8de7700 1 --2- 172.21.15.189:0/1471811596 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8d40a9290 0x7fd8d4134920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:18.941 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.930+0000 7fd8d9de9700 1 --2- 172.21.15.189:0/1471811596 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd8d40a9e20 0x7fd8d41454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:18.941 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.930+0000 7fd8d95e8700 1 -- 172.21.15.189:0/1471811596 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd8d40a9e20 msgr2=0x7fd8d41454e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:18.941 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.930+0000 7fd8d95e8700 1 --2- 172.21.15.189:0/1471811596 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd8d40a9e20 0x7fd8d41454e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:18.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.930+0000 7fd8d95e8700 1 -- 172.21.15.189:0/1471811596 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8d40a9290 msgr2=0x7fd8d4134920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:18.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.930+0000 7fd8d95e8700 1 --2- 172.21.15.189:0/1471811596 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8d40a9290 0x7fd8d4134920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:18.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.930+0000 7fd8d95e8700 1 -- 172.21.15.189:0/1471811596 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d4149250 con 0x7fd8d4068020 2020-11-20T12:05:18.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.930+0000 7fd8d95e8700 1 --2- 172.21.15.189:0/1471811596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d4068020 0x7fd8d4139750 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fd8c800ca60 tx=0x7fd8c8007ef0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:18.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.930+0000 7fd8d17fa700 1 -- 172.21.15.189:0/1471811596 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c800ce30 con 0x7fd8d4068020 2020-11-20T12:05:18.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.934+0000 7fd8d17fa700 1 -- 172.21.15.189:0/1471811596 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c8002030 con 0x7fd8d4068020 2020-11-20T12:05:18.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.934+0000 7fd8db84c700 1 -- 172.21.15.189:0/1471811596 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8d41494f0 con 0x7fd8d4068020 2020-11-20T12:05:18.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.934+0000 7fd8d17fa700 1 -- 172.21.15.189:0/1471811596 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c800a7b0 con 0x7fd8d4068020 2020-11-20T12:05:18.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.934+0000 7fd8db84c700 1 -- 172.21.15.189:0/1471811596 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8d4148a30 con 0x7fd8d4068020 2020-11-20T12:05:18.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.934+0000 7fd8db84c700 1 -- 172.21.15.189:0/1471811596 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8d4044f60 con 0x7fd8d4068020 2020-11-20T12:05:18.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.934+0000 7fd8d17fa700 1 -- 172.21.15.189:0/1471811596 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fd8c8019030 con 0x7fd8d4068020 2020-11-20T12:05:18.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.934+0000 7fd8d17fa700 1 --2- 172.21.15.189:0/1471811596 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd8ac0557c0 0x7fd8ac057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:18.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.934+0000 7fd8d17fa700 1 -- 172.21.15.189:0/1471811596 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd8c806dd30 con 0x7fd8d4068020 2020-11-20T12:05:18.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.934+0000 7fd8d8de7700 1 --2- 172.21.15.189:0/1471811596 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd8ac0557c0 0x7fd8ac057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:18.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.934+0000 7fd8d8de7700 1 --2- 172.21.15.189:0/1471811596 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd8ac0557c0 0x7fd8ac057c80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fd8d40acfe0 tx=0x7fd8c00027f0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:18.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:18.942+0000 7fd8d17fa700 1 -- 172.21.15.189:0/1471811596 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd8c8043bd0 con 0x7fd8d4068020 2020-11-20T12:05:19.094 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.086+0000 7fd8db84c700 1 -- 172.21.15.189:0/1471811596 --> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd8d40aab70 con 0x7fd8ac0557c0 2020-11-20T12:05:19.102 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.094+0000 7fd8d17fa700 1 -- 172.21.15.189:0/1471811596 <== mgr.4105 v2:172.21.15.189:6832/11671 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+82253 (secure 0 0 0) 0x7fd8d40aab70 con 0x7fd8ac0557c0 2020-11-20T12:05:19.103 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:05:19.105 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.098+0000 7fd8db84c700 1 -- 172.21.15.189:0/1471811596 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd8ac0557c0 msgr2=0x7fd8ac057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:19.105 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.098+0000 7fd8db84c700 1 --2- 172.21.15.189:0/1471811596 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd8ac0557c0 0x7fd8ac057c80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fd8d40acfe0 tx=0x7fd8c00027f0).stop 2020-11-20T12:05:19.105 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.098+0000 7fd8db84c700 1 -- 172.21.15.189:0/1471811596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d4068020 msgr2=0x7fd8d4139750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:19.105 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.098+0000 7fd8db84c700 1 --2- 172.21.15.189:0/1471811596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d4068020 0x7fd8d4139750 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fd8c800ca60 tx=0x7fd8c8007ef0).stop 2020-11-20T12:05:19.106 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.098+0000 7fd8db84c700 1 -- 172.21.15.189:0/1471811596 shutdown_connections 2020-11-20T12:05:19.106 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.098+0000 7fd8db84c700 1 --2- 172.21.15.189:0/1471811596 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd8d40a9e20 0x7fd8d41454e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.106 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.098+0000 7fd8db84c700 1 --2- 172.21.15.189:0/1471811596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d4068020 0x7fd8d4139750 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.106 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.098+0000 7fd8db84c700 1 --2- 172.21.15.189:0/1471811596 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd8ac0557c0 0x7fd8ac057c80 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.106 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.098+0000 7fd8db84c700 1 --2- 172.21.15.189:0/1471811596 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8d40a9290 0x7fd8d4134920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.107 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.098+0000 7fd8db84c700 1 -- 172.21.15.189:0/1471811596 >> 172.21.15.189:0/1471811596 conn(0x7fd8d40627f0 msgr2=0x7fd8d405ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:19.107 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.098+0000 7fd8db84c700 1 -- 172.21.15.189:0/1471811596 shutdown_connections 2020-11-20T12:05:19.107 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.098+0000 7fd8db84c700 1 -- 172.21.15.189:0/1471811596 wait complete. 2020-11-20T12:05:19.107 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2020-11-20T12:05:19.120 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2020-11-20T12:05:17.342765+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2747,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":1887024,"kb_used_data":747708032,"kb_used_omap":602,"kb_used_meta":0,"kb_avail":747708016,"statfs":{"total":767585320960,"available":765653008384,"internally_reserved":16384,"allocated":765653024768,"data_stored":765653024768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":617346,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":443,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003773"},"pg_stats":[{"pgid":"4.2","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.025896+0000","last_change":"2020-11-20T12:04:54.024234+0000","last_active":"2020-11-20T12:04:54.025896+0000","last_peered":"2020-11-20T12:04:54.025896+0000","last_clean":"2020-11-20T12:04:54.025896+0000","last_became_active":"2020-11-20T12:04:54.023862+0000","last_became_peered":"2020-11-20T12:04:54.023862+0000","last_unstale":"2020-11-20T12:04:54.025896+0000","last_undegraded":"2020-11-20T12:04:54.025896+0000","last_fullsized":"2020-11-20T12:04:54.025896+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.025135+0000","last_change":"2020-11-20T12:04:51.104716+0000","last_active":"2020-11-20T12:04:54.025135+0000","last_peered":"2020-11-20T12:04:54.025135+0000","last_clean":"2020-11-20T12:04:54.025135+0000","last_became_active":"2020-11-20T12:04:51.104301+0000","last_became_peered":"2020-11-20T12:04:51.104301+0000","last_unstale":"2020-11-20T12:04:54.025135+0000","last_undegraded":"2020-11-20T12:04:54.025135+0000","last_fullsized":"2020-11-20T12:04:54.025135+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"21'6","reported_seq":"16","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.039919+0000","last_change":"2020-11-20T12:04:53.664474+0000","last_active":"2020-11-20T12:04:54.039919+0000","last_peered":"2020-11-20T12:04:54.039919+0000","last_clean":"2020-11-20T12:04:54.039919+0000","last_became_active":"2020-11-20T12:04:53.664073+0000","last_became_peered":"2020-11-20T12:04:53.664073+0000","last_unstale":"2020-11-20T12:04:54.039919+0000","last_undegraded":"2020-11-20T12:04:54.039919+0000","last_fullsized":"2020-11-20T12:04:54.039919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.037879+0000","last_change":"2020-11-20T12:04:54.033863+0000","last_active":"2020-11-20T12:04:54.037879+0000","last_peered":"2020-11-20T12:04:54.037879+0000","last_clean":"2020-11-20T12:04:54.037879+0000","last_became_active":"2020-11-20T12:04:54.032723+0000","last_became_peered":"2020-11-20T12:04:54.032723+0000","last_unstale":"2020-11-20T12:04:54.037879+0000","last_undegraded":"2020-11-20T12:04:54.037879+0000","last_fullsized":"2020-11-20T12:04:54.037879+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.018806+0000","last_change":"2020-11-20T12:04:51.107483+0000","last_active":"2020-11-20T12:04:54.018806+0000","last_peered":"2020-11-20T12:04:54.018806+0000","last_clean":"2020-11-20T12:04:54.018806+0000","last_became_active":"2020-11-20T12:04:51.106350+0000","last_became_peered":"2020-11-20T12:04:51.106350+0000","last_unstale":"2020-11-20T12:04:54.018806+0000","last_undegraded":"2020-11-20T12:04:54.018806+0000","last_fullsized":"2020-11-20T12:04:54.018806+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"21'5","reported_seq":"15","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.034249+0000","last_change":"2020-11-20T12:04:53.660754+0000","last_active":"2020-11-20T12:04:54.034249+0000","last_peered":"2020-11-20T12:04:54.034249+0000","last_clean":"2020-11-20T12:04:54.034249+0000","last_became_active":"2020-11-20T12:04:53.660354+0000","last_became_peered":"2020-11-20T12:04:53.660354+0000","last_unstale":"2020-11-20T12:04:54.034249+0000","last_undegraded":"2020-11-20T12:04:54.034249+0000","last_fullsized":"2020-11-20T12:04:54.034249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.036255+0000","last_change":"2020-11-20T12:04:54.036030+0000","last_active":"2020-11-20T12:04:54.036255+0000","last_peered":"2020-11-20T12:04:54.036255+0000","last_clean":"2020-11-20T12:04:54.036255+0000","last_became_active":"2020-11-20T12:04:54.035638+0000","last_became_peered":"2020-11-20T12:04:54.035638+0000","last_unstale":"2020-11-20T12:04:54.036255+0000","last_undegraded":"2020-11-20T12:04:54.036255+0000","last_fullsized":"2020-11-20T12:04:54.036255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.028309+0000","last_change":"2020-11-20T12:04:50.499140+0000","last_active":"2020-11-20T12:04:54.028309+0000","last_peered":"2020-11-20T12:04:54.028309+0000","last_clean":"2020-11-20T12:04:54.028309+0000","last_became_active":"2020-11-20T12:04:50.498705+0000","last_became_peered":"2020-11-20T12:04:50.498705+0000","last_unstale":"2020-11-20T12:04:54.028309+0000","last_undegraded":"2020-11-20T12:04:54.028309+0000","last_fullsized":"2020-11-20T12:04:54.028309+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"21'8","reported_seq":"18","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.040833+0000","last_change":"2020-11-20T12:04:53.657720+0000","last_active":"2020-11-20T12:04:54.040833+0000","last_peered":"2020-11-20T12:04:54.040833+0000","last_clean":"2020-11-20T12:04:54.040833+0000","last_became_active":"2020-11-20T12:04:53.657238+0000","last_became_peered":"2020-11-20T12:04:53.657238+0000","last_unstale":"2020-11-20T12:04:54.040833+0000","last_undegraded":"2020-11-20T12:04:54.040833+0000","last_fullsized":"2020-11-20T12:04:54.040833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.345488+0000","last_change":"2020-11-20T12:04:54.345320+0000","last_active":"2020-11-20T12:04:54.345488+0000","last_peered":"2020-11-20T12:04:54.345488+0000","last_clean":"2020-11-20T12:04:54.345488+0000","last_became_active":"2020-11-20T12:04:54.344995+0000","last_became_peered":"2020-11-20T12:04:54.344995+0000","last_unstale":"2020-11-20T12:04:54.345488+0000","last_undegraded":"2020-11-20T12:04:54.345488+0000","last_fullsized":"2020-11-20T12:04:54.345488+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.019834+0000","last_change":"2020-11-20T12:04:51.425413+0000","last_active":"2020-11-20T12:04:54.019834+0000","last_peered":"2020-11-20T12:04:54.019834+0000","last_clean":"2020-11-20T12:04:54.019834+0000","last_became_active":"2020-11-20T12:04:51.424595+0000","last_became_peered":"2020-11-20T12:04:51.424595+0000","last_unstale":"2020-11-20T12:04:54.019834+0000","last_undegraded":"2020-11-20T12:04:54.019834+0000","last_fullsized":"2020-11-20T12:04:54.019834+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"14'2","reported_seq":"18","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.344529+0000","last_change":"2020-11-20T12:04:44.090086+0000","last_active":"2020-11-20T12:04:54.344529+0000","last_peered":"2020-11-20T12:04:54.344529+0000","last_clean":"2020-11-20T12:04:54.344529+0000","last_became_active":"2020-11-20T12:04:44.089120+0000","last_became_peered":"2020-11-20T12:04:44.089120+0000","last_unstale":"2020-11-20T12:04:54.344529+0000","last_undegraded":"2020-11-20T12:04:54.344529+0000","last_fullsized":"2020-11-20T12:04:54.344529+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:42.003758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:42.003758+0000","last_clean_scrub_stamp":"2020-11-20T12:04:42.003758+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"21'1","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.034984+0000","last_change":"2020-11-20T12:04:53.660727+0000","last_active":"2020-11-20T12:04:54.034984+0000","last_peered":"2020-11-20T12:04:54.034984+0000","last_clean":"2020-11-20T12:04:54.034984+0000","last_became_active":"2020-11-20T12:04:53.660424+0000","last_became_peered":"2020-11-20T12:04:53.660424+0000","last_unstale":"2020-11-20T12:04:54.034984+0000","last_undegraded":"2020-11-20T12:04:54.034984+0000","last_fullsized":"2020-11-20T12:04:54.034984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.038696+0000","last_change":"2020-11-20T12:04:54.038479+0000","last_active":"2020-11-20T12:04:54.038696+0000","last_peered":"2020-11-20T12:04:54.038696+0000","last_clean":"2020-11-20T12:04:54.038696+0000","last_became_active":"2020-11-20T12:04:54.037927+0000","last_became_peered":"2020-11-20T12:04:54.037927+0000","last_unstale":"2020-11-20T12:04:54.038696+0000","last_undegraded":"2020-11-20T12:04:54.038696+0000","last_fullsized":"2020-11-20T12:04:54.038696+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.020094+0000","last_change":"2020-11-20T12:04:50.499789+0000","last_active":"2020-11-20T12:04:54.020094+0000","last_peered":"2020-11-20T12:04:54.020094+0000","last_clean":"2020-11-20T12:04:54.020094+0000","last_became_active":"2020-11-20T12:04:50.499414+0000","last_became_peered":"2020-11-20T12:04:50.499414+0000","last_unstale":"2020-11-20T12:04:54.020094+0000","last_undegraded":"2020-11-20T12:04:54.020094+0000","last_fullsized":"2020-11-20T12:04:54.020094+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"10","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.019676+0000","last_change":"2020-11-20T12:04:53.665538+0000","last_active":"2020-11-20T12:04:54.019676+0000","last_peered":"2020-11-20T12:04:54.019676+0000","last_clean":"2020-11-20T12:04:54.019676+0000","last_became_active":"2020-11-20T12:04:53.664812+0000","last_became_peered":"2020-11-20T12:04:53.664812+0000","last_unstale":"2020-11-20T12:04:54.019676+0000","last_undegraded":"2020-11-20T12:04:54.019676+0000","last_fullsized":"2020-11-20T12:04:54.019676+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.039072+0000","last_change":"2020-11-20T12:04:54.038559+0000","last_active":"2020-11-20T12:04:54.039072+0000","last_peered":"2020-11-20T12:04:54.039072+0000","last_clean":"2020-11-20T12:04:54.039072+0000","last_became_active":"2020-11-20T12:04:54.038134+0000","last_became_peered":"2020-11-20T12:04:54.038134+0000","last_unstale":"2020-11-20T12:04:54.039072+0000","last_undegraded":"2020-11-20T12:04:54.039072+0000","last_fullsized":"2020-11-20T12:04:54.039072+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.019201+0000","last_change":"2020-11-20T12:04:50.500191+0000","last_active":"2020-11-20T12:04:54.019201+0000","last_peered":"2020-11-20T12:04:54.019201+0000","last_clean":"2020-11-20T12:04:54.019201+0000","last_became_active":"2020-11-20T12:04:50.499854+0000","last_became_peered":"2020-11-20T12:04:50.499854+0000","last_unstale":"2020-11-20T12:04:54.019201+0000","last_undegraded":"2020-11-20T12:04:54.019201+0000","last_fullsized":"2020-11-20T12:04:54.019201+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"22'10","reported_seq":"20","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:05:11.751763+0000","last_change":"2020-11-20T12:04:53.665307+0000","last_active":"2020-11-20T12:05:11.751763+0000","last_peered":"2020-11-20T12:05:11.751763+0000","last_clean":"2020-11-20T12:05:11.751763+0000","last_became_active":"2020-11-20T12:04:53.664857+0000","last_became_peered":"2020-11-20T12:04:53.664857+0000","last_unstale":"2020-11-20T12:05:11.751763+0000","last_undegraded":"2020-11-20T12:05:11.751763+0000","last_fullsized":"2020-11-20T12:05:11.751763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1978,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.346166+0000","last_change":"2020-11-20T12:04:54.345851+0000","last_active":"2020-11-20T12:04:54.346166+0000","last_peered":"2020-11-20T12:04:54.346166+0000","last_clean":"2020-11-20T12:04:54.346166+0000","last_became_active":"2020-11-20T12:04:54.345040+0000","last_became_peered":"2020-11-20T12:04:54.345040+0000","last_unstale":"2020-11-20T12:04:54.346166+0000","last_undegraded":"2020-11-20T12:04:54.346166+0000","last_fullsized":"2020-11-20T12:04:54.346166+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"21'6","reported_seq":"16","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.039735+0000","last_change":"2020-11-20T12:04:53.663035+0000","last_active":"2020-11-20T12:04:54.039735+0000","last_peered":"2020-11-20T12:04:54.039735+0000","last_clean":"2020-11-20T12:04:54.039735+0000","last_became_active":"2020-11-20T12:04:53.662666+0000","last_became_peered":"2020-11-20T12:04:53.662666+0000","last_unstale":"2020-11-20T12:04:54.039735+0000","last_undegraded":"2020-11-20T12:04:54.039735+0000","last_fullsized":"2020-11-20T12:04:54.039735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.025490+0000","last_change":"2020-11-20T12:04:51.447405+0000","last_active":"2020-11-20T12:04:54.025490+0000","last_peered":"2020-11-20T12:04:54.025490+0000","last_clean":"2020-11-20T12:04:54.025490+0000","last_became_active":"2020-11-20T12:04:51.446352+0000","last_became_peered":"2020-11-20T12:04:51.446352+0000","last_unstale":"2020-11-20T12:04:54.025490+0000","last_undegraded":"2020-11-20T12:04:54.025490+0000","last_fullsized":"2020-11-20T12:04:54.025490+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.024608+0000","last_change":"2020-11-20T12:04:54.024326+0000","last_active":"2020-11-20T12:04:54.024608+0000","last_peered":"2020-11-20T12:04:54.024608+0000","last_clean":"2020-11-20T12:04:54.024608+0000","last_became_active":"2020-11-20T12:04:54.023946+0000","last_became_peered":"2020-11-20T12:04:54.023946+0000","last_unstale":"2020-11-20T12:04:54.024608+0000","last_undegraded":"2020-11-20T12:04:54.024608+0000","last_fullsized":"2020-11-20T12:04:54.024608+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.019334+0000","last_change":"2020-11-20T12:04:51.447214+0000","last_active":"2020-11-20T12:04:54.019334+0000","last_peered":"2020-11-20T12:04:54.019334+0000","last_clean":"2020-11-20T12:04:54.019334+0000","last_became_active":"2020-11-20T12:04:51.446767+0000","last_became_peered":"2020-11-20T12:04:51.446767+0000","last_unstale":"2020-11-20T12:04:54.019334+0000","last_undegraded":"2020-11-20T12:04:54.019334+0000","last_fullsized":"2020-11-20T12:04:54.019334+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"22'4","reported_seq":"14","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:55.029034+0000","last_change":"2020-11-20T12:04:53.658055+0000","last_active":"2020-11-20T12:04:55.029034+0000","last_peered":"2020-11-20T12:04:55.029034+0000","last_clean":"2020-11-20T12:04:55.029034+0000","last_became_active":"2020-11-20T12:04:53.657639+0000","last_became_peered":"2020-11-20T12:04:53.657639+0000","last_unstale":"2020-11-20T12:04:55.029034+0000","last_undegraded":"2020-11-20T12:04:55.029034+0000","last_fullsized":"2020-11-20T12:04:55.029034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":2747,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40,"ondisk_log_size":40,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235868,"kb_used_data":93463512,"kb_used_omap":71,"kb_used_meta":0,"kb_avail":93463512,"statfs":{"total":95948165120,"available":95706636288,"internally_reserved":0,"allocated":95706636288,"data_stored":95706636288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":73586,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6779999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2869999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0790000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.79}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4529999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6139999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0049999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":10,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235904,"kb_used_data":93463484,"kb_used_omap":98,"kb_used_meta":0,"kb_avail":93463476,"statfs":{"total":95948165120,"available":95706599424,"internally_reserved":8192,"allocated":95706607616,"data_stored":95706607616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101359,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.984}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8450000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1909999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9510000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.8559999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.6259999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.4550000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235868,"kb_used_data":93463512,"kb_used_omap":52,"kb_used_meta":0,"kb_avail":93463512,"statfs":{"total":95948165120,"available":95706636288,"internally_reserved":0,"allocated":95706636288,"data_stored":95706636288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53960,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4850000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.218}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.621}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7639999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.915}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5830000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.484}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235872,"kb_used_data":93463508,"kb_used_omap":68,"kb_used_meta":0,"kb_avail":93463508,"statfs":{"total":95948165120,"available":95706632192,"internally_reserved":0,"allocated":95706632192,"data_stored":95706632192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":70475,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7949999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5910000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.944}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7389999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.4649999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.3959999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2549999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235880,"kb_used_data":93463500,"kb_used_omap":76,"kb_used_meta":0,"kb_avail":93463500,"statfs":{"total":95948165120,"available":95706624000,"internally_reserved":0,"allocated":95706624000,"data_stored":95706624000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78075,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1280000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1520000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3660000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6479999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8849999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5209999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4590000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235868,"kb_used_data":93463512,"kb_used_omap":68,"kb_used_meta":0,"kb_avail":93463512,"statfs":{"total":95948165120,"available":95706636288,"internally_reserved":0,"allocated":95706636288,"data_stored":95706636288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":70478,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.554}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.48}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2120000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.395}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5939999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1019999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.286}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235876,"kb_used_data":93463504,"kb_used_omap":75,"kb_used_meta":0,"kb_avail":93463504,"statfs":{"total":95948165120,"available":95706628096,"internally_reserved":0,"allocated":95706628096,"data_stored":95706628096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77210,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.431}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.478}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2440000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5910000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.399}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1259999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2170000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":7,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235888,"kb_used_data":93463500,"kb_used_omap":90,"kb_used_meta":0,"kb_avail":93463492,"statfs":{"total":95948165120,"available":95706615808,"internally_reserved":8192,"allocated":95706624000,"data_stored":95706624000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92203,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4180000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9410000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.605}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9729999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8519999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5720000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.6259999999999999}]}]}]}} 2020-11-20T12:05:19.122 INFO:tasks.mds_thrash.ceph_manager:clean! 2020-11-20T12:05:19.123 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-20T12:05:19.243 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296e875700 1 Processor -- start 2020-11-20T12:05:19.244 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296e875700 1 -- start start 2020-11-20T12:05:19.247 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296e875700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f296813a680 0x7f296813ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.247 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296e875700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f296813e320 con 0x7f296813b000 2020-11-20T12:05:19.248 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296e875700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2968066800 con 0x7f29680a8840 2020-11-20T12:05:19.248 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296e875700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2968066aa0 con 0x7f296813a680 2020-11-20T12:05:19.248 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296d873700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f29680a8840 0x7f296813dc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50770/0 (socket says 172.21.15.189:50770) 2020-11-20T12:05:19.248 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296d873700 1 -- 172.21.15.189:0/3047970664 learned_addr learned my addr 172.21.15.189:0/3047970664 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:19.249 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296d072700 1 --2- 172.21.15.189:0/3047970664 >> v2:172.21.15.47:3300/0 conn(0x7f296813a680 0x7f296813ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.249 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296c871700 1 -- 172.21.15.189:0/3047970664 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1887969499 0 0) 0x7f2968066800 con 0x7f29680a8840 2020-11-20T12:05:19.249 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296c871700 1 -- 172.21.15.189:0/3047970664 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2948003240 con 0x7f29680a8840 2020-11-20T12:05:19.249 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296c871700 1 -- 172.21.15.189:0/3047970664 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2182155731 0 0) 0x7f2948003240 con 0x7f29680a8840 2020-11-20T12:05:19.250 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296c871700 1 -- 172.21.15.189:0/3047970664 >> v2:172.21.15.47:3300/0 conn(0x7f296813a680 msgr2=0x7f296813ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:19.250 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296c871700 1 --2- 172.21.15.189:0/3047970664 >> v2:172.21.15.47:3300/0 conn(0x7f296813a680 0x7f296813ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.250 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296c871700 1 -- 172.21.15.189:0/3047970664 >> v1:172.21.15.47:6789/0 conn(0x7f296813b000 legacy=0x7f29681451c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:19.250 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296c871700 1 -- 172.21.15.189:0/3047970664 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29680672f0 con 0x7f29680a8840 2020-11-20T12:05:19.250 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296c871700 1 -- 172.21.15.189:0/3047970664 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f2960001860 con 0x7f29680a8840 2020-11-20T12:05:19.251 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296d072700 1 --2- 172.21.15.189:0/3047970664 >> v2:172.21.15.47:3300/0 conn(0x7f296813a680 0x7f296813ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:05:19.251 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296c871700 1 -- 172.21.15.189:0/3047970664 >> v1:172.21.15.189:6789/0 conn(0x7f29680a8840 legacy=0x7f296813dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:19.251 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296c871700 1 --2- 172.21.15.189:0/3047970664 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2948003730 0x7f2948003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.251 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296c871700 1 --2- 172.21.15.189:0/3047970664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29480044e0 0x7f2948004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.252 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296c871700 1 --2- 172.21.15.189:0/3047970664 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2948004e50 0x7f294800b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.252 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296c871700 1 -- 172.21.15.189:0/3047970664 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2968066aa0 con 0x7f2948004e50 2020-11-20T12:05:19.252 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296c871700 1 -- 172.21.15.189:0/3047970664 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2948004260 con 0x7f29480044e0 2020-11-20T12:05:19.252 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296c871700 1 -- 172.21.15.189:0/3047970664 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f294800bc20 con 0x7f2948003730 2020-11-20T12:05:19.253 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296d072700 1 --2- 172.21.15.189:0/3047970664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29480044e0 0x7f2948004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.254 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296d873700 1 --2- 172.21.15.189:0/3047970664 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2948003730 0x7f2948003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.254 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296e074700 1 --2- 172.21.15.189:0/3047970664 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2948004e50 0x7f294800b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.254 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296d072700 1 -- 172.21.15.189:0/3047970664 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2948003730 msgr2=0x7f2948003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:19.255 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296d072700 1 --2- 172.21.15.189:0/3047970664 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2948003730 0x7f2948003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.255 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296d072700 1 -- 172.21.15.189:0/3047970664 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2948004e50 msgr2=0x7f294800b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:19.255 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296d072700 1 --2- 172.21.15.189:0/3047970664 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2948004e50 0x7f294800b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.255 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296e074700 1 -- 172.21.15.189:0/3047970664 reap_dead start 2020-11-20T12:05:19.255 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296d072700 1 -- 172.21.15.189:0/3047970664 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2948003240 con 0x7f29480044e0 2020-11-20T12:05:19.256 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296d072700 1 --2- 172.21.15.189:0/3047970664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29480044e0 0x7f2948004960 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f295000be80 tx=0x7f2950006df0).ready entity=mon.1 client_cookie=2c3647e48e7b6633 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:19.256 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.238+0000 7f296c871700 1 -- 172.21.15.189:0/3047970664 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295000b590 con 0x7f29480044e0 2020-11-20T12:05:19.256 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296c871700 1 -- 172.21.15.189:0/3047970664 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f295000ec40 con 0x7f29480044e0 2020-11-20T12:05:19.256 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296c871700 1 -- 172.21.15.189:0/3047970664 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29500123f0 con 0x7f29480044e0 2020-11-20T12:05:19.256 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296e875700 1 -- 172.21.15.189:0/3047970664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29480044e0 msgr2=0x7f2948004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:19.257 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296e875700 1 --2- 172.21.15.189:0/3047970664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29480044e0 0x7f2948004960 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f295000be80 tx=0x7f2950006df0).stop 2020-11-20T12:05:19.257 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296e875700 1 -- 172.21.15.189:0/3047970664 shutdown_connections 2020-11-20T12:05:19.257 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296e875700 1 --2- 172.21.15.189:0/3047970664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29480044e0 0x7f2948004960 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.257 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296e875700 1 -- 172.21.15.189:0/3047970664 >> 172.21.15.189:0/3047970664 conn(0x7f2968060170 msgr2=0x7f2968060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:19.257 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296e875700 1 -- 172.21.15.189:0/3047970664 shutdown_connections 2020-11-20T12:05:19.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296e875700 1 -- 172.21.15.189:0/3047970664 wait complete. 2020-11-20T12:05:19.261 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296e875700 1 Processor -- start 2020-11-20T12:05:19.261 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296e875700 1 -- start start 2020-11-20T12:05:19.261 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296e875700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2968133530 0x7f29680a6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.261 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296e875700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29680ac510 0x7f29680a6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.262 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296e875700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f29680a5f50 0x7f296813ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.262 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296e875700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29680a6e40 con 0x7f29680ac510 2020-11-20T12:05:19.262 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296e875700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29680a3e50 con 0x7f2968133530 2020-11-20T12:05:19.262 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296e875700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29680a40f0 con 0x7f29680a5f50 2020-11-20T12:05:19.262 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296d873700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2968133530 0x7f29680a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296d873700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2968133530 0x7f29680a6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49158/0 (socket says 172.21.15.189:49158) 2020-11-20T12:05:19.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296d873700 1 -- 172.21.15.189:0/2745878967 learned_addr learned my addr 172.21.15.189:0/2745878967 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:19.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296e074700 1 --2- 172.21.15.189:0/2745878967 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f29680a5f50 0x7f296813ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296d873700 1 -- 172.21.15.189:0/2745878967 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f29680a5f50 msgr2=0x7f296813ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:19.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296d072700 1 --2- 172.21.15.189:0/2745878967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29680ac510 0x7f29680a6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296d873700 1 --2- 172.21.15.189:0/2745878967 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f29680a5f50 0x7f296813ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296d873700 1 -- 172.21.15.189:0/2745878967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29680ac510 msgr2=0x7f29680a6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:19.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296d873700 1 --2- 172.21.15.189:0/2745878967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29680ac510 0x7f29680a6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296d873700 1 -- 172.21.15.189:0/2745878967 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29680a4390 con 0x7f2968133530 2020-11-20T12:05:19.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296e074700 1 --2- 172.21.15.189:0/2745878967 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f29680a5f50 0x7f296813ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:19.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296d072700 1 --2- 172.21.15.189:0/2745878967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29680ac510 0x7f29680a6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:19.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296d873700 1 --2- 172.21.15.189:0/2745878967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2968133530 0x7f29680a6460 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f2960004fc0 tx=0x7f2968067480).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:19.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f2965ffb700 1 -- 172.21.15.189:0/2745878967 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296000fed0 con 0x7f2968133530 2020-11-20T12:05:19.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f2965ffb700 1 -- 172.21.15.189:0/2745878967 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2960012350 con 0x7f2968133530 2020-11-20T12:05:19.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296e875700 1 -- 172.21.15.189:0/2745878967 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2968133910 con 0x7f2968133530 2020-11-20T12:05:19.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f2965ffb700 1 -- 172.21.15.189:0/2745878967 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29600126f0 con 0x7f2968133530 2020-11-20T12:05:19.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.242+0000 7f296e875700 1 -- 172.21.15.189:0/2745878967 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f296813cf00 con 0x7f2968133530 2020-11-20T12:05:19.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.246+0000 7f2965ffb700 1 -- 172.21.15.189:0/2745878967 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f296001b030 con 0x7f2968133530 2020-11-20T12:05:19.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.246+0000 7f296e875700 1 -- 172.21.15.189:0/2745878967 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29680a9230 con 0x7f2968133530 2020-11-20T12:05:19.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.246+0000 7f2965ffb700 1 --2- 172.21.15.189:0/2745878967 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f29400557c0 0x7f2940057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.246+0000 7f2965ffb700 1 -- 172.21.15.189:0/2745878967 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f296006e3e0 con 0x7f2968133530 2020-11-20T12:05:19.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.246+0000 7f296d072700 1 --2- 172.21.15.189:0/2745878967 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f29400557c0 0x7f2940057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.246+0000 7f296d072700 1 --2- 172.21.15.189:0/2745878967 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f29400557c0 0x7f2940057c80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f296805b6f0 tx=0x7f2950019000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:19.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.254+0000 7f2965ffb700 1 -- 172.21.15.189:0/2745878967 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2960044280 con 0x7f2968133530 2020-11-20T12:05:19.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.402+0000 7f296e875700 1 -- 172.21.15.189:0/2745878967 --> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f29680446d0 con 0x7f29400557c0 2020-11-20T12:05:19.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.410+0000 7f2965ffb700 1 -- 172.21.15.189:0/2745878967 <== mgr.4105 v2:172.21.15.189:6832/11671 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+82251 (secure 0 0 0) 0x7f29680446d0 con 0x7f29400557c0 2020-11-20T12:05:19.419 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:05:19.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.414+0000 7f296e875700 1 -- 172.21.15.189:0/2745878967 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f29400557c0 msgr2=0x7f2940057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:19.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.414+0000 7f296e875700 1 --2- 172.21.15.189:0/2745878967 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f29400557c0 0x7f2940057c80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f296805b6f0 tx=0x7f2950019000).stop 2020-11-20T12:05:19.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.414+0000 7f296e875700 1 -- 172.21.15.189:0/2745878967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2968133530 msgr2=0x7f29680a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:19.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.414+0000 7f296e875700 1 --2- 172.21.15.189:0/2745878967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2968133530 0x7f29680a6460 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f2960004fc0 tx=0x7f2968067480).stop 2020-11-20T12:05:19.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.414+0000 7f296e875700 1 -- 172.21.15.189:0/2745878967 shutdown_connections 2020-11-20T12:05:19.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.414+0000 7f296e875700 1 --2- 172.21.15.189:0/2745878967 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f29680a5f50 0x7f296813ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.414+0000 7f296e875700 1 --2- 172.21.15.189:0/2745878967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2968133530 0x7f29680a6460 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.414+0000 7f296e875700 1 --2- 172.21.15.189:0/2745878967 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f29400557c0 0x7f2940057c80 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.414+0000 7f296e875700 1 --2- 172.21.15.189:0/2745878967 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29680ac510 0x7f29680a6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.414+0000 7f296e875700 1 -- 172.21.15.189:0/2745878967 >> 172.21.15.189:0/2745878967 conn(0x7f2968060170 msgr2=0x7f2968138c10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:19.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.414+0000 7f296e875700 1 -- 172.21.15.189:0/2745878967 shutdown_connections 2020-11-20T12:05:19.423 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.414+0000 7f296e875700 1 -- 172.21.15.189:0/2745878967 wait complete. 2020-11-20T12:05:19.424 INFO:teuthology.orchestra.run.smithi189.stderr:dumped all 2020-11-20T12:05:19.434 INFO:teuthology.orchestra.run.smithi189.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2020-11-20T12:05:19.343178+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3529,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":1887016,"kb_used_data":747708032,"kb_used_omap":603,"kb_used_meta":0,"kb_avail":747708024,"statfs":{"total":767585320960,"available":765653016576,"internally_reserved":8192,"allocated":765653024768,"data_stored":765653024768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":618484,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1225,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003476"},"pg_stats":[{"pgid":"4.2","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.025896+0000","last_change":"2020-11-20T12:04:54.024234+0000","last_active":"2020-11-20T12:04:54.025896+0000","last_peered":"2020-11-20T12:04:54.025896+0000","last_clean":"2020-11-20T12:04:54.025896+0000","last_became_active":"2020-11-20T12:04:54.023862+0000","last_became_peered":"2020-11-20T12:04:54.023862+0000","last_unstale":"2020-11-20T12:04:54.025896+0000","last_undegraded":"2020-11-20T12:04:54.025896+0000","last_fullsized":"2020-11-20T12:04:54.025896+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.025135+0000","last_change":"2020-11-20T12:04:51.104716+0000","last_active":"2020-11-20T12:04:54.025135+0000","last_peered":"2020-11-20T12:04:54.025135+0000","last_clean":"2020-11-20T12:04:54.025135+0000","last_became_active":"2020-11-20T12:04:51.104301+0000","last_became_peered":"2020-11-20T12:04:51.104301+0000","last_unstale":"2020-11-20T12:04:54.025135+0000","last_undegraded":"2020-11-20T12:04:54.025135+0000","last_fullsized":"2020-11-20T12:04:54.025135+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"21'6","reported_seq":"16","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.039919+0000","last_change":"2020-11-20T12:04:53.664474+0000","last_active":"2020-11-20T12:04:54.039919+0000","last_peered":"2020-11-20T12:04:54.039919+0000","last_clean":"2020-11-20T12:04:54.039919+0000","last_became_active":"2020-11-20T12:04:53.664073+0000","last_became_peered":"2020-11-20T12:04:53.664073+0000","last_unstale":"2020-11-20T12:04:54.039919+0000","last_undegraded":"2020-11-20T12:04:54.039919+0000","last_fullsized":"2020-11-20T12:04:54.039919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.037879+0000","last_change":"2020-11-20T12:04:54.033863+0000","last_active":"2020-11-20T12:04:54.037879+0000","last_peered":"2020-11-20T12:04:54.037879+0000","last_clean":"2020-11-20T12:04:54.037879+0000","last_became_active":"2020-11-20T12:04:54.032723+0000","last_became_peered":"2020-11-20T12:04:54.032723+0000","last_unstale":"2020-11-20T12:04:54.037879+0000","last_undegraded":"2020-11-20T12:04:54.037879+0000","last_fullsized":"2020-11-20T12:04:54.037879+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.018806+0000","last_change":"2020-11-20T12:04:51.107483+0000","last_active":"2020-11-20T12:04:54.018806+0000","last_peered":"2020-11-20T12:04:54.018806+0000","last_clean":"2020-11-20T12:04:54.018806+0000","last_became_active":"2020-11-20T12:04:51.106350+0000","last_became_peered":"2020-11-20T12:04:51.106350+0000","last_unstale":"2020-11-20T12:04:54.018806+0000","last_undegraded":"2020-11-20T12:04:54.018806+0000","last_fullsized":"2020-11-20T12:04:54.018806+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"21'5","reported_seq":"15","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.034249+0000","last_change":"2020-11-20T12:04:53.660754+0000","last_active":"2020-11-20T12:04:54.034249+0000","last_peered":"2020-11-20T12:04:54.034249+0000","last_clean":"2020-11-20T12:04:54.034249+0000","last_became_active":"2020-11-20T12:04:53.660354+0000","last_became_peered":"2020-11-20T12:04:53.660354+0000","last_unstale":"2020-11-20T12:04:54.034249+0000","last_undegraded":"2020-11-20T12:04:54.034249+0000","last_fullsized":"2020-11-20T12:04:54.034249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.036255+0000","last_change":"2020-11-20T12:04:54.036030+0000","last_active":"2020-11-20T12:04:54.036255+0000","last_peered":"2020-11-20T12:04:54.036255+0000","last_clean":"2020-11-20T12:04:54.036255+0000","last_became_active":"2020-11-20T12:04:54.035638+0000","last_became_peered":"2020-11-20T12:04:54.035638+0000","last_unstale":"2020-11-20T12:04:54.036255+0000","last_undegraded":"2020-11-20T12:04:54.036255+0000","last_fullsized":"2020-11-20T12:04:54.036255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.028309+0000","last_change":"2020-11-20T12:04:50.499140+0000","last_active":"2020-11-20T12:04:54.028309+0000","last_peered":"2020-11-20T12:04:54.028309+0000","last_clean":"2020-11-20T12:04:54.028309+0000","last_became_active":"2020-11-20T12:04:50.498705+0000","last_became_peered":"2020-11-20T12:04:50.498705+0000","last_unstale":"2020-11-20T12:04:54.028309+0000","last_undegraded":"2020-11-20T12:04:54.028309+0000","last_fullsized":"2020-11-20T12:04:54.028309+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"21'8","reported_seq":"18","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.040833+0000","last_change":"2020-11-20T12:04:53.657720+0000","last_active":"2020-11-20T12:04:54.040833+0000","last_peered":"2020-11-20T12:04:54.040833+0000","last_clean":"2020-11-20T12:04:54.040833+0000","last_became_active":"2020-11-20T12:04:53.657238+0000","last_became_peered":"2020-11-20T12:04:53.657238+0000","last_unstale":"2020-11-20T12:04:54.040833+0000","last_undegraded":"2020-11-20T12:04:54.040833+0000","last_fullsized":"2020-11-20T12:04:54.040833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.345488+0000","last_change":"2020-11-20T12:04:54.345320+0000","last_active":"2020-11-20T12:04:54.345488+0000","last_peered":"2020-11-20T12:04:54.345488+0000","last_clean":"2020-11-20T12:04:54.345488+0000","last_became_active":"2020-11-20T12:04:54.344995+0000","last_became_peered":"2020-11-20T12:04:54.344995+0000","last_unstale":"2020-11-20T12:04:54.345488+0000","last_undegraded":"2020-11-20T12:04:54.345488+0000","last_fullsized":"2020-11-20T12:04:54.345488+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.019834+0000","last_change":"2020-11-20T12:04:51.425413+0000","last_active":"2020-11-20T12:04:54.019834+0000","last_peered":"2020-11-20T12:04:54.019834+0000","last_clean":"2020-11-20T12:04:54.019834+0000","last_became_active":"2020-11-20T12:04:51.424595+0000","last_became_peered":"2020-11-20T12:04:51.424595+0000","last_unstale":"2020-11-20T12:04:54.019834+0000","last_undegraded":"2020-11-20T12:04:54.019834+0000","last_fullsized":"2020-11-20T12:04:54.019834+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"14'2","reported_seq":"18","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.344529+0000","last_change":"2020-11-20T12:04:44.090086+0000","last_active":"2020-11-20T12:04:54.344529+0000","last_peered":"2020-11-20T12:04:54.344529+0000","last_clean":"2020-11-20T12:04:54.344529+0000","last_became_active":"2020-11-20T12:04:44.089120+0000","last_became_peered":"2020-11-20T12:04:44.089120+0000","last_unstale":"2020-11-20T12:04:54.344529+0000","last_undegraded":"2020-11-20T12:04:54.344529+0000","last_fullsized":"2020-11-20T12:04:54.344529+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:42.003758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:42.003758+0000","last_clean_scrub_stamp":"2020-11-20T12:04:42.003758+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"21'1","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.034984+0000","last_change":"2020-11-20T12:04:53.660727+0000","last_active":"2020-11-20T12:04:54.034984+0000","last_peered":"2020-11-20T12:04:54.034984+0000","last_clean":"2020-11-20T12:04:54.034984+0000","last_became_active":"2020-11-20T12:04:53.660424+0000","last_became_peered":"2020-11-20T12:04:53.660424+0000","last_unstale":"2020-11-20T12:04:54.034984+0000","last_undegraded":"2020-11-20T12:04:54.034984+0000","last_fullsized":"2020-11-20T12:04:54.034984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.038696+0000","last_change":"2020-11-20T12:04:54.038479+0000","last_active":"2020-11-20T12:04:54.038696+0000","last_peered":"2020-11-20T12:04:54.038696+0000","last_clean":"2020-11-20T12:04:54.038696+0000","last_became_active":"2020-11-20T12:04:54.037927+0000","last_became_peered":"2020-11-20T12:04:54.037927+0000","last_unstale":"2020-11-20T12:04:54.038696+0000","last_undegraded":"2020-11-20T12:04:54.038696+0000","last_fullsized":"2020-11-20T12:04:54.038696+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.020094+0000","last_change":"2020-11-20T12:04:50.499789+0000","last_active":"2020-11-20T12:04:54.020094+0000","last_peered":"2020-11-20T12:04:54.020094+0000","last_clean":"2020-11-20T12:04:54.020094+0000","last_became_active":"2020-11-20T12:04:50.499414+0000","last_became_peered":"2020-11-20T12:04:50.499414+0000","last_unstale":"2020-11-20T12:04:54.020094+0000","last_undegraded":"2020-11-20T12:04:54.020094+0000","last_fullsized":"2020-11-20T12:04:54.020094+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"10","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.019676+0000","last_change":"2020-11-20T12:04:53.665538+0000","last_active":"2020-11-20T12:04:54.019676+0000","last_peered":"2020-11-20T12:04:54.019676+0000","last_clean":"2020-11-20T12:04:54.019676+0000","last_became_active":"2020-11-20T12:04:53.664812+0000","last_became_peered":"2020-11-20T12:04:53.664812+0000","last_unstale":"2020-11-20T12:04:54.019676+0000","last_undegraded":"2020-11-20T12:04:54.019676+0000","last_fullsized":"2020-11-20T12:04:54.019676+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.039072+0000","last_change":"2020-11-20T12:04:54.038559+0000","last_active":"2020-11-20T12:04:54.039072+0000","last_peered":"2020-11-20T12:04:54.039072+0000","last_clean":"2020-11-20T12:04:54.039072+0000","last_became_active":"2020-11-20T12:04:54.038134+0000","last_became_peered":"2020-11-20T12:04:54.038134+0000","last_unstale":"2020-11-20T12:04:54.039072+0000","last_undegraded":"2020-11-20T12:04:54.039072+0000","last_fullsized":"2020-11-20T12:04:54.039072+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.019201+0000","last_change":"2020-11-20T12:04:50.500191+0000","last_active":"2020-11-20T12:04:54.019201+0000","last_peered":"2020-11-20T12:04:54.019201+0000","last_clean":"2020-11-20T12:04:54.019201+0000","last_became_active":"2020-11-20T12:04:50.499854+0000","last_became_peered":"2020-11-20T12:04:50.499854+0000","last_unstale":"2020-11-20T12:04:54.019201+0000","last_undegraded":"2020-11-20T12:04:54.019201+0000","last_fullsized":"2020-11-20T12:04:54.019201+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"22'11","reported_seq":"21","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:05:11.752348+0000","last_change":"2020-11-20T12:04:53.665307+0000","last_active":"2020-11-20T12:05:11.752348+0000","last_peered":"2020-11-20T12:05:11.752348+0000","last_clean":"2020-11-20T12:05:11.752348+0000","last_became_active":"2020-11-20T12:04:53.664857+0000","last_became_peered":"2020-11-20T12:04:53.664857+0000","last_unstale":"2020-11-20T12:05:11.752348+0000","last_undegraded":"2020-11-20T12:05:11.752348+0000","last_fullsized":"2020-11-20T12:05:11.752348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2760,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.346166+0000","last_change":"2020-11-20T12:04:54.345851+0000","last_active":"2020-11-20T12:04:54.346166+0000","last_peered":"2020-11-20T12:04:54.346166+0000","last_clean":"2020-11-20T12:04:54.346166+0000","last_became_active":"2020-11-20T12:04:54.345040+0000","last_became_peered":"2020-11-20T12:04:54.345040+0000","last_unstale":"2020-11-20T12:04:54.346166+0000","last_undegraded":"2020-11-20T12:04:54.346166+0000","last_fullsized":"2020-11-20T12:04:54.346166+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"21'6","reported_seq":"16","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.039735+0000","last_change":"2020-11-20T12:04:53.663035+0000","last_active":"2020-11-20T12:04:54.039735+0000","last_peered":"2020-11-20T12:04:54.039735+0000","last_clean":"2020-11-20T12:04:54.039735+0000","last_became_active":"2020-11-20T12:04:53.662666+0000","last_became_peered":"2020-11-20T12:04:53.662666+0000","last_unstale":"2020-11-20T12:04:54.039735+0000","last_undegraded":"2020-11-20T12:04:54.039735+0000","last_fullsized":"2020-11-20T12:04:54.039735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.025490+0000","last_change":"2020-11-20T12:04:51.447405+0000","last_active":"2020-11-20T12:04:54.025490+0000","last_peered":"2020-11-20T12:04:54.025490+0000","last_clean":"2020-11-20T12:04:54.025490+0000","last_became_active":"2020-11-20T12:04:51.446352+0000","last_became_peered":"2020-11-20T12:04:51.446352+0000","last_unstale":"2020-11-20T12:04:54.025490+0000","last_undegraded":"2020-11-20T12:04:54.025490+0000","last_fullsized":"2020-11-20T12:04:54.025490+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":"9","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.024608+0000","last_change":"2020-11-20T12:04:54.024326+0000","last_active":"2020-11-20T12:04:54.024608+0000","last_peered":"2020-11-20T12:04:54.024608+0000","last_clean":"2020-11-20T12:04:54.024608+0000","last_became_active":"2020-11-20T12:04:54.023946+0000","last_became_peered":"2020-11-20T12:04:54.023946+0000","last_unstale":"2020-11-20T12:04:54.024608+0000","last_undegraded":"2020-11-20T12:04:54.024608+0000","last_fullsized":"2020-11-20T12:04:54.024608+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:53.637406+0000","last_clean_scrub_stamp":"2020-11-20T12:04:53.637406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-11-20T12:04:54.019334+0000","last_change":"2020-11-20T12:04:51.447214+0000","last_active":"2020-11-20T12:04:54.019334+0000","last_peered":"2020-11-20T12:04:54.019334+0000","last_clean":"2020-11-20T12:04:54.019334+0000","last_became_active":"2020-11-20T12:04:51.446767+0000","last_became_peered":"2020-11-20T12:04:51.446767+0000","last_unstale":"2020-11-20T12:04:54.019334+0000","last_undegraded":"2020-11-20T12:04:54.019334+0000","last_fullsized":"2020-11-20T12:04:54.019334+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:49.471060+0000","last_clean_scrub_stamp":"2020-11-20T12:04:49.471060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"22'4","reported_seq":"14","reported_epoch":"22","state":"active+clean","last_fresh":"2020-11-20T12:04:55.029034+0000","last_change":"2020-11-20T12:04:53.658055+0000","last_active":"2020-11-20T12:04:55.029034+0000","last_peered":"2020-11-20T12:04:55.029034+0000","last_clean":"2020-11-20T12:04:55.029034+0000","last_became_active":"2020-11-20T12:04:53.657639+0000","last_became_peered":"2020-11-20T12:04:53.657639+0000","last_unstale":"2020-11-20T12:04:55.029034+0000","last_undegraded":"2020-11-20T12:04:55.029034+0000","last_fullsized":"2020-11-20T12:04:55.029034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T12:04:52.633667+0000","last_clean_scrub_stamp":"2020-11-20T12:04:52.633667+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":3529,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235868,"kb_used_data":93463512,"kb_used_omap":71,"kb_used_meta":0,"kb_avail":93463512,"statfs":{"total":95948165120,"available":95706636288,"internally_reserved":0,"allocated":95706636288,"data_stored":95706636288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":73586,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.45}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6850000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0569999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8839999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2050000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.052}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1179999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":10,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235900,"kb_used_data":93463484,"kb_used_omap":99,"kb_used_meta":0,"kb_avail":93463480,"statfs":{"total":95948165120,"available":95706603520,"internally_reserved":4096,"allocated":95706607616,"data_stored":95706607616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101928,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4810000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9059999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.754}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4289999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.23}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5230000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.391}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235868,"kb_used_data":93463512,"kb_used_omap":52,"kb_used_meta":0,"kb_avail":93463512,"statfs":{"total":95948165120,"available":95706636288,"internally_reserved":0,"allocated":95706636288,"data_stored":95706636288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53960,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4850000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.218}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.621}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7639999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.915}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5830000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.484}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235872,"kb_used_data":93463508,"kb_used_omap":68,"kb_used_meta":0,"kb_avail":93463508,"statfs":{"total":95948165120,"available":95706632192,"internally_reserved":0,"allocated":95706632192,"data_stored":95706632192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":70475,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7050000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4510000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.661}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.8250000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3849999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1869999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.4500000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235880,"kb_used_data":93463500,"kb_used_omap":76,"kb_used_meta":0,"kb_avail":93463500,"statfs":{"total":95948165120,"available":95706624000,"internally_reserved":0,"allocated":95706624000,"data_stored":95706624000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78075,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8220000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8380000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2069999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1339999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.169}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0870000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235868,"kb_used_data":93463512,"kb_used_omap":68,"kb_used_meta":0,"kb_avail":93463512,"statfs":{"total":95948165120,"available":95706636288,"internally_reserved":0,"allocated":95706636288,"data_stored":95706636288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":70478,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4930000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4279999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1579999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.7240000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8399999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235876,"kb_used_data":93463504,"kb_used_omap":75,"kb_used_meta":0,"kb_avail":93463504,"statfs":{"total":95948165120,"available":95706628096,"internally_reserved":0,"allocated":95706628096,"data_stored":95706628096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77210,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.548}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6359999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1360000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9590000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3330000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":7,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235884,"kb_used_data":93463500,"kb_used_omap":90,"kb_used_meta":0,"kb_avail":93463496,"statfs":{"total":95948165120,"available":95706619904,"internally_reserved":4096,"allocated":95706624000,"data_stored":95706624000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":92772,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5349999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.48}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6519999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.823}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.0449999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.581}]}]}]}} 2020-11-20T12:05:19.437 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T12:05:19.558 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.550+0000 7fcc9bfff700 1 Processor -- start 2020-11-20T12:05:19.558 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.550+0000 7fcc9bfff700 1 -- start start 2020-11-20T12:05:19.559 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.550+0000 7fcc9bfff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc9c0a9f10 0x7fcc9c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.559 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.550+0000 7fcc9bfff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc9c067c30 con 0x7fcc9c137440 2020-11-20T12:05:19.559 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.550+0000 7fcc9bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc9c062b50 con 0x7fcc9c0a67f0 2020-11-20T12:05:19.560 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.550+0000 7fcc9bfff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c065c60 con 0x7fcc9c0a9f10 2020-11-20T12:05:19.560 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc9a7fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fcc9c0a67f0 0x7fcc9c136d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50790/0 (socket says 172.21.15.189:50790) 2020-11-20T12:05:19.560 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc9a7fc700 1 -- 172.21.15.189:0/2080115284 learned_addr learned my addr 172.21.15.189:0/2080115284 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:19.560 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc9affd700 1 --2- 172.21.15.189:0/2080115284 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc9c0a9f10 0x7fcc9c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.561 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc99ffb700 1 -- 172.21.15.189:0/2080115284 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3448545258 0 0) 0x7fcc9c062b50 con 0x7fcc9c0a67f0 2020-11-20T12:05:19.561 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc99ffb700 1 -- 172.21.15.189:0/2080115284 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc88003240 con 0x7fcc9c0a67f0 2020-11-20T12:05:19.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc99ffb700 1 -- 172.21.15.189:0/2080115284 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1381227193 0 0) 0x7fcc88003240 con 0x7fcc9c0a67f0 2020-11-20T12:05:19.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc99ffb700 1 -- 172.21.15.189:0/2080115284 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc9c0a9f10 msgr2=0x7fcc9c0a6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:19.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc99ffb700 1 --2- 172.21.15.189:0/2080115284 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc9c0a9f10 0x7fcc9c0a6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc99ffb700 1 -- 172.21.15.189:0/2080115284 >> v1:172.21.15.47:6789/0 conn(0x7fcc9c137440 legacy=0x7fcc9c1413e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:19.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc99ffb700 1 -- 172.21.15.189:0/2080115284 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc9c065f00 con 0x7fcc9c0a67f0 2020-11-20T12:05:19.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc99ffb700 1 -- 172.21.15.189:0/2080115284 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7fcc900014e0 con 0x7fcc9c0a67f0 2020-11-20T12:05:19.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc99ffb700 1 -- 172.21.15.189:0/2080115284 >> v1:172.21.15.189:6789/0 conn(0x7fcc9c0a67f0 legacy=0x7fcc9c136d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:19.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc99ffb700 1 --2- 172.21.15.189:0/2080115284 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc88003730 0x7fcc88003b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc99ffb700 1 --2- 172.21.15.189:0/2080115284 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc9c0a9f10 0x7fcc880044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc99ffb700 1 --2- 172.21.15.189:0/2080115284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc880049d0 0x7fcc88004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc99ffb700 1 -- 172.21.15.189:0/2080115284 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c065c60 con 0x7fcc88003730 2020-11-20T12:05:19.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc99ffb700 1 -- 172.21.15.189:0/2080115284 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc88004140 con 0x7fcc880049d0 2020-11-20T12:05:19.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc99ffb700 1 -- 172.21.15.189:0/2080115284 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc88004300 con 0x7fcc9c0a9f10 2020-11-20T12:05:19.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc9b7fe700 1 --2- 172.21.15.189:0/2080115284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc880049d0 0x7fcc88004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc9affd700 1 --2- 172.21.15.189:0/2080115284 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc88003730 0x7fcc88003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc9a7fc700 1 --2- 172.21.15.189:0/2080115284 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc9c0a9f10 0x7fcc880044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc9b7fe700 1 -- 172.21.15.189:0/2080115284 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc9c0a9f10 msgr2=0x7fcc880044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:19.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc9b7fe700 1 --2- 172.21.15.189:0/2080115284 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc9c0a9f10 0x7fcc880044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc9b7fe700 1 -- 172.21.15.189:0/2080115284 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc88003730 msgr2=0x7fcc88003b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:19.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc9b7fe700 1 --2- 172.21.15.189:0/2080115284 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc88003730 0x7fcc88003b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc9b7fe700 1 -- 172.21.15.189:0/2080115284 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc88003240 con 0x7fcc880049d0 2020-11-20T12:05:19.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc9affd700 1 --2- 172.21.15.189:0/2080115284 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc88003730 0x7fcc88003b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:19.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc9b7fe700 1 --2- 172.21.15.189:0/2080115284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc880049d0 0x7fcc88004e50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fcc94001f60 tx=0x7fcc94007ef0).ready entity=mon.1 client_cookie=3f4df85cb2a2261e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:19.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc99ffb700 1 -- 172.21.15.189:0/2080115284 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc94001600 con 0x7fcc880049d0 2020-11-20T12:05:19.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc99ffb700 1 -- 172.21.15.189:0/2080115284 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc9400cbc0 con 0x7fcc880049d0 2020-11-20T12:05:19.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc99ffb700 1 -- 172.21.15.189:0/2080115284 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc94013410 con 0x7fcc880049d0 2020-11-20T12:05:19.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc9bfff700 1 -- 172.21.15.189:0/2080115284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc880049d0 msgr2=0x7fcc88004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:19.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc9bfff700 1 --2- 172.21.15.189:0/2080115284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc880049d0 0x7fcc88004e50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fcc94001f60 tx=0x7fcc94007ef0).stop 2020-11-20T12:05:19.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.554+0000 7fcc9b7fe700 1 -- 172.21.15.189:0/2080115284 reap_dead start 2020-11-20T12:05:19.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.558+0000 7fcc9bfff700 1 -- 172.21.15.189:0/2080115284 shutdown_connections 2020-11-20T12:05:19.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.558+0000 7fcc9bfff700 1 -- 172.21.15.189:0/2080115284 >> 172.21.15.189:0/2080115284 conn(0x7fcc9c060170 msgr2=0x7fcc9c060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:19.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.558+0000 7fcc9bfff700 1 -- 172.21.15.189:0/2080115284 shutdown_connections 2020-11-20T12:05:19.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.558+0000 7fcc9bfff700 1 -- 172.21.15.189:0/2080115284 wait complete. 2020-11-20T12:05:19.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.558+0000 7fcc9bfff700 1 Processor -- start 2020-11-20T12:05:19.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.558+0000 7fcc9bfff700 1 -- start start 2020-11-20T12:05:19.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.562+0000 7fcc9bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc880049d0 0x7fcc9c127850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.562+0000 7fcc9bfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc9c125ae0 0x7fcc9c09f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.562+0000 7fcc9bfff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc9c122770 0x7fcc9c122bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.562+0000 7fcc9bfff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c126460 con 0x7fcc9c125ae0 2020-11-20T12:05:19.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.562+0000 7fcc9bfff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c14b4e0 con 0x7fcc880049d0 2020-11-20T12:05:19.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.562+0000 7fcc9bfff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c14b780 con 0x7fcc9c122770 2020-11-20T12:05:19.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.562+0000 7fcc9affd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc880049d0 0x7fcc9c127850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.562+0000 7fcc9affd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc880049d0 0x7fcc9c127850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49178/0 (socket says 172.21.15.189:49178) 2020-11-20T12:05:19.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.562+0000 7fcc9affd700 1 -- 172.21.15.189:0/172648656 learned_addr learned my addr 172.21.15.189:0/172648656 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:19.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.562+0000 7fcc9b7fe700 1 --2- 172.21.15.189:0/172648656 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc9c122770 0x7fcc9c122bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.562+0000 7fcc9a7fc700 1 --2- 172.21.15.189:0/172648656 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc9c125ae0 0x7fcc9c09f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.562+0000 7fcc9affd700 1 -- 172.21.15.189:0/172648656 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc9c122770 msgr2=0x7fcc9c122bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:19.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.562+0000 7fcc9affd700 1 --2- 172.21.15.189:0/172648656 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc9c122770 0x7fcc9c122bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.562+0000 7fcc9affd700 1 -- 172.21.15.189:0/172648656 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc9c125ae0 msgr2=0x7fcc9c09f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:19.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.562+0000 7fcc9affd700 1 --2- 172.21.15.189:0/172648656 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc9c125ae0 0x7fcc9c09f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.562+0000 7fcc9affd700 1 -- 172.21.15.189:0/172648656 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc9c14ba20 con 0x7fcc880049d0 2020-11-20T12:05:19.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.562+0000 7fcc9affd700 1 --2- 172.21.15.189:0/172648656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc880049d0 0x7fcc9c127850 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fcc8c010c20 tx=0x7fcc8c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:19.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.562+0000 7fcc82ffd700 1 -- 172.21.15.189:0/172648656 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c00ea30 con 0x7fcc880049d0 2020-11-20T12:05:19.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.562+0000 7fcc82ffd700 1 -- 172.21.15.189:0/172648656 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc8c00eb90 con 0x7fcc880049d0 2020-11-20T12:05:19.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.562+0000 7fcc9bfff700 1 -- 172.21.15.189:0/172648656 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc9c14bec0 con 0x7fcc880049d0 2020-11-20T12:05:19.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.562+0000 7fcc82ffd700 1 -- 172.21.15.189:0/172648656 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c014470 con 0x7fcc880049d0 2020-11-20T12:05:19.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.562+0000 7fcc9bfff700 1 -- 172.21.15.189:0/172648656 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc9c0a0580 con 0x7fcc880049d0 2020-11-20T12:05:19.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.562+0000 7fcc9bfff700 1 -- 172.21.15.189:0/172648656 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc9c14b300 con 0x7fcc880049d0 2020-11-20T12:05:19.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.562+0000 7fcc82ffd700 1 -- 172.21.15.189:0/172648656 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fcc8c024070 con 0x7fcc880049d0 2020-11-20T12:05:19.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.566+0000 7fcc82ffd700 1 --2- 172.21.15.189:0/172648656 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcc780557c0 0x7fcc78057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.566+0000 7fcc82ffd700 1 -- 172.21.15.189:0/172648656 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcc8c06f060 con 0x7fcc880049d0 2020-11-20T12:05:19.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.566+0000 7fcc9a7fc700 1 --2- 172.21.15.189:0/172648656 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcc780557c0 0x7fcc78057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.566+0000 7fcc9a7fc700 1 --2- 172.21.15.189:0/172648656 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcc780557c0 0x7fcc78057c80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fcc9c0a7e60 tx=0x7fcc90009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:19.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.570+0000 7fcc82ffd700 1 -- 172.21.15.189:0/172648656 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcc8c044f80 con 0x7fcc880049d0 2020-11-20T12:05:19.727 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.722+0000 7fcc9bfff700 1 -- 172.21.15.189:0/172648656 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc9c0446d0 con 0x7fcc880049d0 2020-11-20T12:05:19.728 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.722+0000 7fcc82ffd700 1 -- 172.21.15.189:0/172648656 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16826 (secure 0 0 0) 0x7fcc8c018070 con 0x7fcc880049d0 2020-11-20T12:05:19.729 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:05:19.729 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":22,"fsid":"cda18d0e-64b3-41bb-964a-084069c3e648","created":"2020-11-20T12:04:37.538706+0000","modified":"2020-11-20T12:04:55.015818+0000","last_up_change":"2020-11-20T12:04:43.006631+0000","last_in_change":"2020-11-20T12:04:43.006631+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T12:04:41.395395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T12:04:48.550454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T12:04:52.101837+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T12:04:53.018196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c81214e1-4f84-4aaa-8ceb-99c398b2a89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6811","nonce":11760},{"type":"v1","addr":"172.21.15.189:6814","nonce":11760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":11760},{"type":"v1","addr":"172.21.15.189:6818","nonce":11760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":11760},{"type":"v1","addr":"172.21.15.189:6828","nonce":11760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6821","nonce":11760},{"type":"v1","addr":"172.21.15.189:6824","nonce":11760}]},"public_addr":"172.21.15.189:6814/11760","cluster_addr":"172.21.15.189:6818/11760","heartbeat_back_addr":"172.21.15.189:6828/11760","heartbeat_front_addr":"172.21.15.189:6824/11760","state":["exists","up"]},{"osd":1,"uuid":"d2f3ec7d-d403-484f-b5ed-b2e6b079d63d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6805","nonce":11761},{"type":"v1","addr":"172.21.15.189:6807","nonce":11761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":11761},{"type":"v1","addr":"172.21.15.189:6812","nonce":11761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6817","nonce":11761},{"type":"v1","addr":"172.21.15.189:6820","nonce":11761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6813","nonce":11761},{"type":"v1","addr":"172.21.15.189:6815","nonce":11761}]},"public_addr":"172.21.15.189:6807/11761","cluster_addr":"172.21.15.189:6812/11761","heartbeat_back_addr":"172.21.15.189:6820/11761","heartbeat_front_addr":"172.21.15.189:6815/11761","state":["exists","up"]},{"osd":2,"uuid":"e6cf4790-000c-43eb-8844-175f04c60503","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6819","nonce":11751},{"type":"v1","addr":"172.21.15.189:6822","nonce":11751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":11751},{"type":"v1","addr":"172.21.15.189:6825","nonce":11751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":11751},{"type":"v1","addr":"172.21.15.189:6831","nonce":11751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6827","nonce":11751},{"type":"v1","addr":"172.21.15.189:6829","nonce":11751}]},"public_addr":"172.21.15.189:6822/11751","cluster_addr":"172.21.15.189:6825/11751","heartbeat_back_addr":"172.21.15.189:6831/11751","heartbeat_front_addr":"172.21.15.189:6829/11751","state":["exists","up"]},{"osd":3,"uuid":"ba49dfac-f2b9-476e-853d-42f05e21a77a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":11762},{"type":"v1","addr":"172.21.15.189:6801","nonce":11762}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":11762},{"type":"v1","addr":"172.21.15.189:6803","nonce":11762}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":11762},{"type":"v1","addr":"172.21.15.189:6809","nonce":11762}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":11762},{"type":"v1","addr":"172.21.15.189:6806","nonce":11762}]},"public_addr":"172.21.15.189:6801/11762","cluster_addr":"172.21.15.189:6803/11762","heartbeat_back_addr":"172.21.15.189:6809/11762","heartbeat_front_addr":"172.21.15.189:6806/11762","state":["exists","up"]},{"osd":4,"uuid":"3ca7b438-38b2-4689-8b18-fb5a8ddf6aee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6824","nonce":12114},{"type":"v1","addr":"172.21.15.47:6825","nonce":12114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6826","nonce":12114},{"type":"v1","addr":"172.21.15.47:6827","nonce":12114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6830","nonce":12114},{"type":"v1","addr":"172.21.15.47:6831","nonce":12114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6828","nonce":12114},{"type":"v1","addr":"172.21.15.47:6829","nonce":12114}]},"public_addr":"172.21.15.47:6825/12114","cluster_addr":"172.21.15.47:6827/12114","heartbeat_back_addr":"172.21.15.47:6831/12114","heartbeat_front_addr":"172.21.15.47:6829/12114","state":["exists","up"]},{"osd":5,"uuid":"6a45fca7-83d3-44fd-a1d4-b01e4201a101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6800","nonce":12111},{"type":"v1","addr":"172.21.15.47:6801","nonce":12111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6802","nonce":12111},{"type":"v1","addr":"172.21.15.47:6803","nonce":12111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6806","nonce":12111},{"type":"v1","addr":"172.21.15.47:6807","nonce":12111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6804","nonce":12111},{"type":"v1","addr":"172.21.15.47:6805","nonce":12111}]},"public_addr":"172.21.15.47:6801/12111","cluster_addr":"172.21.15.47:6803/12111","heartbeat_back_addr":"172.21.15.47:6807/12111","heartbeat_front_addr":"172.21.15.47:6805/12111","state":["exists","up"]},{"osd":6,"uuid":"c4b1b9ed-61f8-4677-9755-dff59a30e447","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6816","nonce":12113},{"type":"v1","addr":"172.21.15.47:6817","nonce":12113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6818","nonce":12113},{"type":"v1","addr":"172.21.15.47:6819","nonce":12113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6822","nonce":12113},{"type":"v1","addr":"172.21.15.47:6823","nonce":12113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6820","nonce":12113},{"type":"v1","addr":"172.21.15.47:6821","nonce":12113}]},"public_addr":"172.21.15.47:6817/12113","cluster_addr":"172.21.15.47:6819/12113","heartbeat_back_addr":"172.21.15.47:6823/12113","heartbeat_front_addr":"172.21.15.47:6821/12113","state":["exists","up"]},{"osd":7,"uuid":"4608e4ed-cc40-4c5c-ba8b-3779bbc62f65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6808","nonce":12112},{"type":"v1","addr":"172.21.15.47:6809","nonce":12112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6810","nonce":12112},{"type":"v1","addr":"172.21.15.47:6811","nonce":12112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6814","nonce":12112},{"type":"v1","addr":"172.21.15.47:6815","nonce":12112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6812","nonce":12112},{"type":"v1","addr":"172.21.15.47:6813","nonce":12112}]},"public_addr":"172.21.15.47:6809/12112","cluster_addr":"172.21.15.47:6811/12112","heartbeat_back_addr":"172.21.15.47:6815/12112","heartbeat_front_addr":"172.21.15.47:6813/12112","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.435265+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.472776+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.388370+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.417366+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.102392+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.116529+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.167430+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T12:04:41.148139+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T12:05:19.730 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.722+0000 7fcc9bfff700 1 -- 172.21.15.189:0/172648656 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcc780557c0 msgr2=0x7fcc78057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:19.731 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.722+0000 7fcc9bfff700 1 --2- 172.21.15.189:0/172648656 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcc780557c0 0x7fcc78057c80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fcc9c0a7e60 tx=0x7fcc90009040).stop 2020-11-20T12:05:19.731 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.722+0000 7fcc9bfff700 1 -- 172.21.15.189:0/172648656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc880049d0 msgr2=0x7fcc9c127850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:19.731 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.722+0000 7fcc9bfff700 1 --2- 172.21.15.189:0/172648656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc880049d0 0x7fcc9c127850 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fcc8c010c20 tx=0x7fcc8c008040).stop 2020-11-20T12:05:19.731 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.722+0000 7fcc9bfff700 1 -- 172.21.15.189:0/172648656 shutdown_connections 2020-11-20T12:05:19.732 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.722+0000 7fcc9bfff700 1 --2- 172.21.15.189:0/172648656 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc9c122770 0x7fcc9c122bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.732 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.722+0000 7fcc9bfff700 1 --2- 172.21.15.189:0/172648656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc880049d0 0x7fcc9c127850 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.732 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.722+0000 7fcc9bfff700 1 --2- 172.21.15.189:0/172648656 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcc780557c0 0x7fcc78057c80 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.732 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.722+0000 7fcc9bfff700 1 --2- 172.21.15.189:0/172648656 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc9c125ae0 0x7fcc9c09f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.732 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.722+0000 7fcc9bfff700 1 -- 172.21.15.189:0/172648656 >> 172.21.15.189:0/172648656 conn(0x7fcc9c060170 msgr2=0x7fcc9c060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:19.733 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.722+0000 7fcc9bfff700 1 -- 172.21.15.189:0/172648656 shutdown_connections 2020-11-20T12:05:19.733 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.722+0000 7fcc9bfff700 1 -- 172.21.15.189:0/172648656 wait complete. 2020-11-20T12:05:19.742 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T12:04:41.395395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T12:04:48.550454+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T12:04:52.101837+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T12:04:53.018196+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T12:05:19.743 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T12:05:19.869 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.862+0000 7f0496472700 1 Processor -- start 2020-11-20T12:05:19.870 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.862+0000 7f0496472700 1 -- start start 2020-11-20T12:05:19.870 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.862+0000 7f0496472700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0490060ba0 0x7f04900a8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.871 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.862+0000 7f0496472700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0490067d90 0x7f0490066a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.871 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.862+0000 7f0496472700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04900a9410 con 0x7f04900a97d0 2020-11-20T12:05:19.871 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.866+0000 7f0496472700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04900684c0 con 0x7f0490060ba0 2020-11-20T12:05:19.871 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.866+0000 7f0496472700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04900678f0 con 0x7f0490067d90 2020-11-20T12:05:19.873 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.866+0000 7f0494c6f700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0490067d90 0x7f0490066a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.873 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.866+0000 7f0495470700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0490060ba0 0x7f04900a8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.873 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.866+0000 7f0495c71700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f04900a97d0 0x7f04901373f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.189:45476/0 (socket says 172.21.15.189:45476) 2020-11-20T12:05:19.873 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.866+0000 7f0494c6f700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0490067d90 0x7f0490066a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.189:42222/0 (socket says 172.21.15.189:42222) 2020-11-20T12:05:19.874 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.866+0000 7f0495470700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0490060ba0 0x7f04900a8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.189:55410/0 (socket says 172.21.15.189:55410) 2020-11-20T12:05:19.874 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.866+0000 7f0495c71700 1 -- 172.21.15.189:0/136825847 learned_addr learned my addr 172.21.15.189:0/136825847 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:19.874 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.866+0000 7f0494c6f700 1 -- 172.21.15.189:0/136825847 >> v2:172.21.15.47:3300/0 conn(0x7f0490060ba0 msgr2=0x7f04900a8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:19.874 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.866+0000 7f0494c6f700 1 --2- 172.21.15.189:0/136825847 >> v2:172.21.15.47:3300/0 conn(0x7f0490060ba0 0x7f04900a8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.874 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.866+0000 7f0494c6f700 1 -- 172.21.15.189:0/136825847 >> v1:172.21.15.47:6789/0 conn(0x7f04900a97d0 legacy=0x7f04901373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:19.875 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.866+0000 7f0494c6f700 1 -- 172.21.15.189:0/136825847 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0490060f80 con 0x7f0490067d90 2020-11-20T12:05:19.875 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.866+0000 7f0495470700 1 --2- 172.21.15.189:0/136825847 >> v2:172.21.15.47:3300/0 conn(0x7f0490060ba0 0x7f04900a8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:05:19.875 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.866+0000 7f0494c6f700 1 --2- 172.21.15.189:0/136825847 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0490067d90 0x7f0490066a40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f048000b9a0 tx=0x7f0480006df0).ready entity=mon.2 client_cookie=4d68698fce3489a0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:19.875 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.866+0000 7f0487fff700 1 -- 172.21.15.189:0/136825847 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04800097b0 con 0x7f0490067d90 2020-11-20T12:05:19.875 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.866+0000 7f0487fff700 1 -- 172.21.15.189:0/136825847 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0480009910 con 0x7f0490067d90 2020-11-20T12:05:19.876 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.866+0000 7f0487fff700 1 -- 172.21.15.189:0/136825847 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0480009c90 con 0x7f0490067d90 2020-11-20T12:05:19.877 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.866+0000 7f0496472700 1 -- 172.21.15.189:0/136825847 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0490067d90 msgr2=0x7f0490066a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:19.878 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.866+0000 7f0496472700 1 --2- 172.21.15.189:0/136825847 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0490067d90 0x7f0490066a40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f048000b9a0 tx=0x7f0480006df0).stop 2020-11-20T12:05:19.878 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.866+0000 7f0496472700 1 -- 172.21.15.189:0/136825847 shutdown_connections 2020-11-20T12:05:19.879 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.866+0000 7f0496472700 1 --2- 172.21.15.189:0/136825847 >> v2:172.21.15.47:3300/0 conn(0x7f0490060ba0 0x7f04900a8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.879 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.866+0000 7f0496472700 1 --2- 172.21.15.189:0/136825847 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0490067d90 0x7f0490066a40 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.879 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.866+0000 7f0496472700 1 -- 172.21.15.189:0/136825847 >> 172.21.15.189:0/136825847 conn(0x7f04900ad8e0 msgr2=0x7f04900adcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:19.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.866+0000 7f0496472700 1 -- 172.21.15.189:0/136825847 shutdown_connections 2020-11-20T12:05:19.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.866+0000 7f0496472700 1 -- 172.21.15.189:0/136825847 wait complete. 2020-11-20T12:05:19.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.870+0000 7f0496472700 1 Processor -- start 2020-11-20T12:05:19.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.870+0000 7f0496472700 1 -- start start 2020-11-20T12:05:19.881 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.870+0000 7f0496472700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0490060ba0 0x7f049014cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.881 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.870+0000 7f0496472700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0490067d90 0x7f049014a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.881 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.870+0000 7f0496472700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04900a97d0 0x7f04900ad0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.881 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.870+0000 7f0496472700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049013bce0 con 0x7f0490067d90 2020-11-20T12:05:19.881 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.870+0000 7f0496472700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0490149b30 con 0x7f04900a97d0 2020-11-20T12:05:19.882 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.870+0000 7f0496472700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0490149d90 con 0x7f0490060ba0 2020-11-20T12:05:19.882 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.870+0000 7f0495c71700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04900a97d0 0x7f04900ad0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.882 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.870+0000 7f0495470700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0490060ba0 0x7f049014cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.882 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.870+0000 7f0495c71700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04900a97d0 0x7f04900ad0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49196/0 (socket says 172.21.15.189:49196) 2020-11-20T12:05:19.882 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.870+0000 7f0495c71700 1 -- 172.21.15.189:0/3593623605 learned_addr learned my addr 172.21.15.189:0/3593623605 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:19.883 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.870+0000 7f0494c6f700 1 --2- 172.21.15.189:0/3593623605 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0490067d90 0x7f049014a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.884 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.870+0000 7f0495c71700 1 -- 172.21.15.189:0/3593623605 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0490060ba0 msgr2=0x7f049014cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:19.884 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.870+0000 7f0495c71700 1 --2- 172.21.15.189:0/3593623605 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0490060ba0 0x7f049014cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.884 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.870+0000 7f0495c71700 1 -- 172.21.15.189:0/3593623605 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0490067d90 msgr2=0x7f049014a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:19.885 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.870+0000 7f0495c71700 1 --2- 172.21.15.189:0/3593623605 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0490067d90 0x7f049014a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:19.885 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.870+0000 7f0495c71700 1 -- 172.21.15.189:0/3593623605 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049014a030 con 0x7f04900a97d0 2020-11-20T12:05:19.885 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.870+0000 7f0495c71700 1 --2- 172.21.15.189:0/3593623605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04900a97d0 0x7f04900ad0a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f048800ca90 tx=0x7f0488001360).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:19.885 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.874+0000 7f04857fa700 1 -- 172.21.15.189:0/3593623605 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048800ce60 con 0x7f04900a97d0 2020-11-20T12:05:19.886 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.874+0000 7f04857fa700 1 -- 172.21.15.189:0/3593623605 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0488005b80 con 0x7f04900a97d0 2020-11-20T12:05:19.886 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.874+0000 7f0496472700 1 -- 172.21.15.189:0/3593623605 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0490139160 con 0x7f04900a97d0 2020-11-20T12:05:19.886 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.874+0000 7f04857fa700 1 -- 172.21.15.189:0/3593623605 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0488007110 con 0x7f04900a97d0 2020-11-20T12:05:19.886 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.874+0000 7f0496472700 1 -- 172.21.15.189:0/3593623605 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0490139680 con 0x7f04900a97d0 2020-11-20T12:05:19.886 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.874+0000 7f0496472700 1 -- 172.21.15.189:0/3593623605 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f049014a1c0 con 0x7f04900a97d0 2020-11-20T12:05:19.887 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.874+0000 7f04857fa700 1 -- 172.21.15.189:0/3593623605 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f0488010030 con 0x7f04900a97d0 2020-11-20T12:05:19.887 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.874+0000 7f04857fa700 1 --2- 172.21.15.189:0/3593623605 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f04680557c0 0x7f0468057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:19.887 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.874+0000 7f04857fa700 1 -- 172.21.15.189:0/3593623605 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f048806e530 con 0x7f04900a97d0 2020-11-20T12:05:19.887 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.874+0000 7f0495470700 1 --2- 172.21.15.189:0/3593623605 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f04680557c0 0x7f0468057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:19.887 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.878+0000 7f0495470700 1 --2- 172.21.15.189:0/3593623605 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f04680557c0 0x7f0468057c80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f048c006fd0 tx=0x7f048c006df0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:19.889 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:19.882+0000 7f04857fa700 1 -- 172.21.15.189:0/3593623605 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f04880443d0 con 0x7f04900a97d0 2020-11-20T12:05:20.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.038+0000 7f0496472700 1 -- 172.21.15.189:0/3593623605 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f04900446d0 con 0x7f04900a97d0 2020-11-20T12:05:20.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.042+0000 7f04857fa700 1 -- 172.21.15.189:0/3593623605 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v22) v1 ==== 110+0+10 (secure 0 0 0) 0x7f04880479f0 con 0x7f04900a97d0 2020-11-20T12:05:20.047 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 1 2020-11-20T12:05:20.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.042+0000 7f0496472700 1 -- 172.21.15.189:0/3593623605 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f04680557c0 msgr2=0x7f0468057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.042+0000 7f0496472700 1 --2- 172.21.15.189:0/3593623605 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f04680557c0 0x7f0468057c80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f048c006fd0 tx=0x7f048c006df0).stop 2020-11-20T12:05:20.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.042+0000 7f0496472700 1 -- 172.21.15.189:0/3593623605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04900a97d0 msgr2=0x7f04900ad0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.042+0000 7f0496472700 1 --2- 172.21.15.189:0/3593623605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04900a97d0 0x7f04900ad0a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f048800ca90 tx=0x7f0488001360).stop 2020-11-20T12:05:20.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.042+0000 7f0496472700 1 -- 172.21.15.189:0/3593623605 shutdown_connections 2020-11-20T12:05:20.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.042+0000 7f0496472700 1 --2- 172.21.15.189:0/3593623605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04900a97d0 0x7f04900ad0a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.042+0000 7f0496472700 1 --2- 172.21.15.189:0/3593623605 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0490060ba0 0x7f049014cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.042+0000 7f0496472700 1 --2- 172.21.15.189:0/3593623605 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f04680557c0 0x7f0468057c80 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.042+0000 7f0496472700 1 --2- 172.21.15.189:0/3593623605 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0490067d90 0x7f049014a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.042+0000 7f0496472700 1 -- 172.21.15.189:0/3593623605 >> 172.21.15.189:0/3593623605 conn(0x7f04900ad8e0 msgr2=0x7f04900adcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:20.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.042+0000 7f0496472700 1 -- 172.21.15.189:0/3593623605 shutdown_connections 2020-11-20T12:05:20.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.042+0000 7f0496472700 1 -- 172.21.15.189:0/3593623605 wait complete. 2020-11-20T12:05:20.061 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T12:05:20.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.182+0000 7fd25ee40700 1 Processor -- start 2020-11-20T12:05:20.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.182+0000 7fd25ee40700 1 -- start start 2020-11-20T12:05:20.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.182+0000 7fd25ee40700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd2580b19e0 0x7fd25813a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.182+0000 7fd25ee40700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd258066a30 0x7fd25813ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.182+0000 7fd25ee40700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd258066fb0 con 0x7fd258062930 2020-11-20T12:05:20.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.182+0000 7fd25ee40700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2580666b0 con 0x7fd2580b19e0 2020-11-20T12:05:20.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.182+0000 7fd25ee40700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd258067250 con 0x7fd258066a30 2020-11-20T12:05:20.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.182+0000 7fd25e63f700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd258066a30 0x7fd25813ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.182+0000 7fd25d63d700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd2580b19e0 0x7fd25813a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.182+0000 7fd25e63f700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd258066a30 0x7fd25813ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.189:42236/0 (socket says 172.21.15.189:42236) 2020-11-20T12:05:20.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.182+0000 7fd25d63d700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd2580b19e0 0x7fd25813a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.189:55426/0 (socket says 172.21.15.189:55426) 2020-11-20T12:05:20.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.182+0000 7fd25de3e700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fd258062930 0x7fd2580b12d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.189:45486/0 (socket says 172.21.15.189:45486) 2020-11-20T12:05:20.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.182+0000 7fd25e63f700 1 -- 172.21.15.189:0/886629836 learned_addr learned my addr 172.21.15.189:0/886629836 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:20.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.182+0000 7fd25d63d700 1 -- 172.21.15.189:0/886629836 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd258066a30 msgr2=0x7fd25813ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:20.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.182+0000 7fd25d63d700 1 --2- 172.21.15.189:0/886629836 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd258066a30 0x7fd25813ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.182+0000 7fd25d63d700 1 -- 172.21.15.189:0/886629836 >> v1:172.21.15.47:6789/0 conn(0x7fd258062930 legacy=0x7fd2580b12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.182+0000 7fd25d63d700 1 -- 172.21.15.189:0/886629836 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2580b0e30 con 0x7fd2580b19e0 2020-11-20T12:05:20.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.182+0000 7fd25e63f700 1 --2- 172.21.15.189:0/886629836 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd258066a30 0x7fd25813ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:05:20.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.182+0000 7fd25d63d700 1 --2- 172.21.15.189:0/886629836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2580b19e0 0x7fd25813a7f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fd25400b9a0 tx=0x7fd254006df0).ready entity=mon.0 client_cookie=c7f0ea3b138343a2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:20.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.182+0000 7fd25ce3c700 1 -- 172.21.15.189:0/886629836 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2540097b0 con 0x7fd2580b19e0 2020-11-20T12:05:20.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.182+0000 7fd25ce3c700 1 -- 172.21.15.189:0/886629836 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd254009910 con 0x7fd2580b19e0 2020-11-20T12:05:20.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.182+0000 7fd25ce3c700 1 -- 172.21.15.189:0/886629836 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd254009c90 con 0x7fd2580b19e0 2020-11-20T12:05:20.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.186+0000 7fd25ee40700 1 -- 172.21.15.189:0/886629836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2580b19e0 msgr2=0x7fd25813a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.186+0000 7fd25ee40700 1 --2- 172.21.15.189:0/886629836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2580b19e0 0x7fd25813a7f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fd25400b9a0 tx=0x7fd254006df0).stop 2020-11-20T12:05:20.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.186+0000 7fd25ee40700 1 -- 172.21.15.189:0/886629836 shutdown_connections 2020-11-20T12:05:20.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.186+0000 7fd25ee40700 1 --2- 172.21.15.189:0/886629836 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd258066a30 0x7fd25813ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.186+0000 7fd25ee40700 1 --2- 172.21.15.189:0/886629836 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2580b19e0 0x7fd25813a7f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.186+0000 7fd25ee40700 1 -- 172.21.15.189:0/886629836 >> 172.21.15.189:0/886629836 conn(0x7fd258065080 msgr2=0x7fd258062080 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:20.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.186+0000 7fd25ee40700 1 -- 172.21.15.189:0/886629836 shutdown_connections 2020-11-20T12:05:20.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.186+0000 7fd25ee40700 1 -- 172.21.15.189:0/886629836 wait complete. 2020-11-20T12:05:20.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.186+0000 7fd25ee40700 1 Processor -- start 2020-11-20T12:05:20.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.186+0000 7fd25ee40700 1 -- start start 2020-11-20T12:05:20.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.186+0000 7fd25ee40700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd258062930 0x7fd25813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.186+0000 7fd25ee40700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd258066a30 0x7fd25814a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.186+0000 7fd25ee40700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd2580b19e0 0x7fd2580a96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.186+0000 7fd25ee40700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25814a6b0 con 0x7fd258062930 2020-11-20T12:05:20.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.186+0000 7fd25ee40700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd258138560 con 0x7fd258066a30 2020-11-20T12:05:20.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.186+0000 7fd25ee40700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd25814a8d0 con 0x7fd2580b19e0 2020-11-20T12:05:20.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.190+0000 7fd25d63d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd258066a30 0x7fd25814a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.190+0000 7fd25d63d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd258066a30 0x7fd25814a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49208/0 (socket says 172.21.15.189:49208) 2020-11-20T12:05:20.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.190+0000 7fd25d63d700 1 -- 172.21.15.189:0/735419202 learned_addr learned my addr 172.21.15.189:0/735419202 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:20.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.190+0000 7fd25de3e700 1 --2- 172.21.15.189:0/735419202 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd258062930 0x7fd25813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.190+0000 7fd25e63f700 1 --2- 172.21.15.189:0/735419202 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd2580b19e0 0x7fd2580a96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.190+0000 7fd25d63d700 1 -- 172.21.15.189:0/735419202 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd2580b19e0 msgr2=0x7fd2580a96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.190+0000 7fd25d63d700 1 --2- 172.21.15.189:0/735419202 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd2580b19e0 0x7fd2580a96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.190+0000 7fd25d63d700 1 -- 172.21.15.189:0/735419202 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd258062930 msgr2=0x7fd25813f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.190+0000 7fd25d63d700 1 --2- 172.21.15.189:0/735419202 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd258062930 0x7fd25813f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.190+0000 7fd25d63d700 1 -- 172.21.15.189:0/735419202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd25814ab70 con 0x7fd258066a30 2020-11-20T12:05:20.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.190+0000 7fd25e63f700 1 --2- 172.21.15.189:0/735419202 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd2580b19e0 0x7fd2580a96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:20.202 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.190+0000 7fd25d63d700 1 --2- 172.21.15.189:0/735419202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd258066a30 0x7fd25814a1c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fd254010000 tx=0x7fd254015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:20.202 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.190+0000 7fd245ffb700 1 -- 172.21.15.189:0/735419202 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25401e460 con 0x7fd258066a30 2020-11-20T12:05:20.202 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.190+0000 7fd245ffb700 1 -- 172.21.15.189:0/735419202 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd25401e5c0 con 0x7fd258066a30 2020-11-20T12:05:20.202 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.190+0000 7fd25ee40700 1 -- 172.21.15.189:0/735419202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2580a9200 con 0x7fd258066a30 2020-11-20T12:05:20.202 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.190+0000 7fd245ffb700 1 -- 172.21.15.189:0/735419202 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25401e960 con 0x7fd258066a30 2020-11-20T12:05:20.203 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.190+0000 7fd25ee40700 1 -- 172.21.15.189:0/735419202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2580a9b90 con 0x7fd258066a30 2020-11-20T12:05:20.203 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.190+0000 7fd25ee40700 1 -- 172.21.15.189:0/735419202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2580a9390 con 0x7fd258066a30 2020-11-20T12:05:20.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.190+0000 7fd245ffb700 1 -- 172.21.15.189:0/735419202 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fd254025030 con 0x7fd258066a30 2020-11-20T12:05:20.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.190+0000 7fd245ffb700 1 --2- 172.21.15.189:0/735419202 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd2340557c0 0x7fd234057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.190+0000 7fd245ffb700 1 -- 172.21.15.189:0/735419202 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd25406fcb0 con 0x7fd258066a30 2020-11-20T12:05:20.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.190+0000 7fd25de3e700 1 --2- 172.21.15.189:0/735419202 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd2340557c0 0x7fd234057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.205 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.194+0000 7fd25de3e700 1 --2- 172.21.15.189:0/735419202 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd2340557c0 0x7fd234057c80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fd2480045d0 tx=0x7fd248007b80).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:20.207 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.202+0000 7fd245ffb700 1 -- 172.21.15.189:0/735419202 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd254045c00 con 0x7fd258066a30 2020-11-20T12:05:20.358 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.350+0000 7fd25ee40700 1 -- 172.21.15.189:0/735419202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd258141f00 con 0x7fd258066a30 2020-11-20T12:05:20.359 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.354+0000 7fd245ffb700 1 -- 172.21.15.189:0/735419202 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v22) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd254023030 con 0x7fd258066a30 2020-11-20T12:05:20.359 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 8 2020-11-20T12:05:20.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.354+0000 7fd25ee40700 1 -- 172.21.15.189:0/735419202 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd2340557c0 msgr2=0x7fd234057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.354+0000 7fd25ee40700 1 --2- 172.21.15.189:0/735419202 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd2340557c0 0x7fd234057c80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fd2480045d0 tx=0x7fd248007b80).stop 2020-11-20T12:05:20.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.354+0000 7fd25ee40700 1 -- 172.21.15.189:0/735419202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd258066a30 msgr2=0x7fd25814a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.354+0000 7fd25ee40700 1 --2- 172.21.15.189:0/735419202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd258066a30 0x7fd25814a1c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fd254010000 tx=0x7fd254015000).stop 2020-11-20T12:05:20.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.354+0000 7fd25ee40700 1 -- 172.21.15.189:0/735419202 shutdown_connections 2020-11-20T12:05:20.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.354+0000 7fd25ee40700 1 --2- 172.21.15.189:0/735419202 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd2340557c0 0x7fd234057c80 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.354+0000 7fd25ee40700 1 --2- 172.21.15.189:0/735419202 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd258062930 0x7fd25813f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.354+0000 7fd25ee40700 1 --2- 172.21.15.189:0/735419202 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd2580b19e0 0x7fd2580a96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.354+0000 7fd25ee40700 1 --2- 172.21.15.189:0/735419202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd258066a30 0x7fd25814a1c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.354+0000 7fd25ee40700 1 -- 172.21.15.189:0/735419202 >> 172.21.15.189:0/735419202 conn(0x7fd258065080 msgr2=0x7fd258137350 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:20.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.354+0000 7fd25ee40700 1 -- 172.21.15.189:0/735419202 shutdown_connections 2020-11-20T12:05:20.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.354+0000 7fd25ee40700 1 -- 172.21.15.189:0/735419202 wait complete. 2020-11-20T12:05:20.380 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T12:05:20.506 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.498+0000 7f7a3270c700 1 Processor -- start 2020-11-20T12:05:20.506 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.498+0000 7f7a3270c700 1 -- start start 2020-11-20T12:05:20.508 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a3270c700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7a2c0a8660 0x7f7a2c0a9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.508 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a3270c700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7a2c0661e0 0x7f7a2c0a9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.508 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a3270c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7a2c0ac9a0 0x7f7a2c0ace50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.508 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a3270c700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a2c0aa100 con 0x7f7a2c0661e0 2020-11-20T12:05:20.509 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a3270c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a2c0a9360 con 0x7f7a2c0ac9a0 2020-11-20T12:05:20.509 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a3270c700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c066760 con 0x7f7a2c0a8660 2020-11-20T12:05:20.509 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a31f0b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7a2c0ac9a0 0x7f7a2c0ace50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.509 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a31f0b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7a2c0ac9a0 0x7f7a2c0ace50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49216/0 (socket says 172.21.15.189:49216) 2020-11-20T12:05:20.510 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a31f0b700 1 -- 172.21.15.189:0/469649935 learned_addr learned my addr 172.21.15.189:0/469649935 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:20.510 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a30f09700 1 --2- 172.21.15.189:0/469649935 >> v2:172.21.15.47:3300/0 conn(0x7f7a2c0661e0 0x7f7a2c0a9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.510 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a3170a700 1 --2- 172.21.15.189:0/469649935 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7a2c0a8660 0x7f7a2c0a9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.512 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a31f0b700 1 -- 172.21.15.189:0/469649935 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7a2c0a8660 msgr2=0x7f7a2c0a9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:20.512 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a31f0b700 1 --2- 172.21.15.189:0/469649935 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7a2c0a8660 0x7f7a2c0a9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.512 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a31f0b700 1 -- 172.21.15.189:0/469649935 >> v2:172.21.15.47:3300/0 conn(0x7f7a2c0661e0 msgr2=0x7f7a2c0a9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:20.512 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a31f0b700 1 --2- 172.21.15.189:0/469649935 >> v2:172.21.15.47:3300/0 conn(0x7f7a2c0661e0 0x7f7a2c0a9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.513 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a31f0b700 1 -- 172.21.15.189:0/469649935 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2c0a6a70 con 0x7f7a2c0ac9a0 2020-11-20T12:05:20.513 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a31f0b700 1 --2- 172.21.15.189:0/469649935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a2c0ac9a0 0x7f7a2c0ace50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f7a2c0aa520 tx=0x7f7a24009290).ready entity=mon.1 client_cookie=caffd46e97ab5428 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:20.513 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a17fff700 1 -- 172.21.15.189:0/469649935 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2400bb10 con 0x7f7a2c0ac9a0 2020-11-20T12:05:20.513 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a17fff700 1 -- 172.21.15.189:0/469649935 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a2400bc70 con 0x7f7a2c0ac9a0 2020-11-20T12:05:20.513 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a17fff700 1 -- 172.21.15.189:0/469649935 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a24011400 con 0x7f7a2c0ac9a0 2020-11-20T12:05:20.514 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a3270c700 1 -- 172.21.15.189:0/469649935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a2c0ac9a0 msgr2=0x7f7a2c0ace50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.514 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a3270c700 1 --2- 172.21.15.189:0/469649935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a2c0ac9a0 0x7f7a2c0ace50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f7a2c0aa520 tx=0x7f7a24009290).stop 2020-11-20T12:05:20.514 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a3270c700 1 -- 172.21.15.189:0/469649935 shutdown_connections 2020-11-20T12:05:20.514 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a3270c700 1 --2- 172.21.15.189:0/469649935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a2c0ac9a0 0x7f7a2c0ace50 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.514 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a3270c700 1 --2- 172.21.15.189:0/469649935 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7a2c0a8660 0x7f7a2c0a9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.515 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a3270c700 1 --2- 172.21.15.189:0/469649935 >> v2:172.21.15.47:3300/0 conn(0x7f7a2c0661e0 0x7f7a2c0a9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.515 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a3270c700 1 -- 172.21.15.189:0/469649935 >> 172.21.15.189:0/469649935 conn(0x7f7a2c065420 msgr2=0x7f7a2c062080 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:20.515 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a3270c700 1 -- 172.21.15.189:0/469649935 shutdown_connections 2020-11-20T12:05:20.515 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.502+0000 7f7a3270c700 1 -- 172.21.15.189:0/469649935 wait complete. 2020-11-20T12:05:20.515 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.506+0000 7f7a3270c700 1 Processor -- start 2020-11-20T12:05:20.516 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.506+0000 7f7a3270c700 1 -- start start 2020-11-20T12:05:20.518 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.506+0000 7f7a3270c700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7a2c0661e0 0x7f7a2c13b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.518 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.506+0000 7f7a3270c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a2c0a8660 0x7f7a2c1378f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.518 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.506+0000 7f7a3270c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a2c0ac9a0 0x7f7a2c138cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.519 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.506+0000 7f7a3270c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c13b680 con 0x7f7a2c0a8660 2020-11-20T12:05:20.519 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.506+0000 7f7a3270c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c138580 con 0x7f7a2c0ac9a0 2020-11-20T12:05:20.519 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.506+0000 7f7a3270c700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c138820 con 0x7f7a2c0661e0 2020-11-20T12:05:20.519 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.506+0000 7f7a31f0b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a2c0ac9a0 0x7f7a2c138cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.519 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.506+0000 7f7a3170a700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7a2c0661e0 0x7f7a2c13b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.520 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.506+0000 7f7a31f0b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a2c0ac9a0 0x7f7a2c138cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49222/0 (socket says 172.21.15.189:49222) 2020-11-20T12:05:20.520 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.506+0000 7f7a31f0b700 1 -- 172.21.15.189:0/762829937 learned_addr learned my addr 172.21.15.189:0/762829937 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:20.520 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.506+0000 7f7a30f09700 1 --2- 172.21.15.189:0/762829937 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a2c0a8660 0x7f7a2c1378f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.520 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.506+0000 7f7a31f0b700 1 -- 172.21.15.189:0/762829937 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7a2c0661e0 msgr2=0x7f7a2c13b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.520 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.506+0000 7f7a31f0b700 1 --2- 172.21.15.189:0/762829937 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7a2c0661e0 0x7f7a2c13b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.521 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.506+0000 7f7a31f0b700 1 -- 172.21.15.189:0/762829937 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a2c0a8660 msgr2=0x7f7a2c1378f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.521 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.506+0000 7f7a31f0b700 1 --2- 172.21.15.189:0/762829937 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a2c0a8660 0x7f7a2c1378f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.521 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.506+0000 7f7a31f0b700 1 -- 172.21.15.189:0/762829937 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2c13a0a0 con 0x7f7a2c0ac9a0 2020-11-20T12:05:20.521 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.506+0000 7f7a31f0b700 1 --2- 172.21.15.189:0/762829937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a2c0ac9a0 0x7f7a2c138cc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f7a2400c010 tx=0x7f7a24016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:20.522 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.506+0000 7f7a1dffb700 1 -- 172.21.15.189:0/762829937 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2401b6b0 con 0x7f7a2c0ac9a0 2020-11-20T12:05:20.522 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.510+0000 7f7a1dffb700 1 -- 172.21.15.189:0/762829937 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a2401b810 con 0x7f7a2c0ac9a0 2020-11-20T12:05:20.522 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.510+0000 7f7a3270c700 1 -- 172.21.15.189:0/762829937 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a2c137de0 con 0x7f7a2c0ac9a0 2020-11-20T12:05:20.522 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.510+0000 7f7a1dffb700 1 -- 172.21.15.189:0/762829937 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2401bbb0 con 0x7f7a2c0ac9a0 2020-11-20T12:05:20.522 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.510+0000 7f7a3270c700 1 -- 172.21.15.189:0/762829937 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a2c1393d0 con 0x7f7a2c0ac9a0 2020-11-20T12:05:20.523 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.510+0000 7f7a1dffb700 1 -- 172.21.15.189:0/762829937 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f7a24017070 con 0x7f7a2c0ac9a0 2020-11-20T12:05:20.523 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.510+0000 7f7a1dffb700 1 --2- 172.21.15.189:0/762829937 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f7a000557f0 0x7f7a00057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.523 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.510+0000 7f7a3170a700 1 --2- 172.21.15.189:0/762829937 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f7a000557f0 0x7f7a00057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.524 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.510+0000 7f7a1dffb700 1 -- 172.21.15.189:0/762829937 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7a240707b0 con 0x7f7a2c0ac9a0 2020-11-20T12:05:20.524 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.510+0000 7f7a3270c700 1 -- 172.21.15.189:0/762829937 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a2c1389e0 con 0x7f7a2c0ac9a0 2020-11-20T12:05:20.524 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.510+0000 7f7a3170a700 1 --2- 172.21.15.189:0/762829937 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f7a000557f0 0x7f7a00057cb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f7a20006fd0 tx=0x7f7a20006df0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:20.524 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.518+0000 7f7a1dffb700 1 -- 172.21.15.189:0/762829937 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7a24015080 con 0x7f7a2c0ac9a0 2020-11-20T12:05:20.675 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.670+0000 7f7a3270c700 1 -- 172.21.15.189:0/762829937 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7a2c044f60 con 0x7f7a2c0ac9a0 2020-11-20T12:05:20.676 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.670+0000 7f7a1dffb700 1 -- 172.21.15.189:0/762829937 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+10 (secure 0 0 0) 0x7f7a24046650 con 0x7f7a2c0ac9a0 2020-11-20T12:05:20.676 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 8 2020-11-20T12:05:20.678 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.670+0000 7f7a3270c700 1 -- 172.21.15.189:0/762829937 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f7a000557f0 msgr2=0x7f7a00057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.678 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.670+0000 7f7a3270c700 1 --2- 172.21.15.189:0/762829937 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f7a000557f0 0x7f7a00057cb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f7a20006fd0 tx=0x7f7a20006df0).stop 2020-11-20T12:05:20.678 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.670+0000 7f7a3270c700 1 -- 172.21.15.189:0/762829937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a2c0ac9a0 msgr2=0x7f7a2c138cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.678 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.670+0000 7f7a3270c700 1 --2- 172.21.15.189:0/762829937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a2c0ac9a0 0x7f7a2c138cc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f7a2400c010 tx=0x7f7a24016000).stop 2020-11-20T12:05:20.679 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.670+0000 7f7a3270c700 1 -- 172.21.15.189:0/762829937 shutdown_connections 2020-11-20T12:05:20.679 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.670+0000 7f7a3270c700 1 --2- 172.21.15.189:0/762829937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a2c0ac9a0 0x7f7a2c138cc0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.679 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.670+0000 7f7a3270c700 1 --2- 172.21.15.189:0/762829937 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7a2c0661e0 0x7f7a2c13b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.679 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.670+0000 7f7a3270c700 1 --2- 172.21.15.189:0/762829937 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f7a000557f0 0x7f7a00057cb0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.680 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.670+0000 7f7a3270c700 1 --2- 172.21.15.189:0/762829937 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7a2c0a8660 0x7f7a2c1378f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.680 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.670+0000 7f7a3270c700 1 -- 172.21.15.189:0/762829937 >> 172.21.15.189:0/762829937 conn(0x7f7a2c065420 msgr2=0x7f7a2c141100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:20.680 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.670+0000 7f7a3270c700 1 -- 172.21.15.189:0/762829937 shutdown_connections 2020-11-20T12:05:20.680 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.670+0000 7f7a3270c700 1 -- 172.21.15.189:0/762829937 wait complete. 2020-11-20T12:05:20.690 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T12:05:20.819 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.810+0000 7f843c80d700 1 Processor -- start 2020-11-20T12:05:20.819 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f843c80d700 1 -- start start 2020-11-20T12:05:20.820 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f843c80d700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f84380a8660 0x7f84380a9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.821 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f843c80d700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f84380661e0 0x7f84380a9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.821 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f843c80d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84380a9360 con 0x7f84380ac9a0 2020-11-20T12:05:20.821 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f843c80d700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84380aa100 con 0x7f84380661e0 2020-11-20T12:05:20.821 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f843c80d700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8438066760 con 0x7f84380a8660 2020-11-20T12:05:20.821 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f8437fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f84380ac9a0 0x7f84380ace50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50852/0 (socket says 172.21.15.189:50852) 2020-11-20T12:05:20.822 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f8437fff700 1 -- 172.21.15.189:0/1567265460 learned_addr learned my addr 172.21.15.189:0/1567265460 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:05:20.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f84377fe700 1 --2- 172.21.15.189:0/1567265460 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f84380a8660 0x7f84380a9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f84367fc700 1 -- 172.21.15.189:0/1567265460 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1333953268 0 0) 0x7f84380a9360 con 0x7f84380ac9a0 2020-11-20T12:05:20.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f8436ffd700 1 --2- 172.21.15.189:0/1567265460 >> v2:172.21.15.47:3300/0 conn(0x7f84380661e0 0x7f84380a9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f84367fc700 1 -- 172.21.15.189:0/1567265460 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8424003240 con 0x7f84380ac9a0 2020-11-20T12:05:20.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f84367fc700 1 -- 172.21.15.189:0/1567265460 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 486430516 0 0) 0x7f8424003240 con 0x7f84380ac9a0 2020-11-20T12:05:20.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f84367fc700 1 -- 172.21.15.189:0/1567265460 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f84380a8660 msgr2=0x7f84380a9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:20.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f84367fc700 1 --2- 172.21.15.189:0/1567265460 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f84380a8660 0x7f84380a9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f84367fc700 1 -- 172.21.15.189:0/1567265460 >> v2:172.21.15.47:3300/0 conn(0x7f84380661e0 msgr2=0x7f84380a9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:20.827 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f84367fc700 1 --2- 172.21.15.189:0/1567265460 >> v2:172.21.15.47:3300/0 conn(0x7f84380661e0 0x7f84380a9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.827 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f84367fc700 1 -- 172.21.15.189:0/1567265460 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84380a6a70 con 0x7f84380ac9a0 2020-11-20T12:05:20.827 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f84367fc700 1 -- 172.21.15.189:0/1567265460 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f8430001860 con 0x7f84380ac9a0 2020-11-20T12:05:20.827 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f84367fc700 1 -- 172.21.15.189:0/1567265460 >> v1:172.21.15.189:6789/0 conn(0x7f84380ac9a0 legacy=0x7f84380ace50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.827 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f84367fc700 1 --2- 172.21.15.189:0/1567265460 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8424003730 0x7f8424003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.828 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f84367fc700 1 --2- 172.21.15.189:0/1567265460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84240044e0 0x7f8424004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.828 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f84367fc700 1 --2- 172.21.15.189:0/1567265460 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f84380a8660 0x7f8424004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.828 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f84367fc700 1 -- 172.21.15.189:0/1567265460 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84380aa100 con 0x7f8424003730 2020-11-20T12:05:20.828 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f84367fc700 1 -- 172.21.15.189:0/1567265460 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8438066760 con 0x7f84240044e0 2020-11-20T12:05:20.828 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f84367fc700 1 -- 172.21.15.189:0/1567265460 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8424004270 con 0x7f84380a8660 2020-11-20T12:05:20.829 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f8436ffd700 1 --2- 172.21.15.189:0/1567265460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84240044e0 0x7f8424004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.829 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f84377fe700 1 --2- 172.21.15.189:0/1567265460 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8424003730 0x7f8424003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.829 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f8437fff700 1 --2- 172.21.15.189:0/1567265460 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f84380a8660 0x7f8424004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.829 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f8436ffd700 1 -- 172.21.15.189:0/1567265460 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f84380a8660 msgr2=0x7f8424004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:20.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f8436ffd700 1 --2- 172.21.15.189:0/1567265460 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f84380a8660 0x7f8424004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f8436ffd700 1 -- 172.21.15.189:0/1567265460 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8424003730 msgr2=0x7f8424003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:05:20.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f8436ffd700 1 --2- 172.21.15.189:0/1567265460 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8424003730 0x7f8424003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f8436ffd700 1 -- 172.21.15.189:0/1567265460 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8424003240 con 0x7f84240044e0 2020-11-20T12:05:20.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f8437fff700 1 --2- 172.21.15.189:0/1567265460 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f84380a8660 0x7f8424004e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:05:20.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.814+0000 7f8436ffd700 1 --2- 172.21.15.189:0/1567265460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84240044e0 0x7f8424004960 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f8438067840 tx=0x7f842c006df0).ready entity=mon.1 client_cookie=a8144c02f8c63b77 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:20.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.818+0000 7f84367fc700 1 -- 172.21.15.189:0/1567265460 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842c00b330 con 0x7f84240044e0 2020-11-20T12:05:20.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.818+0000 7f84367fc700 1 -- 172.21.15.189:0/1567265460 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f842c00eae0 con 0x7f84240044e0 2020-11-20T12:05:20.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.818+0000 7f84367fc700 1 -- 172.21.15.189:0/1567265460 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842c00ee60 con 0x7f84240044e0 2020-11-20T12:05:20.832 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.818+0000 7f843c80d700 1 -- 172.21.15.189:0/1567265460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84240044e0 msgr2=0x7f8424004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.832 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.818+0000 7f843c80d700 1 --2- 172.21.15.189:0/1567265460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84240044e0 0x7f8424004960 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f8438067840 tx=0x7f842c006df0).stop 2020-11-20T12:05:20.832 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.818+0000 7f8437fff700 1 -- 172.21.15.189:0/1567265460 reap_dead start 2020-11-20T12:05:20.832 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.818+0000 7f843c80d700 1 -- 172.21.15.189:0/1567265460 shutdown_connections 2020-11-20T12:05:20.832 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.818+0000 7f843c80d700 1 -- 172.21.15.189:0/1567265460 >> 172.21.15.189:0/1567265460 conn(0x7f8438065420 msgr2=0x7f8438062080 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:20.833 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.818+0000 7f843c80d700 1 -- 172.21.15.189:0/1567265460 shutdown_connections 2020-11-20T12:05:20.833 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.818+0000 7f843c80d700 1 -- 172.21.15.189:0/1567265460 wait complete. 2020-11-20T12:05:20.835 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.818+0000 7f843c80d700 1 Processor -- start 2020-11-20T12:05:20.836 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.818+0000 7f843c80d700 1 -- start start 2020-11-20T12:05:20.836 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f843c80d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84380a7550 0x7f8438135d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.836 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f843c80d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84380a0820 0x7f8438136200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.836 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f843c80d700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f84381306a0 0x7f843814b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.836 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f843c80d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8438135080 con 0x7f84380a0820 2020-11-20T12:05:20.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f843c80d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8438135270 con 0x7f84380a7550 2020-11-20T12:05:20.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f843c80d700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8438135510 con 0x7f84381306a0 2020-11-20T12:05:20.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f84377fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84380a7550 0x7f8438135d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f84377fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84380a7550 0x7f8438135d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49238/0 (socket says 172.21.15.189:49238) 2020-11-20T12:05:20.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f84377fe700 1 -- 172.21.15.189:0/242325587 learned_addr learned my addr 172.21.15.189:0/242325587 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:05:20.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f8437fff700 1 --2- 172.21.15.189:0/242325587 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f84381306a0 0x7f843814b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f8436ffd700 1 --2- 172.21.15.189:0/242325587 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84380a0820 0x7f8438136200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f84377fe700 1 -- 172.21.15.189:0/242325587 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f84381306a0 msgr2=0x7f843814b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f84377fe700 1 --2- 172.21.15.189:0/242325587 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f84381306a0 0x7f843814b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.839 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f84377fe700 1 -- 172.21.15.189:0/242325587 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84380a0820 msgr2=0x7f8438136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.839 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f84377fe700 1 --2- 172.21.15.189:0/242325587 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84380a0820 0x7f8438136200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.839 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f84377fe700 1 -- 172.21.15.189:0/242325587 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84381357b0 con 0x7f84380a7550 2020-11-20T12:05:20.839 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f8436ffd700 1 --2- 172.21.15.189:0/242325587 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84380a0820 0x7f8438136200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:05:20.839 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f84377fe700 1 --2- 172.21.15.189:0/242325587 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84380a7550 0x7f8438135d10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f8428010c20 tx=0x7f8428008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:20.840 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f84177fe700 1 -- 172.21.15.189:0/242325587 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842800e960 con 0x7f84380a7550 2020-11-20T12:05:20.840 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f84177fe700 1 -- 172.21.15.189:0/242325587 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f842800eac0 con 0x7f84380a7550 2020-11-20T12:05:20.840 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f84177fe700 1 -- 172.21.15.189:0/242325587 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842800ee40 con 0x7f84380a7550 2020-11-20T12:05:20.840 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f843c80d700 1 -- 172.21.15.189:0/242325587 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84380a7930 con 0x7f84380a7550 2020-11-20T12:05:20.840 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f843c80d700 1 -- 172.21.15.189:0/242325587 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f843814b990 con 0x7f84380a7550 2020-11-20T12:05:20.841 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f84177fe700 1 -- 172.21.15.189:0/242325587 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f8428024070 con 0x7f84380a7550 2020-11-20T12:05:20.841 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f84177fe700 1 --2- 172.21.15.189:0/242325587 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f84100557c0 0x7f8410057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:05:20.841 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f84177fe700 1 -- 172.21.15.189:0/242325587 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f842806efc0 con 0x7f84380a7550 2020-11-20T12:05:20.841 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f8436ffd700 1 --2- 172.21.15.189:0/242325587 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f84100557c0 0x7f8410057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:05:20.841 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.822+0000 7f843c80d700 1 -- 172.21.15.189:0/242325587 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8438135940 con 0x7f84380a7550 2020-11-20T12:05:20.842 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.826+0000 7f8436ffd700 1 --2- 172.21.15.189:0/242325587 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f84100557c0 0x7f8410057c80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f8438069d40 tx=0x7f842c016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:05:20.842 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.830+0000 7f84177fe700 1 -- 172.21.15.189:0/242325587 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8428018020 con 0x7f84380a7550 2020-11-20T12:05:20.991 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.986+0000 7f843c80d700 1 -- 172.21.15.189:0/242325587 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8438044f60 con 0x7f84380a7550 2020-11-20T12:05:20.992 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.986+0000 7f84177fe700 1 -- 172.21.15.189:0/242325587 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+10 (secure 0 0 0) 0x7f8428018020 con 0x7f84380a7550 2020-11-20T12:05:20.992 INFO:teuthology.orchestra.run.smithi189.stdout:pg_num: 8 2020-11-20T12:05:20.995 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.986+0000 7f843c80d700 1 -- 172.21.15.189:0/242325587 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f84100557c0 msgr2=0x7f8410057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.995 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.986+0000 7f843c80d700 1 --2- 172.21.15.189:0/242325587 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f84100557c0 0x7f8410057c80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f8438069d40 tx=0x7f842c016000).stop 2020-11-20T12:05:20.995 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.986+0000 7f843c80d700 1 -- 172.21.15.189:0/242325587 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84380a7550 msgr2=0x7f8438135d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:05:20.995 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.986+0000 7f843c80d700 1 --2- 172.21.15.189:0/242325587 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84380a7550 0x7f8438135d10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f8428010c20 tx=0x7f8428008040).stop 2020-11-20T12:05:20.996 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.986+0000 7f843c80d700 1 -- 172.21.15.189:0/242325587 shutdown_connections 2020-11-20T12:05:20.996 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.986+0000 7f843c80d700 1 --2- 172.21.15.189:0/242325587 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f84381306a0 0x7f843814b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.996 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.986+0000 7f843c80d700 1 --2- 172.21.15.189:0/242325587 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84380a7550 0x7f8438135d10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.996 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.986+0000 7f843c80d700 1 --2- 172.21.15.189:0/242325587 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f84100557c0 0x7f8410057c80 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.996 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.986+0000 7f843c80d700 1 --2- 172.21.15.189:0/242325587 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f84380a0820 0x7f8438136200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:05:20.997 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.990+0000 7f843c80d700 1 -- 172.21.15.189:0/242325587 >> 172.21.15.189:0/242325587 conn(0x7f8438065420 msgr2=0x7f843812dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:05:20.997 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.990+0000 7f843c80d700 1 -- 172.21.15.189:0/242325587 shutdown_connections 2020-11-20T12:05:20.997 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:05:20.990+0000 7f843c80d700 1 -- 172.21.15.189:0/242325587 wait complete. 2020-11-20T12:05:21.013 DEBUG:tasks.mds_thrash:Yielding 2020-11-20T12:05:21.013 INFO:teuthology.run_tasks:Running task workunit... 2020-11-20T12:05:21.036 INFO:tasks.workunit:Pulling workunits from ref b86e691bf191566922a8f08e2a5abf64fde8cfcb 2020-11-20T12:05:21.038 INFO:tasks.workunit:Making a separate scratch dir for every client... 2020-11-20T12:05:21.038 INFO:tasks.workunit:timeout=3h 2020-11-20T12:05:21.038 INFO:tasks.workunit:cleanup=True 2020-11-20T12:05:21.039 INFO:teuthology.orchestra.run.smithi189:> stat -- /home/ubuntu/cephtest/mnt.0 2020-11-20T12:05:21.039 INFO:tasks.mds_thrash.fs.[None]:starting mds_do_thrash for fs None 2020-11-20T12:05:21.040 INFO:tasks.mds_thrash.fs.[None]:waiting for 76 secs before thrashing 2020-11-20T12:05:21.048 INFO:teuthology.orchestra.run.smithi189.stdout: File: /home/ubuntu/cephtest/mnt.0 2020-11-20T12:05:21.048 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2020-11-20T12:05:21.048 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 31h/49d Inode: 1 Links: 2 2020-11-20T12:05:21.048 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-20T12:05:21.049 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2020-11-20T12:05:21.049 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2020-11-20 12:04:54.019127511 +0000 2020-11-20T12:05:21.049 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2020-11-20 12:05:13.551795219 +0000 2020-11-20T12:05:21.049 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: - 2020-11-20T12:05:21.050 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2020-11-20T12:05:21.050 INFO:teuthology.orchestra.run.smithi189:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2020-11-20T12:05:21.115 INFO:teuthology.orchestra.run.smithi189:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout b86e691bf191566922a8f08e2a5abf64fde8cfcb 2020-11-20T12:05:21.258 INFO:tasks.workunit.client.0.smithi189.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2020-11-20T12:05:21.804 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:05:21.806 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:05:51.844 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:05:51.847 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:06:10.146 INFO:tasks.workunit.client.0.smithi189.stderr:Note: checking out 'b86e691bf191566922a8f08e2a5abf64fde8cfcb'. 2020-11-20T12:06:10.146 INFO:tasks.workunit.client.0.smithi189.stderr: 2020-11-20T12:06:10.147 INFO:tasks.workunit.client.0.smithi189.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2020-11-20T12:06:10.147 INFO:tasks.workunit.client.0.smithi189.stderr:changes and commit them, and you can discard any commits you make in this 2020-11-20T12:06:10.147 INFO:tasks.workunit.client.0.smithi189.stderr:state without impacting any branches by performing another checkout. 2020-11-20T12:06:10.147 INFO:tasks.workunit.client.0.smithi189.stderr: 2020-11-20T12:06:10.147 INFO:tasks.workunit.client.0.smithi189.stderr:If you want to create a new branch to retain commits you create, you may 2020-11-20T12:06:10.148 INFO:tasks.workunit.client.0.smithi189.stderr:do so (now or later) by using -b with the checkout command again. Example: 2020-11-20T12:06:10.148 INFO:tasks.workunit.client.0.smithi189.stderr: 2020-11-20T12:06:10.148 INFO:tasks.workunit.client.0.smithi189.stderr: git checkout -b 2020-11-20T12:06:10.148 INFO:tasks.workunit.client.0.smithi189.stderr: 2020-11-20T12:06:10.148 INFO:tasks.workunit.client.0.smithi189.stderr:HEAD is now at b86e691bf1 Merge PR #37174 into wip-pdonnell-testing-20201120.053228 2020-11-20T12:06:10.149 INFO:teuthology.orchestra.run.smithi189:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2020-11-20T12:06:10.164 INFO:tasks.workunit.client.0.smithi189.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2020-11-20T12:06:10.166 INFO:tasks.workunit.client.0.smithi189.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2020-11-20T12:06:10.166 INFO:tasks.workunit.client.0.smithi189.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2020-11-20T12:06:10.827 INFO:tasks.workunit.client.0.smithi189.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2020-11-20T12:06:10.949 INFO:tasks.workunit.client.0.smithi189.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2020-11-20T12:06:11.001 INFO:tasks.workunit.client.0.smithi189.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2020-11-20T12:06:11.003 INFO:tasks.workunit.client.0.smithi189.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2020-11-20T12:06:11.004 INFO:tasks.workunit.client.0.smithi189.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2020-11-20T12:06:11.058 INFO:tasks.workunit.client.0.smithi189.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2020-11-20T12:06:11.062 INFO:teuthology.orchestra.run.smithi189:> set -ex 2020-11-20T12:06:11.063 INFO:teuthology.orchestra.run.smithi189:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2020-11-20T12:06:11.110 INFO:teuthology.orchestra.run.smithi189.stderr:+ dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2020-11-20T12:06:11.113 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-tests/ceph-admin-commands.shrgw/test_rgw_reshard.pyrgw/s3_user_quota.plrgw/test_rgw_gc_log.shrgw/run-s3tests.shrgw/run-reshard.shrgw/s3_multipart_upload.plrgw/test_rgw_obj.shrgw/s3_bucket_quota.plrgw/test_rgw_orphan_list.shrgw/test_rgw_throttle.shosdc/stress_objectcacher.shceph-helpers-root.shobjectstore/test_fuse.shrbd/test_rbd_mirror.shrbd/rbd_groups.shrbd/permissions.shrbd/image_read.shrbd/import_export.shrbd/krbd_fallocate.shrbd/rbd-ggate.shrbd/test_rbdmap_RBDMAPFILE.shrbd/krbd_data_pool.shrbd/test_librbd.shrbd/notify_slave.shrbd/krbd_latest_osdmap_on_map.shrbd/huge-tickets.shrbd/rbd_mirror_journal.shrbd/test_admin_socket.shrbd/diff.shrbd/test_librbd_python.shrbd/concurrent.shrbd/merge_diff.shrbd/krbd_exclusive_option.shrbd/krbd_udev_symlinks.shrbd/krbd_namespaces.shrbd/simple_big.shrbd/issue-20295.shrbd/krbd_udev_enumerate.shrbd/test_lock_fence.shrbd/rbd_mirror_fsx_compare.shrbd/qemu_rebuild_object_map.shrbd/rbd_mirror_ha.shrbd/read-flags.shrbd/krbd_stable_pages_required.shrbd/journal.shrbd/krbd_udev_netns.shrbd/verify_pool.shrbd/qemu_dynamic_features.shrbd/map-unmap.shrbd/notify_master.shrbd/cli_generic.shrbd/krbd_wac.shrbd/rbd_mirror_helpers.shrbd/krbd_huge_osdmap.shrbd/rbd_mirror_fsx_prepare.shrbd/krbd_udev_netlink_enobufs.shrbd/rbd_mirror_stress.shrbd/rbd-nbd.shrbd/qemu-iotests.shrbd/kernel.shrbd/test_rbd_tasks.shrbd/qos.shrbd/map-snapshot-io.shrbd/rbd_mirror_snapshot.shrbd/rbd_mirror_bootstrap.shrbd/diff_continuous.shtrue.shcaps/mon_commands.shlibcephfs/test.shsuites/iozone.shsuites/fio.shsuites/iozone-sync.shsuites/fsstress.shsuites/pjd.shsuites/iogen.shsuites/bonnie.shsuites/cephfs_journal_tool_smoke.shsuites/dbench-short.shsuites/dbench.shsuites/blogbench.shsuites/fsync-tester.shsuites/fsx.shsuites/ffsb.shmon/rbd_snaps_ops.shmon/pool_ops.shmon/pg_autoscaler.shmon/osd.shmon/config.shmon/test_config_key_caps.shmon/ping.pymon/crush_ops.shmon/auth_caps.shmon/caps.shmon/test_mon_osdmap_prune.shmon/test_mon_config_key.pycls/test_cls_log.shcls/test_cls_lock.shcls/test_cls_refcount.shcls/test_cls_numops.shcls/test_cls_hello.shcls/test_cls_journal.shcls/test_cls_2pc_queue.shcls/test_cls_cmpomap.shcls/test_cls_sdk.shcls/test_cls_rbd.shcls/test_cls_rgw.shmgr/test_localpool.shdirect_io/test_short_dio_readdirect_io/direct_io_testdirect_io/test_sync_iodirect_io/misc.shdirect_io/big.shcephadm/test_cephadm.shcephadm/test_dashboard_e2e.shcephadm/create_iscsi_disks.shcephadm/test_adoption.shcephadm/test_repos.shrename/pri_rem.shrename/prepare.shrename/pri_pri.shrename/dir_pri_nul.shrename/rem_rem.shrename/pri_nul.shrename/rem_nul.shrename/dir_pri_pri.shrename/all.shrename/rem_pri.shpost-file.sherasure-code/encode-decode-non-regression.sherasure-code/bench.shhadoop/terasort.shhadoop/wordcount.shhadoop/repl.shkernel_untar_build.shrest/test_mgr_rest_api.pyrest/test-restful.shrados/load-gen-big.shrados/stress_watch.shrados/load-gen-mix.shrados/test_dedup_tool.shrados/test_cache_pool.shrados/clone.shrados/load-gen-mix-small-long.shrados/load-gen-mostlyread.shrados/load-gen-mix-small.shrados/version_number_sanity.shrados/test_crash.shrados/test_envlibrados_for_rocksdb.shrados/test_rados_tool.shrados/test_hang.shrados/test_librados_build.shrados/test_pool_quota.shrados/test_health_warnings.shrados/test_large_omap_detection.pyrados/test_alloc_hint.shrados/test_pool_access.shrados/test_python.shrados/test_rados_timeouts.shrados/test.shfs/misc/acl.shfs/misc/dirfrag.shfs/misc/filelock_interrupt.pyfs/misc/direct_io.pyfs/misc/subvolume.shfs/misc/layout_vxattrs.shfs/misc/trivial_sync.shfs/misc/xattrs.shfs/misc/i_complete_vs_rename.shfs/misc/chmod.shfs/misc/filelock_deadlock.pyfs/misc/rstats.shfs/misc/multiple_rsync.shfs/misc/mkpool_layout_vxattrs.shfs/quota/quota.shfs/multiclient_sync_read_eof.pyfs/snap-hierarchy.shfs/test_o_truncfs/snaps/snap-rm-diff.shfs/snaps/snaptest-estale.shfs/snaps/snaptest-snap-rename.shfs/snaps/snaptest-capwb.shfs/snaps/snaptest-authwb.shfs/snaps/snaptest-double-null.shfs/snaps/snaptest-2.shfs/snaps/snaptest-parents.shfs/snaps/snaptest-intodir.shfs/snaps/snaptest-multiple-capsnaps.shfs/snaps/snaptest-1.shfs/snaps/snaptest-upchildrealms.shfs/snaps/snaptest-xattrwb.shfs/snaps/snaptest-realm-split.shfs/snaps/snaptest-dir-rename.shfs/snaps/snaptest-git-ceph.shfs/snaps/snaptest-0.shfs/snaps/snaptest-snap-rm-cmp.shfs/snaps/untar_snap_rm.shfs/snaps/snaptest-hardlink.shfs/upgrade/volume_clientfs/test_o_trunc.shfs/test_python.shfs/norstats/kernel_untar_tar.shrestart/test-backtraces.pycephtool/test_kvstore_tool.shcephtool/test_daemon.shcephtool/test.sh 2020-11-20T12:06:11.113 INFO:teuthology.orchestra.run.smithi189.stderr:9+1 records in 2020-11-20T12:06:11.113 INFO:teuthology.orchestra.run.smithi189.stderr:9+1 records out 2020-11-20T12:06:11.113 INFO:teuthology.orchestra.run.smithi189.stderr:4744 bytes (4.7 kB, 4.6 KiB) copied, 0.00019843 s, 23.9 MB/s 2020-11-20T12:06:11.114 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2020-11-20T12:06:11.115 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2020-11-20T12:06:11.115 INFO:teuthology.orchestra.run.smithi189:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b86e691bf191566922a8f08e2a5abf64fde8cfcb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2020-11-20T12:06:11.209 INFO:tasks.workunit.client.0.smithi189.stderr:--2020-11-20 12:06:11-- http://download.ceph.com/qa/ffsb.tar.bz2 2020-11-20T12:06:11.210 INFO:tasks.workunit.client.0.smithi189.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2020-11-20T12:06:11.291 INFO:tasks.workunit.client.0.smithi189.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2020-11-20T12:06:11.368 INFO:tasks.workunit.client.0.smithi189.stderr:HTTP request sent, awaiting response... 200 OK 2020-11-20T12:06:11.368 INFO:tasks.workunit.client.0.smithi189.stderr:Length: 128931 (126K) [application/octet-stream] 2020-11-20T12:06:11.373 INFO:tasks.workunit.client.0.smithi189.stderr:Saving to: 'ffsb.tar.bz2' 2020-11-20T12:06:11.380 INFO:tasks.workunit.client.0.smithi189.stderr: 2020-11-20T12:06:11.764 INFO:tasks.workunit.client.0.smithi189.stderr: 0K .......... .......... .......... .......... .......... 39% 128K 1s 2020-11-20T12:06:11.831 INFO:tasks.workunit.client.0.smithi189.stderr: 50K .......... .......... .......... .......... .......... 79% 739K 0s 2020-11-20T12:06:11.833 INFO:tasks.workunit.client.0.smithi189.stderr: 100K .......... .......... ..... 100% 17.1M=0.5s 2020-11-20T12:06:11.833 INFO:tasks.workunit.client.0.smithi189.stderr: 2020-11-20T12:06:11.834 INFO:tasks.workunit.client.0.smithi189.stderr:2020-11-20 12:06:11 (274 KB/s) - 'ffsb.tar.bz2' saved [128931/128931] 2020-11-20T12:06:11.834 INFO:tasks.workunit.client.0.smithi189.stderr: 2020-11-20T12:06:11.861 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/ 2020-11-20T12:06:11.863 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/ffsb_fs.c 2020-11-20T12:06:11.868 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/INSTALL 2020-11-20T12:06:11.872 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/fileops.c 2020-11-20T12:06:11.874 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/missing 2020-11-20T12:06:11.877 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/config.h.in 2020-11-20T12:06:11.881 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/config.guess 2020-11-20T12:06:11.884 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/fh.h 2020-11-20T12:06:11.887 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/COPYING 2020-11-20T12:06:11.890 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/util.h 2020-11-20T12:06:11.892 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/main.c 2020-11-20T12:06:11.894 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/ChangeLog 2020-11-20T12:06:11.896 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/rand.c 2020-11-20T12:06:11.897 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/ffsb_fs.h 2020-11-20T12:06:11.899 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/install-sh 2020-11-20T12:06:11.902 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/rwlock.h 2020-11-20T12:06:11.904 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/parser.h 2020-11-20T12:06:11.906 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/filelist.h 2020-11-20T12:06:11.908 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/LICENSE 2020-11-20T12:06:11.911 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/Makefile.in 2020-11-20T12:06:11.913 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/filelist.c 2020-11-20T12:06:11.916 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/configure 2020-11-20T12:06:11.925 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/README 2020-11-20T12:06:11.928 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/cirlist.c 2020-11-20T12:06:11.931 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/metaops.c 2020-11-20T12:06:11.933 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/registration-description 2020-11-20T12:06:11.936 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/list.c 2020-11-20T12:06:11.938 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/list.h 2020-11-20T12:06:11.940 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/rand.h 2020-11-20T12:06:11.942 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/parser.c 2020-11-20T12:06:11.945 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/ffsb_tg.c 2020-11-20T12:06:11.947 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/cirlist.h 2020-11-20T12:06:11.949 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/AUTHORS 2020-11-20T12:06:11.952 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/aclocal.m4 2020-11-20T12:06:11.955 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/configure.in 2020-11-20T12:06:11.958 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/examples/ 2020-11-20T12:06:11.960 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/examples/profile_everything 2020-11-20T12:06:11.963 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/depcomp 2020-11-20T12:06:11.966 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/ffsb_stats.c 2020-11-20T12:06:11.968 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/util.c 2020-11-20T12:06:11.970 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/Makefile.am 2020-11-20T12:06:11.972 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/ffsb_thread.c 2020-11-20T12:06:11.975 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/rbt.c 2020-11-20T12:06:11.977 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/rbt.h 2020-11-20T12:06:11.979 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/ffsb_tg.h 2020-11-20T12:06:11.982 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/stamp-h.in 2020-11-20T12:06:11.984 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/config.sub 2020-11-20T12:06:11.987 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/ffsb_thread.h 2020-11-20T12:06:11.990 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/fileops.h 2020-11-20T12:06:11.992 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/ffsb_op.h 2020-11-20T12:06:11.994 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/public-description 2020-11-20T12:06:11.996 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/mkinstalldirs 2020-11-20T12:06:11.998 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/ffsb.h 2020-11-20T12:06:12.000 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/metaops.h 2020-11-20T12:06:12.001 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/ffsb_op.c 2020-11-20T12:06:12.003 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/fh.c 2020-11-20T12:06:12.006 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/NEWS 2020-11-20T12:06:12.008 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/rwlock.c 2020-11-20T12:06:12.010 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/ffsb_stats.h 2020-11-20T12:06:12.012 INFO:tasks.workunit.client.0.smithi189.stdout:ffsb-6.0-rc2/ffsb_fc.c 2020-11-20T12:06:12.031 INFO:tasks.workunit.client.0.smithi189.stdout:patching file parser.c 2020-11-20T12:06:12.311 INFO:tasks.workunit.client.0.smithi189.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2020-11-20T12:06:13.330 INFO:tasks.workunit.client.0.smithi189.stdout:checking whether build environment is sane... yes 2020-11-20T12:06:13.357 INFO:tasks.workunit.client.0.smithi189.stdout:checking for a thread-safe mkdir -p... /bin/mkdir -p 2020-11-20T12:06:13.358 INFO:tasks.workunit.client.0.smithi189.stdout:checking for gawk... gawk 2020-11-20T12:06:13.516 INFO:tasks.workunit.client.0.smithi189.stdout:checking whether make sets $(MAKE)... yes 2020-11-20T12:06:13.617 INFO:tasks.workunit.client.0.smithi189.stdout:checking build system type... x86_64-unknown-linux-gnu 2020-11-20T12:06:13.619 INFO:tasks.workunit.client.0.smithi189.stdout:checking host system type... x86_64-unknown-linux-gnu 2020-11-20T12:06:13.656 INFO:tasks.workunit.client.0.smithi189.stdout:checking for style of include used by make... GNU 2020-11-20T12:06:13.665 INFO:tasks.workunit.client.0.smithi189.stdout:checking for gcc... gcc 2020-11-20T12:06:13.826 INFO:tasks.workunit.client.0.smithi189.stdout:checking for C compiler default output file name... a.out 2020-11-20T12:06:13.831 INFO:tasks.workunit.client.0.smithi189.stdout:checking whether the C compiler works... yes 2020-11-20T12:06:13.839 INFO:tasks.workunit.client.0.smithi189.stdout:checking whether we are cross compiling... no 2020-11-20T12:06:13.944 INFO:tasks.workunit.client.0.smithi189.stdout:checking for suffix of executables... 2020-11-20T12:06:14.018 INFO:tasks.workunit.client.0.smithi189.stdout:checking for suffix of object files... o 2020-11-20T12:06:14.079 INFO:tasks.workunit.client.0.smithi189.stdout:checking whether we are using the GNU C compiler... yes 2020-11-20T12:06:14.140 INFO:tasks.workunit.client.0.smithi189.stdout:checking whether gcc accepts -g... yes 2020-11-20T12:06:14.265 INFO:tasks.workunit.client.0.smithi189.stdout:checking for gcc option to accept ISO C89... none needed 2020-11-20T12:06:14.509 INFO:tasks.workunit.client.0.smithi189.stdout:checking dependency style of gcc... gcc3 2020-11-20T12:06:14.625 INFO:tasks.workunit.client.0.smithi189.stdout:checking how to run the C preprocessor... gcc -E 2020-11-20T12:06:14.710 INFO:tasks.workunit.client.0.smithi189.stdout:checking for grep that handles long lines and -e... /bin/grep 2020-11-20T12:06:14.713 INFO:tasks.workunit.client.0.smithi189.stdout:checking for egrep... /bin/grep -E 2020-11-20T12:06:14.731 INFO:tasks.workunit.client.0.smithi189.stdout:checking for AIX... no 2020-11-20T12:06:14.769 INFO:tasks.workunit.client.0.smithi189.stdout:checking for gcc... (cached) gcc 2020-11-20T12:06:14.792 INFO:tasks.workunit.client.0.smithi189.stdout:checking whether we are using the GNU C compiler... (cached) yes 2020-11-20T12:06:14.797 INFO:tasks.workunit.client.0.smithi189.stdout:checking whether gcc accepts -g... (cached) yes 2020-11-20T12:06:14.798 INFO:tasks.workunit.client.0.smithi189.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2020-11-20T12:06:14.799 INFO:tasks.workunit.client.0.smithi189.stdout:checking dependency style of gcc... (cached) gcc3 2020-11-20T12:06:14.910 INFO:tasks.workunit.client.0.smithi189.stdout:checking for main in -lm... yes 2020-11-20T12:06:15.032 INFO:tasks.workunit.client.0.smithi189.stdout:checking for main in -lpthread... yes 2020-11-20T12:06:15.348 INFO:tasks.workunit.client.0.smithi189.stdout:checking for ANSI C header files... yes 2020-11-20T12:06:15.466 INFO:tasks.workunit.client.0.smithi189.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2020-11-20T12:06:15.572 INFO:tasks.workunit.client.0.smithi189.stdout:checking for sys/types.h... yes 2020-11-20T12:06:15.671 INFO:tasks.workunit.client.0.smithi189.stdout:checking for sys/stat.h... yes 2020-11-20T12:06:15.780 INFO:tasks.workunit.client.0.smithi189.stdout:checking for stdlib.h... yes 2020-11-20T12:06:15.896 INFO:tasks.workunit.client.0.smithi189.stdout:checking for string.h... yes 2020-11-20T12:06:16.001 INFO:tasks.workunit.client.0.smithi189.stdout:checking for memory.h... yes 2020-11-20T12:06:16.101 INFO:tasks.workunit.client.0.smithi189.stdout:checking for strings.h... yes 2020-11-20T12:06:16.211 INFO:tasks.workunit.client.0.smithi189.stdout:checking for inttypes.h... yes 2020-11-20T12:06:16.322 INFO:tasks.workunit.client.0.smithi189.stdout:checking for stdint.h... yes 2020-11-20T12:06:16.440 INFO:tasks.workunit.client.0.smithi189.stdout:checking for unistd.h... yes 2020-11-20T12:06:16.576 INFO:tasks.workunit.client.0.smithi189.stdout:checking pthread.h usability... yes 2020-11-20T12:06:16.629 INFO:tasks.workunit.client.0.smithi189.stdout:checking pthread.h presence... yes 2020-11-20T12:06:16.631 INFO:tasks.workunit.client.0.smithi189.stdout:checking for pthread.h... yes 2020-11-20T12:06:16.746 INFO:tasks.workunit.client.0.smithi189.stdout:checking fcntl.h usability... yes 2020-11-20T12:06:16.792 INFO:tasks.workunit.client.0.smithi189.stdout:checking fcntl.h presence... yes 2020-11-20T12:06:16.793 INFO:tasks.workunit.client.0.smithi189.stdout:checking for fcntl.h... yes 2020-11-20T12:06:16.909 INFO:tasks.workunit.client.0.smithi189.stdout:checking limits.h usability... yes 2020-11-20T12:06:16.950 INFO:tasks.workunit.client.0.smithi189.stdout:checking limits.h presence... yes 2020-11-20T12:06:16.952 INFO:tasks.workunit.client.0.smithi189.stdout:checking for limits.h... yes 2020-11-20T12:06:16.989 INFO:tasks.workunit.client.0.smithi189.stdout:checking for stdint.h... (cached) yes 2020-11-20T12:06:17.077 INFO:tasks.workunit.client.0.smithi189.stdout:checking sys/time.h usability... yes 2020-11-20T12:06:17.112 INFO:tasks.workunit.client.0.smithi189.stdout:checking sys/time.h presence... yes 2020-11-20T12:06:17.113 INFO:tasks.workunit.client.0.smithi189.stdout:checking for sys/time.h... yes 2020-11-20T12:06:17.152 INFO:tasks.workunit.client.0.smithi189.stdout:checking for unistd.h... (cached) yes 2020-11-20T12:06:17.271 INFO:tasks.workunit.client.0.smithi189.stdout:checking sys/vfs.h usability... yes 2020-11-20T12:06:17.316 INFO:tasks.workunit.client.0.smithi189.stdout:checking sys/vfs.h presence... yes 2020-11-20T12:06:17.318 INFO:tasks.workunit.client.0.smithi189.stdout:checking for sys/vfs.h... yes 2020-11-20T12:06:17.442 INFO:tasks.workunit.client.0.smithi189.stdout:checking sys/limits.h usability... no 2020-11-20T12:06:17.491 INFO:tasks.workunit.client.0.smithi189.stdout:checking sys/limits.h presence... no 2020-11-20T12:06:17.492 INFO:tasks.workunit.client.0.smithi189.stdout:checking for sys/limits.h... no 2020-11-20T12:06:17.569 INFO:tasks.workunit.client.0.smithi189.stdout:checking for an ANSI C-conforming const... yes 2020-11-20T12:06:17.644 INFO:tasks.workunit.client.0.smithi189.stdout:checking whether time.h and sys/time.h may both be included... yes 2020-11-20T12:06:17.648 INFO:tasks.workunit.client.0.smithi189.stdout:checking for function prototypes... yes 2020-11-20T12:06:17.728 INFO:tasks.workunit.client.0.smithi189.stdout:checking whether setvbuf arguments are reversed... no 2020-11-20T12:06:17.894 INFO:tasks.workunit.client.0.smithi189.stdout:checking for system... yes 2020-11-20T12:06:18.057 INFO:tasks.workunit.client.0.smithi189.stdout:checking for gettimeofday... yes 2020-11-20T12:06:18.187 INFO:tasks.workunit.client.0.smithi189.stdout:checking for mkdir... yes 2020-11-20T12:06:18.341 INFO:tasks.workunit.client.0.smithi189.stdout:checking for strerror... yes 2020-11-20T12:06:18.494 INFO:tasks.workunit.client.0.smithi189.stdout:checking for open64... yes 2020-11-20T12:06:18.647 INFO:tasks.workunit.client.0.smithi189.stdout:checking for stat64... yes 2020-11-20T12:06:18.795 INFO:tasks.workunit.client.0.smithi189.stdout:checking for fseeko64... yes 2020-11-20T12:06:18.944 INFO:tasks.workunit.client.0.smithi189.stdout:checking for lrand48_r... yes 2020-11-20T12:06:19.085 INFO:tasks.workunit.client.0.smithi189.stdout:checking for srand48_r... yes 2020-11-20T12:06:19.131 INFO:tasks.workunit.client.0.smithi189.stdout:configure: creating ./config.status 2020-11-20T12:06:19.377 INFO:tasks.workunit.client.0.smithi189.stdout:config.status: creating Makefile 2020-11-20T12:06:19.411 INFO:tasks.workunit.client.0.smithi189.stdout:config.status: creating config.h 2020-11-20T12:06:19.453 INFO:tasks.workunit.client.0.smithi189.stdout:config.status: executing depfiles commands 2020-11-20T12:06:19.577 INFO:tasks.workunit.client.0.smithi189.stdout:config.status: executing default commands 2020-11-20T12:06:19.741 INFO:tasks.workunit.client.0.smithi189.stdout:make all-am 2020-11-20T12:06:19.777 INFO:tasks.workunit.client.0.smithi189.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb-6.0-rc2' 2020-11-20T12:06:19.777 INFO:tasks.workunit.client.0.smithi189.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2020-11-20T12:06:19.871 INFO:tasks.workunit.client.0.smithi189.stderr:fileops.c: In function 'ffsb_readfile': 2020-11-20T12:06:19.871 INFO:tasks.workunit.client.0.smithi189.stderr:fileops.c:138:35: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2020-11-20T12:06:19.872 INFO:tasks.workunit.client.0.smithi189.stderr: printf("Error: read size %llu bytes too big " 2020-11-20T12:06:19.872 INFO:tasks.workunit.client.0.smithi189.stderr: ~~~^ 2020-11-20T12:06:19.873 INFO:tasks.workunit.client.0.smithi189.stderr: %lu 2020-11-20T12:06:19.873 INFO:tasks.workunit.client.0.smithi189.stderr:fileops.c:138:14: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 5 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2020-11-20T12:06:19.873 INFO:tasks.workunit.client.0.smithi189.stderr: printf("Error: read size %llu bytes too big " 2020-11-20T12:06:19.874 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2020-11-20T12:06:19.874 INFO:tasks.workunit.client.0.smithi189.stderr:fileops.c:140:29: note: format string is defined here 2020-11-20T12:06:19.874 INFO:tasks.workunit.client.0.smithi189.stderr: " for file of size %llu bytes\n" 2020-11-20T12:06:19.875 INFO:tasks.workunit.client.0.smithi189.stderr: ~~~^ 2020-11-20T12:06:19.875 INFO:tasks.workunit.client.0.smithi189.stderr: %lu 2020-11-20T12:06:19.876 INFO:tasks.workunit.client.0.smithi189.stderr:fileops.c:144:14: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2020-11-20T12:06:19.876 INFO:tasks.workunit.client.0.smithi189.stderr: printf("minimum file size must be at least " 2020-11-20T12:06:19.876 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2020-11-20T12:06:19.878 INFO:tasks.workunit.client.0.smithi189.stderr:fileops.c:145:12: note: format string is defined here 2020-11-20T12:06:19.878 INFO:tasks.workunit.client.0.smithi189.stderr: " %llu bytes\n", minfilesize); 2020-11-20T12:06:19.878 INFO:tasks.workunit.client.0.smithi189.stderr: ~~~^ 2020-11-20T12:06:19.879 INFO:tasks.workunit.client.0.smithi189.stderr: %lu 2020-11-20T12:06:19.880 INFO:tasks.workunit.client.0.smithi189.stderr:fileops.c: In function 'ffsb_stat': 2020-11-20T12:06:19.880 INFO:tasks.workunit.client.0.smithi189.stderr:fileops.c:542:2: warning: implicit declaration of function 'fhstat'; did you mean 'fstat'? [-Wimplicit-function-declaration] 2020-11-20T12:06:19.881 INFO:tasks.workunit.client.0.smithi189.stderr: fhstat(curfile->name, ft, fs); 2020-11-20T12:06:19.881 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~ 2020-11-20T12:06:19.882 INFO:tasks.workunit.client.0.smithi189.stderr: fstat 2020-11-20T12:06:20.380 INFO:tasks.workunit.client.0.smithi189.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2020-11-20T12:06:20.397 INFO:tasks.workunit.client.0.smithi189.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2020-11-20T12:06:20.777 INFO:tasks.workunit.client.0.smithi189.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2020-11-20T12:06:20.796 INFO:tasks.workunit.client.0.smithi189.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2020-11-20T12:06:20.968 INFO:tasks.workunit.client.0.smithi189.stdout:mv -f .deps/main.Tpo .deps/main.Po 2020-11-20T12:06:20.983 INFO:tasks.workunit.client.0.smithi189.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2020-11-20T12:06:21.070 INFO:tasks.workunit.client.0.smithi189.stderr:fh.c: In function 'fhwrite': 2020-11-20T12:06:21.070 INFO:tasks.workunit.client.0.smithi189.stderr:fh.c:180:18: warning: format '%d' expects argument of type 'int', but argument 2 has type 'ssize_t {aka long int}' [-Wformat=] 2020-11-20T12:06:21.071 INFO:tasks.workunit.client.0.smithi189.stderr: printf("Wrote %d instead of %d bytes.\n" 2020-11-20T12:06:21.071 INFO:tasks.workunit.client.0.smithi189.stderr: ~^ 2020-11-20T12:06:21.071 INFO:tasks.workunit.client.0.smithi189.stderr: %ld 2020-11-20T12:06:21.071 INFO:tasks.workunit.client.0.smithi189.stderr:fh.c: In function 'fhseek': 2020-11-20T12:06:21.071 INFO:tasks.workunit.client.0.smithi189.stderr:fh.c:212:41: warning: format '%lld' expects argument of type 'long long int', but argument 3 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2020-11-20T12:06:21.072 INFO:tasks.workunit.client.0.smithi189.stderr: fprintf(stderr, "tried to seek to %lld\n", offset); 2020-11-20T12:06:21.072 INFO:tasks.workunit.client.0.smithi189.stderr: ~~~^ 2020-11-20T12:06:21.072 INFO:tasks.workunit.client.0.smithi189.stderr: %ld 2020-11-20T12:06:21.072 INFO:tasks.workunit.client.0.smithi189.stderr:fh.c:214:20: warning: format '%lld' expects argument of type 'long long int', but argument 3 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2020-11-20T12:06:21.072 INFO:tasks.workunit.client.0.smithi189.stderr: fprintf(stderr, "tried to seek from current " 2020-11-20T12:06:21.073 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2020-11-20T12:06:21.073 INFO:tasks.workunit.client.0.smithi189.stderr:fh.c:215:21: note: format string is defined here 2020-11-20T12:06:21.073 INFO:tasks.workunit.client.0.smithi189.stderr: "position to %lld\n", offset); 2020-11-20T12:06:21.073 INFO:tasks.workunit.client.0.smithi189.stderr: ~~~^ 2020-11-20T12:06:21.073 INFO:tasks.workunit.client.0.smithi189.stderr: %ld 2020-11-20T12:06:21.153 INFO:tasks.workunit.client.0.smithi189.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2020-11-20T12:06:21.167 INFO:tasks.workunit.client.0.smithi189.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2020-11-20T12:06:21.365 INFO:tasks.workunit.client.0.smithi189.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2020-11-20T12:06:21.374 INFO:tasks.workunit.client.0.smithi189.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2020-11-20T12:06:21.501 INFO:tasks.workunit.client.0.smithi189.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2020-11-20T12:06:21.513 INFO:tasks.workunit.client.0.smithi189.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2020-11-20T12:06:21.602 INFO:tasks.workunit.client.0.smithi189.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2020-11-20T12:06:21.613 INFO:tasks.workunit.client.0.smithi189.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2020-11-20T12:06:21.721 INFO:tasks.workunit.client.0.smithi189.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2020-11-20T12:06:21.734 INFO:tasks.workunit.client.0.smithi189.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2020-11-20T12:06:21.881 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:06:21.884 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:06:21.951 INFO:tasks.workunit.client.0.smithi189.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2020-11-20T12:06:21.968 INFO:tasks.workunit.client.0.smithi189.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2020-11-20T12:06:22.051 INFO:tasks.workunit.client.0.smithi189.stderr:ffsb_tg.c: In function 'tg_print_config_helper': 2020-11-20T12:06:22.051 INFO:tasks.workunit.client.0.smithi189.stderr:ffsb_tg.c:274:35: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2020-11-20T12:06:22.052 INFO:tasks.workunit.client.0.smithi189.stderr: printf("\t read_size = %llu\t(%s)\n", tg->read_size, 2020-11-20T12:06:22.052 INFO:tasks.workunit.client.0.smithi189.stderr: ~~~^ ~~~~~~~~~~~~~ 2020-11-20T12:06:22.052 INFO:tasks.workunit.client.0.smithi189.stderr: %lu 2020-11-20T12:06:22.052 INFO:tasks.workunit.client.0.smithi189.stderr:ffsb_tg.c:283:35: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2020-11-20T12:06:22.052 INFO:tasks.workunit.client.0.smithi189.stderr: printf("\t write_size = %llu\t(%s)\n", tg->write_size, 2020-11-20T12:06:22.053 INFO:tasks.workunit.client.0.smithi189.stderr: ~~~^ ~~~~~~~~~~~~~~ 2020-11-20T12:06:22.053 INFO:tasks.workunit.client.0.smithi189.stderr: %lu 2020-11-20T12:06:22.167 INFO:tasks.workunit.client.0.smithi189.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2020-11-20T12:06:22.180 INFO:tasks.workunit.client.0.smithi189.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2020-11-20T12:06:22.288 INFO:tasks.workunit.client.0.smithi189.stderr:ffsb_fs.c: In function 'verify_file': 2020-11-20T12:06:22.288 INFO:tasks.workunit.client.0.smithi189.stderr:ffsb_fs.c:217:19: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2020-11-20T12:06:22.289 INFO:tasks.workunit.client.0.smithi189.stderr: printf("size %llu bytes for file %s is invalid\n", 2020-11-20T12:06:22.289 INFO:tasks.workunit.client.0.smithi189.stderr: ~~~^ 2020-11-20T12:06:22.289 INFO:tasks.workunit.client.0.smithi189.stderr: %lu 2020-11-20T12:06:22.290 INFO:tasks.workunit.client.0.smithi189.stderr:ffsb_fs.c: In function 'fs_print_config': 2020-11-20T12:06:22.290 INFO:tasks.workunit.client.0.smithi189.stderr:ffsb_fs.c:595:22: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2020-11-20T12:06:22.291 INFO:tasks.workunit.client.0.smithi189.stderr: printf("\t\t %12llu (%6s) -> %u (%.2f\%)\n", 2020-11-20T12:06:22.291 INFO:tasks.workunit.client.0.smithi189.stderr: ~~~~~^ 2020-11-20T12:06:22.291 INFO:tasks.workunit.client.0.smithi189.stderr: %12lu 2020-11-20T12:06:22.291 INFO:tasks.workunit.client.0.smithi189.stderr: fs->size_weights[i].size, 2020-11-20T12:06:22.291 INFO:tasks.workunit.client.0.smithi189.stderr: ~~~~~~~~~~~~~~~~~~~~~~~~ 2020-11-20T12:06:22.292 INFO:tasks.workunit.client.0.smithi189.stderr:ffsb_fs.c:595:43: warning: unknown conversion type character ')' in format [-Wformat=] 2020-11-20T12:06:22.292 INFO:tasks.workunit.client.0.smithi189.stderr: printf("\t\t %12llu (%6s) -> %u (%.2f\%)\n", 2020-11-20T12:06:22.292 INFO:tasks.workunit.client.0.smithi189.stderr: ^ 2020-11-20T12:06:22.292 INFO:tasks.workunit.client.0.smithi189.stderr:ffsb_fs.c:603:36: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2020-11-20T12:06:22.293 INFO:tasks.workunit.client.0.smithi189.stderr: printf("\t min file size = %llu\t(%s)\n", fs->minfilesize, 2020-11-20T12:06:22.293 INFO:tasks.workunit.client.0.smithi189.stderr: ~~~^ ~~~~~~~~~~~~~~~ 2020-11-20T12:06:22.293 INFO:tasks.workunit.client.0.smithi189.stderr: %lu 2020-11-20T12:06:22.293 INFO:tasks.workunit.client.0.smithi189.stderr:ffsb_fs.c:605:36: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2020-11-20T12:06:22.294 INFO:tasks.workunit.client.0.smithi189.stderr: printf("\t max file size = %llu\t(%s)\n", fs->maxfilesize, 2020-11-20T12:06:22.294 INFO:tasks.workunit.client.0.smithi189.stderr: ~~~^ ~~~~~~~~~~~~~~~ 2020-11-20T12:06:22.294 INFO:tasks.workunit.client.0.smithi189.stderr: %lu 2020-11-20T12:06:22.294 INFO:tasks.workunit.client.0.smithi189.stderr:ffsb_fs.c:616:41: warning: unknown conversion type character '\x0a' in format [-Wformat=] 2020-11-20T12:06:22.294 INFO:tasks.workunit.client.0.smithi189.stderr: printf("\t current utilization = %.2f\%\n", getfsutil(fs->basedir)*100); 2020-11-20T12:06:22.295 INFO:tasks.workunit.client.0.smithi189.stderr: ^~ 2020-11-20T12:06:22.295 INFO:tasks.workunit.client.0.smithi189.stderr:ffsb_fs.c:618:42: warning: unknown conversion type character '\x0a' in format [-Wformat=] 2020-11-20T12:06:22.295 INFO:tasks.workunit.client.0.smithi189.stderr: printf("\t desired utilization = %.2lf%\n", fs->desired_fsutil * 100); 2020-11-20T12:06:22.295 INFO:tasks.workunit.client.0.smithi189.stderr: ^~ 2020-11-20T12:06:22.295 INFO:tasks.workunit.client.0.smithi189.stderr:ffsb_fs.c: In function 'fs_needs_stats': 2020-11-20T12:06:22.296 INFO:tasks.workunit.client.0.smithi189.stderr:ffsb_fs.c:627:24: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2020-11-20T12:06:22.296 INFO:tasks.workunit.client.0.smithi189.stderr: return (fs != NULL) ? (int)fs->fsd.config : 0; 2020-11-20T12:06:22.296 INFO:tasks.workunit.client.0.smithi189.stderr: ^ 2020-11-20T12:06:22.434 INFO:tasks.workunit.client.0.smithi189.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2020-11-20T12:06:22.451 INFO:tasks.workunit.client.0.smithi189.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2020-11-20T12:06:22.608 INFO:tasks.workunit.client.0.smithi189.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2020-11-20T12:06:22.619 INFO:tasks.workunit.client.0.smithi189.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2020-11-20T12:06:22.709 INFO:tasks.workunit.client.0.smithi189.stderr:ffsb_op.c: In function 'print_op_results': 2020-11-20T12:06:22.710 INFO:tasks.workunit.client.0.smithi189.stderr:ffsb_op.c:83:23: warning: embedded '\0' in format [-Wformat-contains-nul] 2020-11-20T12:06:22.710 INFO:tasks.workunit.client.0.smithi189.stderr: sprintf(buf, "%s/sec\0", buf); 2020-11-20T12:06:22.710 INFO:tasks.workunit.client.0.smithi189.stderr: ^~ 2020-11-20T12:06:22.710 INFO:tasks.workunit.client.0.smithi189.stderr:ffsb_op.c:86:19: warning: embedded '\0' in format [-Wformat-contains-nul] 2020-11-20T12:06:22.711 INFO:tasks.workunit.client.0.smithi189.stderr: sprintf(buf, "NA\0"); 2020-11-20T12:06:22.711 INFO:tasks.workunit.client.0.smithi189.stderr: ^~ 2020-11-20T12:06:22.711 INFO:tasks.workunit.client.0.smithi189.stderr:ffsb_op.c: In function 'print_results': 2020-11-20T12:06:22.711 INFO:tasks.workunit.client.0.smithi189.stderr:ffsb_op.c:112:61: warning: unknown conversion type character 'T' in format [-Wformat=] 2020-11-20T12:06:22.711 INFO:tasks.workunit.client.0.smithi189.stderr: printf(" Op Name Transactions\t Trans/sec\t% Trans\t % Op Weight\t Throughput\n"); 2020-11-20T12:06:22.712 INFO:tasks.workunit.client.0.smithi189.stderr: ^ 2020-11-20T12:06:22.712 INFO:tasks.workunit.client.0.smithi189.stderr:ffsb_op.c:112:74: warning: unknown conversion type character 'O' in format [-Wformat=] 2020-11-20T12:06:22.712 INFO:tasks.workunit.client.0.smithi189.stderr: printf(" Op Name Transactions\t Trans/sec\t% Trans\t % Op Weight\t Throughput\n"); 2020-11-20T12:06:22.712 INFO:tasks.workunit.client.0.smithi189.stderr: ^ 2020-11-20T12:06:22.777 INFO:tasks.workunit.client.0.smithi189.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2020-11-20T12:06:22.790 INFO:tasks.workunit.client.0.smithi189.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2020-11-20T12:06:22.880 INFO:tasks.workunit.client.0.smithi189.stderr:util.c: In function 'ffsb_bench_gettimeofday': 2020-11-20T12:06:22.881 INFO:tasks.workunit.client.0.smithi189.stderr:util.c:333:46: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2020-11-20T12:06:22.881 INFO:tasks.workunit.client.0.smithi189.stderr: printf("average time for gettimeofday(): %llu nsec\n", average); 2020-11-20T12:06:22.881 INFO:tasks.workunit.client.0.smithi189.stderr: ~~~^ 2020-11-20T12:06:22.881 INFO:tasks.workunit.client.0.smithi189.stderr: %lu 2020-11-20T12:06:22.882 INFO:tasks.workunit.client.0.smithi189.stderr:util.c: In function 'ffsb_bench_getpid': 2020-11-20T12:06:22.882 INFO:tasks.workunit.client.0.smithi189.stderr:util.c:350:40: warning: format '%llu' expects argument of type 'long long unsigned int', but argument 2 has type 'uint64_t {aka long unsigned int}' [-Wformat=] 2020-11-20T12:06:22.882 INFO:tasks.workunit.client.0.smithi189.stderr: printf("average time for getpid(): %llu nsec\n", average); 2020-11-20T12:06:22.882 INFO:tasks.workunit.client.0.smithi189.stderr: ~~~^ 2020-11-20T12:06:22.882 INFO:tasks.workunit.client.0.smithi189.stderr: %lu 2020-11-20T12:06:22.981 INFO:tasks.workunit.client.0.smithi189.stdout:mv -f .deps/util.Tpo .deps/util.Po 2020-11-20T12:06:22.993 INFO:tasks.workunit.client.0.smithi189.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2020-11-20T12:06:23.084 INFO:tasks.workunit.client.0.smithi189.stderr:In file included from parser.h:22:0, 2020-11-20T12:06:23.085 INFO:tasks.workunit.client.0.smithi189.stderr: from parser.c:25: 2020-11-20T12:06:23.085 INFO:tasks.workunit.client.0.smithi189.stderr:parser.c: In function 'init_filesys': 2020-11-20T12:06:23.086 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2020-11-20T12:06:23.087 INFO:tasks.workunit.client.0.smithi189.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2020-11-20T12:06:23.087 INFO:tasks.workunit.client.0.smithi189.stderr: ^ 2020-11-20T12:06:23.087 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:33:28: note: in expansion of macro 'offsetof' 2020-11-20T12:06:23.087 INFO:tasks.workunit.client.0.smithi189.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2020-11-20T12:06:23.087 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~ 2020-11-20T12:06:23.088 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:36:2: note: in expansion of macro 'container_of' 2020-11-20T12:06:23.088 INFO:tasks.workunit.client.0.smithi189.stderr: container_of(ptr, type, member) 2020-11-20T12:06:23.088 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~~~ 2020-11-20T12:06:23.088 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:39:13: note: in expansion of macro 'list_entry' 2020-11-20T12:06:23.088 INFO:tasks.workunit.client.0.smithi189.stderr: for (pos = list_entry((head)->next, typeof(*pos), member); \ 2020-11-20T12:06:23.089 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~ 2020-11-20T12:06:23.089 INFO:tasks.workunit.client.0.smithi189.stderr:parser.c:934:3: note: in expansion of macro 'list_for_each_entry' 2020-11-20T12:06:23.089 INFO:tasks.workunit.client.0.smithi189.stderr: list_for_each_entry(tmp_list, &list_head->list, list) 2020-11-20T12:06:23.089 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~~~~~~~~~~ 2020-11-20T12:06:23.089 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2020-11-20T12:06:23.090 INFO:tasks.workunit.client.0.smithi189.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2020-11-20T12:06:23.090 INFO:tasks.workunit.client.0.smithi189.stderr: ^ 2020-11-20T12:06:23.090 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:33:28: note: in expansion of macro 'offsetof' 2020-11-20T12:06:23.090 INFO:tasks.workunit.client.0.smithi189.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2020-11-20T12:06:23.090 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~ 2020-11-20T12:06:23.091 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:36:2: note: in expansion of macro 'container_of' 2020-11-20T12:06:23.091 INFO:tasks.workunit.client.0.smithi189.stderr: container_of(ptr, type, member) 2020-11-20T12:06:23.091 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~~~ 2020-11-20T12:06:23.091 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:41:13: note: in expansion of macro 'list_entry' 2020-11-20T12:06:23.091 INFO:tasks.workunit.client.0.smithi189.stderr: pos = list_entry(pos->member.next, typeof(*pos), member)) 2020-11-20T12:06:23.092 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~ 2020-11-20T12:06:23.092 INFO:tasks.workunit.client.0.smithi189.stderr:parser.c:934:3: note: in expansion of macro 'list_for_each_entry' 2020-11-20T12:06:23.092 INFO:tasks.workunit.client.0.smithi189.stderr: list_for_each_entry(tmp_list, &list_head->list, list) 2020-11-20T12:06:23.092 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~~~~~~~~~~ 2020-11-20T12:06:23.092 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2020-11-20T12:06:23.093 INFO:tasks.workunit.client.0.smithi189.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2020-11-20T12:06:23.093 INFO:tasks.workunit.client.0.smithi189.stderr: ^ 2020-11-20T12:06:23.093 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:33:28: note: in expansion of macro 'offsetof' 2020-11-20T12:06:23.093 INFO:tasks.workunit.client.0.smithi189.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2020-11-20T12:06:23.093 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~ 2020-11-20T12:06:23.094 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:36:2: note: in expansion of macro 'container_of' 2020-11-20T12:06:23.094 INFO:tasks.workunit.client.0.smithi189.stderr: container_of(ptr, type, member) 2020-11-20T12:06:23.094 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~~~ 2020-11-20T12:06:23.094 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:39:13: note: in expansion of macro 'list_entry' 2020-11-20T12:06:23.094 INFO:tasks.workunit.client.0.smithi189.stderr: for (pos = list_entry((head)->next, typeof(*pos), member); \ 2020-11-20T12:06:23.095 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~ 2020-11-20T12:06:23.095 INFO:tasks.workunit.client.0.smithi189.stderr:parser.c:941:3: note: in expansion of macro 'list_for_each_entry' 2020-11-20T12:06:23.095 INFO:tasks.workunit.client.0.smithi189.stderr: list_for_each_entry(tmp_list, &list_head->list, list) { 2020-11-20T12:06:23.095 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~~~~~~~~~~ 2020-11-20T12:06:23.095 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2020-11-20T12:06:23.096 INFO:tasks.workunit.client.0.smithi189.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2020-11-20T12:06:23.096 INFO:tasks.workunit.client.0.smithi189.stderr: ^ 2020-11-20T12:06:23.096 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:33:28: note: in expansion of macro 'offsetof' 2020-11-20T12:06:23.096 INFO:tasks.workunit.client.0.smithi189.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2020-11-20T12:06:23.096 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~ 2020-11-20T12:06:23.097 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:36:2: note: in expansion of macro 'container_of' 2020-11-20T12:06:23.097 INFO:tasks.workunit.client.0.smithi189.stderr: container_of(ptr, type, member) 2020-11-20T12:06:23.097 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~~~ 2020-11-20T12:06:23.097 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:41:13: note: in expansion of macro 'list_entry' 2020-11-20T12:06:23.097 INFO:tasks.workunit.client.0.smithi189.stderr: pos = list_entry(pos->member.next, typeof(*pos), member)) 2020-11-20T12:06:23.098 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~ 2020-11-20T12:06:23.098 INFO:tasks.workunit.client.0.smithi189.stderr:parser.c:941:3: note: in expansion of macro 'list_for_each_entry' 2020-11-20T12:06:23.098 INFO:tasks.workunit.client.0.smithi189.stderr: list_for_each_entry(tmp_list, &list_head->list, list) { 2020-11-20T12:06:23.098 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~~~~~~~~~~ 2020-11-20T12:06:23.099 INFO:tasks.workunit.client.0.smithi189.stderr:parser.c: In function 'init_tg_stats': 2020-11-20T12:06:23.099 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2020-11-20T12:06:23.100 INFO:tasks.workunit.client.0.smithi189.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2020-11-20T12:06:23.100 INFO:tasks.workunit.client.0.smithi189.stderr: ^ 2020-11-20T12:06:23.100 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:33:28: note: in expansion of macro 'offsetof' 2020-11-20T12:06:23.100 INFO:tasks.workunit.client.0.smithi189.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2020-11-20T12:06:23.101 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~ 2020-11-20T12:06:23.101 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:36:2: note: in expansion of macro 'container_of' 2020-11-20T12:06:23.101 INFO:tasks.workunit.client.0.smithi189.stderr: container_of(ptr, type, member) 2020-11-20T12:06:23.101 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~~~ 2020-11-20T12:06:23.101 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:39:13: note: in expansion of macro 'list_entry' 2020-11-20T12:06:23.102 INFO:tasks.workunit.client.0.smithi189.stderr: for (pos = list_entry((head)->next, typeof(*pos), member); \ 2020-11-20T12:06:23.102 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~ 2020-11-20T12:06:23.102 INFO:tasks.workunit.client.0.smithi189.stderr:parser.c:971:6: note: in expansion of macro 'list_for_each_entry' 2020-11-20T12:06:23.102 INFO:tasks.workunit.client.0.smithi189.stderr: list_for_each_entry(tmp_list, 2020-11-20T12:06:23.103 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~~~~~~~~~~ 2020-11-20T12:06:23.103 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2020-11-20T12:06:23.103 INFO:tasks.workunit.client.0.smithi189.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2020-11-20T12:06:23.103 INFO:tasks.workunit.client.0.smithi189.stderr: ^ 2020-11-20T12:06:23.103 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:33:28: note: in expansion of macro 'offsetof' 2020-11-20T12:06:23.103 INFO:tasks.workunit.client.0.smithi189.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2020-11-20T12:06:23.104 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~ 2020-11-20T12:06:23.104 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:36:2: note: in expansion of macro 'container_of' 2020-11-20T12:06:23.104 INFO:tasks.workunit.client.0.smithi189.stderr: container_of(ptr, type, member) 2020-11-20T12:06:23.104 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~~~ 2020-11-20T12:06:23.104 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:41:13: note: in expansion of macro 'list_entry' 2020-11-20T12:06:23.105 INFO:tasks.workunit.client.0.smithi189.stderr: pos = list_entry(pos->member.next, typeof(*pos), member)) 2020-11-20T12:06:23.105 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~ 2020-11-20T12:06:23.105 INFO:tasks.workunit.client.0.smithi189.stderr:parser.c:971:6: note: in expansion of macro 'list_for_each_entry' 2020-11-20T12:06:23.105 INFO:tasks.workunit.client.0.smithi189.stderr: list_for_each_entry(tmp_list, 2020-11-20T12:06:23.105 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~~~~~~~~~~ 2020-11-20T12:06:23.106 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2020-11-20T12:06:23.106 INFO:tasks.workunit.client.0.smithi189.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2020-11-20T12:06:23.106 INFO:tasks.workunit.client.0.smithi189.stderr: ^ 2020-11-20T12:06:23.106 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:33:28: note: in expansion of macro 'offsetof' 2020-11-20T12:06:23.106 INFO:tasks.workunit.client.0.smithi189.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2020-11-20T12:06:23.107 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~ 2020-11-20T12:06:23.107 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:36:2: note: in expansion of macro 'container_of' 2020-11-20T12:06:23.107 INFO:tasks.workunit.client.0.smithi189.stderr: container_of(ptr, type, member) 2020-11-20T12:06:23.107 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~~~ 2020-11-20T12:06:23.107 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:39:13: note: in expansion of macro 'list_entry' 2020-11-20T12:06:23.108 INFO:tasks.workunit.client.0.smithi189.stderr: for (pos = list_entry((head)->next, typeof(*pos), member); \ 2020-11-20T12:06:23.108 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~ 2020-11-20T12:06:23.108 INFO:tasks.workunit.client.0.smithi189.stderr:parser.c:980:6: note: in expansion of macro 'list_for_each_entry' 2020-11-20T12:06:23.108 INFO:tasks.workunit.client.0.smithi189.stderr: list_for_each_entry(tmp_list, 2020-11-20T12:06:23.108 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~~~~~~~~~~ 2020-11-20T12:06:23.109 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2020-11-20T12:06:23.109 INFO:tasks.workunit.client.0.smithi189.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2020-11-20T12:06:23.109 INFO:tasks.workunit.client.0.smithi189.stderr: ^ 2020-11-20T12:06:23.109 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:33:28: note: in expansion of macro 'offsetof' 2020-11-20T12:06:23.109 INFO:tasks.workunit.client.0.smithi189.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2020-11-20T12:06:23.110 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~ 2020-11-20T12:06:23.110 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:36:2: note: in expansion of macro 'container_of' 2020-11-20T12:06:23.110 INFO:tasks.workunit.client.0.smithi189.stderr: container_of(ptr, type, member) 2020-11-20T12:06:23.110 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~~~ 2020-11-20T12:06:23.110 INFO:tasks.workunit.client.0.smithi189.stderr:list.h:41:13: note: in expansion of macro 'list_entry' 2020-11-20T12:06:23.111 INFO:tasks.workunit.client.0.smithi189.stderr: pos = list_entry(pos->member.next, typeof(*pos), member)) 2020-11-20T12:06:23.111 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~ 2020-11-20T12:06:23.111 INFO:tasks.workunit.client.0.smithi189.stderr:parser.c:980:6: note: in expansion of macro 'list_for_each_entry' 2020-11-20T12:06:23.111 INFO:tasks.workunit.client.0.smithi189.stderr: list_for_each_entry(tmp_list, 2020-11-20T12:06:23.111 INFO:tasks.workunit.client.0.smithi189.stderr: ^~~~~~~~~~~~~~~~~~~ 2020-11-20T12:06:23.362 INFO:tasks.workunit.client.0.smithi189.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2020-11-20T12:06:23.372 INFO:tasks.workunit.client.0.smithi189.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2020-11-20T12:06:23.505 INFO:tasks.workunit.client.0.smithi189.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2020-11-20T12:06:23.517 INFO:tasks.workunit.client.0.smithi189.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2020-11-20T12:06:23.663 INFO:tasks.workunit.client.0.smithi189.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2020-11-20T12:06:23.679 INFO:tasks.workunit.client.0.smithi189.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2020-11-20T12:06:23.761 INFO:tasks.workunit.client.0.smithi189.stdout:mv -f .deps/list.Tpo .deps/list.Po 2020-11-20T12:06:23.780 INFO:tasks.workunit.client.0.smithi189.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2020-11-20T12:06:24.024 INFO:tasks.workunit.client.0.smithi189.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb-6.0-rc2' 2020-11-20T12:06:24.086 INFO:tasks.workunit.client.0.smithi189.stdout:FFSB version 6.0-RC2 started 2020-11-20T12:06:24.086 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-20T12:06:24.089 INFO:tasks.workunit.client.0.smithi189.stdout:benchmark time = 300 2020-11-20T12:06:24.089 INFO:tasks.workunit.client.0.smithi189.stdout:ThreadGroup 0 2020-11-20T12:06:24.090 INFO:tasks.workunit.client.0.smithi189.stdout:================ 2020-11-20T12:06:24.090 INFO:tasks.workunit.client.0.smithi189.stdout: num_threads = 32 2020-11-20T12:06:24.090 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-20T12:06:24.090 INFO:tasks.workunit.client.0.smithi189.stdout: read_random = off 2020-11-20T12:06:24.091 INFO:tasks.workunit.client.0.smithi189.stdout: read_size = 0 (0B) 2020-11-20T12:06:24.091 INFO:tasks.workunit.client.0.smithi189.stdout: read_blocksize = 0 (0B) 2020-11-20T12:06:24.091 INFO:tasks.workunit.client.0.smithi189.stdout: read_skip = off 2020-11-20T12:06:24.091 INFO:tasks.workunit.client.0.smithi189.stdout: read_skipsize = 0 (0B) 2020-11-20T12:06:24.091 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-20T12:06:24.092 INFO:tasks.workunit.client.0.smithi189.stdout: write_random = on 2020-11-20T12:06:24.092 INFO:tasks.workunit.client.0.smithi189.stdout: write_size = 5242880 (5MB) 2020-11-20T12:06:24.092 INFO:tasks.workunit.client.0.smithi189.stdout: fsync_file = 0 2020-11-20T12:06:24.092 INFO:tasks.workunit.client.0.smithi189.stdout: write_blocksize = 4096 (4KB) 2020-11-20T12:06:24.092 INFO:tasks.workunit.client.0.smithi189.stdout: wait time = 0 2020-11-20T12:06:24.093 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-20T12:06:24.093 INFO:tasks.workunit.client.0.smithi189.stdout: op weights 2020-11-20T12:06:24.093 INFO:tasks.workunit.client.0.smithi189.stdout: read = 0 (0.00%) 2020-11-20T12:06:24.093 INFO:tasks.workunit.client.0.smithi189.stdout: readall = 0 (0.00%) 2020-11-20T12:06:24.093 INFO:tasks.workunit.client.0.smithi189.stdout: write = 1 (100.00%) 2020-11-20T12:06:24.094 INFO:tasks.workunit.client.0.smithi189.stdout: create = 0 (0.00%) 2020-11-20T12:06:24.094 INFO:tasks.workunit.client.0.smithi189.stdout: append = 0 (0.00%) 2020-11-20T12:06:24.094 INFO:tasks.workunit.client.0.smithi189.stdout: delete = 0 (0.00%) 2020-11-20T12:06:24.094 INFO:tasks.workunit.client.0.smithi189.stdout: metaop = 0 (0.00%) 2020-11-20T12:06:24.095 INFO:tasks.workunit.client.0.smithi189.stdout: createdir = 0 (0.00%) 2020-11-20T12:06:24.095 INFO:tasks.workunit.client.0.smithi189.stdout: stat = 0 (0.00%) 2020-11-20T12:06:24.095 INFO:tasks.workunit.client.0.smithi189.stdout: writeall = 0 (0.00%) 2020-11-20T12:06:24.095 INFO:tasks.workunit.client.0.smithi189.stdout: writeall_fsync = 0 (0.00%) 2020-11-20T12:06:24.095 INFO:tasks.workunit.client.0.smithi189.stdout: open_close = 0 (0.00%) 2020-11-20T12:06:24.096 INFO:tasks.workunit.client.0.smithi189.stdout: write_fsync = 0 (0.00%) 2020-11-20T12:06:24.096 INFO:tasks.workunit.client.0.smithi189.stdout: create_fsync = 0 (0.00%) 2020-11-20T12:06:24.096 INFO:tasks.workunit.client.0.smithi189.stdout: append_fsync = 0 (0.00%) 2020-11-20T12:06:24.096 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-20T12:06:24.096 INFO:tasks.workunit.client.0.smithi189.stdout:FileSystem . 2020-11-20T12:06:24.097 INFO:tasks.workunit.client.0.smithi189.stdout:========== 2020-11-20T12:06:24.097 INFO:tasks.workunit.client.0.smithi189.stdout: num_dirs = 0 2020-11-20T12:06:24.097 INFO:tasks.workunit.client.0.smithi189.stdout: starting files = 128 2020-11-20T12:06:24.097 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-20T12:06:24.098 INFO:tasks.workunit.client.0.smithi189.stdout: min file size = 104857600 (100MB) 2020-11-20T12:06:24.098 INFO:tasks.workunit.client.0.smithi189.stdout: max file size = 104857600 (100MB) 2020-11-20T12:06:24.098 INFO:tasks.workunit.client.0.smithi189.stdout: directio = off 2020-11-20T12:06:24.098 INFO:tasks.workunit.client.0.smithi189.stdout: alignedio = on 2020-11-20T12:06:24.098 INFO:tasks.workunit.client.0.smithi189.stdout: bufferedio = off 2020-11-20T12:06:24.099 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-20T12:06:24.099 INFO:tasks.workunit.client.0.smithi189.stdout: aging is off 2020-11-20T12:06:24.099 INFO:tasks.workunit.client.0.smithi189.stdout: current utilization = 0.00% 2020-11-20T12:06:24.100 INFO:tasks.workunit.client.0.smithi189.stdout: 2020-11-20T12:06:24.100 INFO:tasks.workunit.client.0.smithi189.stdout:checking existing fs: . 2020-11-20T12:06:24.100 INFO:tasks.workunit.client.0.smithi189.stdout:recreating new fileset 2020-11-20T12:06:24.100 INFO:tasks.workunit.client.0.smithi189.stderr:opendir: No such file or directory 2020-11-20T12:06:37.041 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:06:37.175 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.170+0000 7fce54a5c700 1 Processor -- start 2020-11-20T12:06:37.175 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.170+0000 7fce54a5c700 1 -- start start 2020-11-20T12:06:37.177 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.170+0000 7fce54a5c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fce50062220 0x7fce5013a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:37.177 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.170+0000 7fce54a5c700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fce50062600 0x7fce50066fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:37.179 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.170+0000 7fce54a5c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce500a9bb0 con 0x7fce50137750 2020-11-20T12:06:37.179 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.170+0000 7fce54a5c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce500674c0 con 0x7fce50062220 2020-11-20T12:06:37.179 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.170+0000 7fce54a5c700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce5013ab40 con 0x7fce50062600 2020-11-20T12:06:37.179 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.170+0000 7fce4f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fce50062220 0x7fce5013a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:37.179 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.170+0000 7fce4ffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fce50137750 0x7fce50060a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50892/0 (socket says 172.21.15.189:50892) 2020-11-20T12:06:37.180 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.170+0000 7fce4ffff700 1 -- 172.21.15.189:0/2112008276 learned_addr learned my addr 172.21.15.189:0/2112008276 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:06:37.180 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.170+0000 7fce4effd700 1 --2- 172.21.15.189:0/2112008276 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fce50062600 0x7fce50066fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:37.180 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.170+0000 7fce4e7fc700 1 -- 172.21.15.189:0/2112008276 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 659942693 0 0) 0x7fce48001450 con 0x7fce50137750 2020-11-20T12:06:37.180 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.170+0000 7fce4f7fe700 1 -- 172.21.15.189:0/2112008276 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fce50062600 msgr2=0x7fce50066fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:37.180 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.170+0000 7fce4f7fe700 1 --2- 172.21.15.189:0/2112008276 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fce50062600 0x7fce50066fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:37.181 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.170+0000 7fce4f7fe700 1 -- 172.21.15.189:0/2112008276 >> v1:172.21.15.189:6789/0 conn(0x7fce50137750 legacy=0x7fce50060a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:37.181 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.170+0000 7fce4f7fe700 1 -- 172.21.15.189:0/2112008276 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce5013ade0 con 0x7fce50062220 2020-11-20T12:06:37.181 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.170+0000 7fce4f7fe700 1 --2- 172.21.15.189:0/2112008276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce50062220 0x7fce5013a4a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fce4000b9a0 tx=0x7fce40006df0).ready entity=mon.1 client_cookie=d1dc29db5312d3b4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:37.181 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.170+0000 7fce4e7fc700 1 -- 172.21.15.189:0/2112008276 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce40009840 con 0x7fce50062220 2020-11-20T12:06:37.182 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.170+0000 7fce4e7fc700 1 -- 172.21.15.189:0/2112008276 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce400099a0 con 0x7fce50062220 2020-11-20T12:06:37.182 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.170+0000 7fce4e7fc700 1 -- 172.21.15.189:0/2112008276 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce40009d20 con 0x7fce50062220 2020-11-20T12:06:37.185 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce54a5c700 1 -- 172.21.15.189:0/2112008276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce50062220 msgr2=0x7fce5013a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:37.185 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce54a5c700 1 --2- 172.21.15.189:0/2112008276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce50062220 0x7fce5013a4a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fce4000b9a0 tx=0x7fce40006df0).stop 2020-11-20T12:06:37.185 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce54a5c700 1 -- 172.21.15.189:0/2112008276 shutdown_connections 2020-11-20T12:06:37.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce54a5c700 1 --2- 172.21.15.189:0/2112008276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce50062220 0x7fce5013a4a0 secure :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0x7fce4000b9a0 tx=0x7fce40006df0).stop 2020-11-20T12:06:37.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce54a5c700 1 --2- 172.21.15.189:0/2112008276 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fce50062600 0x7fce50066fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:37.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce54a5c700 1 -- 172.21.15.189:0/2112008276 >> 172.21.15.189:0/2112008276 conn(0x7fce50065fe0 msgr2=0x7fce50061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:37.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce54a5c700 1 -- 172.21.15.189:0/2112008276 shutdown_connections 2020-11-20T12:06:37.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce54a5c700 1 -- 172.21.15.189:0/2112008276 wait complete. 2020-11-20T12:06:37.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce54a5c700 1 Processor -- start 2020-11-20T12:06:37.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce54a5c700 1 -- start start 2020-11-20T12:06:37.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce54a5c700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fce50062600 0x7fce5013e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:37.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce54a5c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fce50137750 0x7fce5013d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:37.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce54a5c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce5014d480 0x7fce50139d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:37.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce54a5c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5013de80 con 0x7fce50137750 2020-11-20T12:06:37.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce54a5c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5014d900 con 0x7fce5014d480 2020-11-20T12:06:37.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce54a5c700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce5013dae0 con 0x7fce50062600 2020-11-20T12:06:37.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce4ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce5014d480 0x7fce50139d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:37.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce4ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce5014d480 0x7fce50139d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49274/0 (socket says 172.21.15.189:49274) 2020-11-20T12:06:37.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce4ffff700 1 -- 172.21.15.189:0/616504226 learned_addr learned my addr 172.21.15.189:0/616504226 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:06:37.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce4effd700 1 --2- 172.21.15.189:0/616504226 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fce50137750 0x7fce5013d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:37.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce4f7fe700 1 --2- 172.21.15.189:0/616504226 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fce50062600 0x7fce5013e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:37.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce4ffff700 1 -- 172.21.15.189:0/616504226 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fce50062600 msgr2=0x7fce5013e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:37.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce4ffff700 1 --2- 172.21.15.189:0/616504226 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fce50062600 0x7fce5013e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:37.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce4ffff700 1 -- 172.21.15.189:0/616504226 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fce50137750 msgr2=0x7fce5013d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:37.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce4ffff700 1 --2- 172.21.15.189:0/616504226 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fce50137750 0x7fce5013d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:37.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce4ffff700 1 -- 172.21.15.189:0/616504226 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce5013e060 con 0x7fce5014d480 2020-11-20T12:06:37.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce4ffff700 1 --2- 172.21.15.189:0/616504226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce5014d480 0x7fce50139d10 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fce4800cae0 tx=0x7fce48001320).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:37.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce2f7fe700 1 -- 172.21.15.189:0/616504226 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce48005b80 con 0x7fce5014d480 2020-11-20T12:06:37.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce2f7fe700 1 -- 172.21.15.189:0/616504226 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce48005ce0 con 0x7fce5014d480 2020-11-20T12:06:37.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce54a5c700 1 -- 172.21.15.189:0/616504226 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce501381a0 con 0x7fce5014d480 2020-11-20T12:06:37.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.174+0000 7fce2f7fe700 1 -- 172.21.15.189:0/616504226 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce48007340 con 0x7fce5014d480 2020-11-20T12:06:37.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.178+0000 7fce54a5c700 1 -- 172.21.15.189:0/616504226 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce50131ae0 con 0x7fce5014d480 2020-11-20T12:06:37.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.178+0000 7fce54a5c700 1 -- 172.21.15.189:0/616504226 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce5013d930 con 0x7fce5014d480 2020-11-20T12:06:37.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.178+0000 7fce2f7fe700 1 -- 172.21.15.189:0/616504226 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fce48010030 con 0x7fce5014d480 2020-11-20T12:06:37.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.178+0000 7fce2f7fe700 1 --2- 172.21.15.189:0/616504226 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fce280557c0 0x7fce28057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:37.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.178+0000 7fce2f7fe700 1 -- 172.21.15.189:0/616504226 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fce4806eb10 con 0x7fce5014d480 2020-11-20T12:06:37.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.178+0000 7fce4f7fe700 1 --2- 172.21.15.189:0/616504226 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fce280557c0 0x7fce28057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:37.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.178+0000 7fce4f7fe700 1 --2- 172.21.15.189:0/616504226 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fce280557c0 0x7fce28057c80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fce50065b60 tx=0x7fce40015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:37.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.182+0000 7fce2f7fe700 1 -- 172.21.15.189:0/616504226 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fce48044a30 con 0x7fce5014d480 2020-11-20T12:06:37.404 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.398+0000 7fce54a5c700 1 -- 172.21.15.189:0/616504226 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce5013e1f0 con 0x7fce5014d480 2020-11-20T12:06:37.405 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.398+0000 7fce2f7fe700 1 -- 172.21.15.189:0/616504226 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2134 (secure 0 0 0) 0x7fce48048050 con 0x7fce5014d480 2020-11-20T12:06:37.405 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:06:37.405 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2660417436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2660417436},{"type":"v1","addr":"172.21.15.47:6833","nonce":2660417436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:04:56.025055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.189:6835/1042140684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1042140684},{"type":"v1","addr":"172.21.15.189:6835","nonce":1042140684}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:06:37.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.402+0000 7fce54a5c700 1 -- 172.21.15.189:0/616504226 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fce280557c0 msgr2=0x7fce28057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:37.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.402+0000 7fce54a5c700 1 --2- 172.21.15.189:0/616504226 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fce280557c0 0x7fce28057c80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fce50065b60 tx=0x7fce40015000).stop 2020-11-20T12:06:37.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.402+0000 7fce54a5c700 1 -- 172.21.15.189:0/616504226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce5014d480 msgr2=0x7fce50139d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:37.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.402+0000 7fce54a5c700 1 --2- 172.21.15.189:0/616504226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce5014d480 0x7fce50139d10 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fce4800cae0 tx=0x7fce48001320).stop 2020-11-20T12:06:37.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.402+0000 7fce54a5c700 1 -- 172.21.15.189:0/616504226 shutdown_connections 2020-11-20T12:06:37.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.402+0000 7fce54a5c700 1 --2- 172.21.15.189:0/616504226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce5014d480 0x7fce50139d10 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:37.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.402+0000 7fce54a5c700 1 --2- 172.21.15.189:0/616504226 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fce50062600 0x7fce5013e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:37.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.402+0000 7fce54a5c700 1 --2- 172.21.15.189:0/616504226 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fce280557c0 0x7fce28057c80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:37.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.402+0000 7fce54a5c700 1 --2- 172.21.15.189:0/616504226 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fce50137750 0x7fce5013d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:37.409 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.402+0000 7fce54a5c700 1 -- 172.21.15.189:0/616504226 >> 172.21.15.189:0/616504226 conn(0x7fce50065fe0 msgr2=0x7fce500aa290 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:37.409 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.402+0000 7fce54a5c700 1 -- 172.21.15.189:0/616504226 shutdown_connections 2020-11-20T12:06:37.409 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.402+0000 7fce54a5c700 1 -- 172.21.15.189:0/616504226 wait complete. 2020-11-20T12:06:37.409 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 6 2020-11-20T12:06:37.420 INFO:tasks.mds_thrash.fs.[None]:kill mds.a (rank=0) 2020-11-20T12:06:37.421 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T12:06:37.421 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:06:37.433 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T12:06:37.434 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:06:37.611 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.606+0000 7fa92359e700 1 Processor -- start 2020-11-20T12:06:37.612 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.606+0000 7fa92359e700 1 -- start start 2020-11-20T12:06:37.613 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.606+0000 7fa92359e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa924060310 0x7fa9240a1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:37.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.606+0000 7fa92359e700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa9240a8450 0x7fa9240a9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:37.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.606+0000 7fa92359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9240a8fa0 con 0x7fa9240a2440 2020-11-20T12:06:37.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.606+0000 7fa92359e700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9240618b0 con 0x7fa9240a8450 2020-11-20T12:06:37.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.606+0000 7fa92359e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9240a9240 con 0x7fa924060310 2020-11-20T12:06:37.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.606+0000 7fa91bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa9240a2440 0x7fa9240a7d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50906/0 (socket says 172.21.15.189:50906) 2020-11-20T12:06:37.615 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.606+0000 7fa91bfff700 1 -- 172.21.15.189:0/447901485 learned_addr learned my addr 172.21.15.189:0/447901485 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:06:37.615 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.606+0000 7fa92259c700 1 --2- 172.21.15.189:0/447901485 >> v2:172.21.15.189:3300/0 conn(0x7fa924060310 0x7fa9240a1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:37.615 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.606+0000 7fa922d9d700 1 --2- 172.21.15.189:0/447901485 >> v2:172.21.15.47:3300/0 conn(0x7fa9240a8450 0x7fa9240a9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:37.615 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.606+0000 7fa921d9b700 1 -- 172.21.15.189:0/447901485 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682985886 0 0) 0x7fa9240a8fa0 con 0x7fa9240a2440 2020-11-20T12:06:37.615 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.606+0000 7fa921d9b700 1 -- 172.21.15.189:0/447901485 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa904003240 con 0x7fa9240a2440 2020-11-20T12:06:37.616 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.606+0000 7fa92259c700 1 -- 172.21.15.189:0/447901485 >> v2:172.21.15.47:3300/0 conn(0x7fa9240a8450 msgr2=0x7fa9240a9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:37.616 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.606+0000 7fa92259c700 1 --2- 172.21.15.189:0/447901485 >> v2:172.21.15.47:3300/0 conn(0x7fa9240a8450 0x7fa9240a9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:37.616 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.606+0000 7fa92259c700 1 -- 172.21.15.189:0/447901485 >> v1:172.21.15.189:6789/0 conn(0x7fa9240a2440 legacy=0x7fa9240a7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:37.616 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.606+0000 7fa92259c700 1 -- 172.21.15.189:0/447901485 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9240a94e0 con 0x7fa924060310 2020-11-20T12:06:37.617 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.610+0000 7fa92259c700 1 --2- 172.21.15.189:0/447901485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa924060310 0x7fa9240a1f50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fa9240ae330 tx=0x7fa914006df0).ready entity=mon.1 client_cookie=ac74f271b1e22fb server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:37.617 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.610+0000 7fa921d9b700 1 -- 172.21.15.189:0/447901485 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9140096d0 con 0x7fa924060310 2020-11-20T12:06:37.617 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.610+0000 7fa921d9b700 1 -- 172.21.15.189:0/447901485 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa914009830 con 0x7fa924060310 2020-11-20T12:06:37.617 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.610+0000 7fa921d9b700 1 -- 172.21.15.189:0/447901485 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa914009bb0 con 0x7fa924060310 2020-11-20T12:06:37.620 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.610+0000 7fa92359e700 1 -- 172.21.15.189:0/447901485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa924060310 msgr2=0x7fa9240a1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:37.620 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.610+0000 7fa92359e700 1 --2- 172.21.15.189:0/447901485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa924060310 0x7fa9240a1f50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fa9240ae330 tx=0x7fa914006df0).stop 2020-11-20T12:06:37.620 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.610+0000 7fa92359e700 1 -- 172.21.15.189:0/447901485 shutdown_connections 2020-11-20T12:06:37.620 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.610+0000 7fa92359e700 1 --2- 172.21.15.189:0/447901485 >> v2:172.21.15.47:3300/0 conn(0x7fa9240a8450 0x7fa9240a9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:37.621 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.610+0000 7fa92359e700 1 --2- 172.21.15.189:0/447901485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa924060310 0x7fa9240a1f50 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:37.621 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.610+0000 7fa92359e700 1 -- 172.21.15.189:0/447901485 >> 172.21.15.189:0/447901485 conn(0x7fa924063210 msgr2=0x7fa924060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:37.621 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.610+0000 7fa92359e700 1 -- 172.21.15.189:0/447901485 shutdown_connections 2020-11-20T12:06:37.621 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.610+0000 7fa92359e700 1 -- 172.21.15.189:0/447901485 wait complete. 2020-11-20T12:06:37.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.610+0000 7fa92359e700 1 Processor -- start 2020-11-20T12:06:37.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.610+0000 7fa92359e700 1 -- start start 2020-11-20T12:06:37.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.610+0000 7fa92359e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa924060310 0x7fa9240ac050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:37.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.610+0000 7fa92359e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa9240a2440 0x7fa924066b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:37.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.610+0000 7fa92359e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9240a8450 0x7fa924067360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:37.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.610+0000 7fa92359e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9241362c0 con 0x7fa924060310 2020-11-20T12:06:37.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.610+0000 7fa92359e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa924067c70 con 0x7fa9240a8450 2020-11-20T12:06:37.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.610+0000 7fa92359e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa924066420 con 0x7fa9240a2440 2020-11-20T12:06:37.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.610+0000 7fa922d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9240a8450 0x7fa924067360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:37.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.610+0000 7fa922d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9240a8450 0x7fa924067360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49292/0 (socket says 172.21.15.189:49292) 2020-11-20T12:06:37.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.610+0000 7fa922d9d700 1 -- 172.21.15.189:0/2329184265 learned_addr learned my addr 172.21.15.189:0/2329184265 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:06:37.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.614+0000 7fa91bfff700 1 --2- 172.21.15.189:0/2329184265 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa9240a2440 0x7fa924066b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:37.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.614+0000 7fa92259c700 1 --2- 172.21.15.189:0/2329184265 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa924060310 0x7fa9240ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:37.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.614+0000 7fa922d9d700 1 -- 172.21.15.189:0/2329184265 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa9240a2440 msgr2=0x7fa924066b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:37.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.614+0000 7fa922d9d700 1 --2- 172.21.15.189:0/2329184265 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa9240a2440 0x7fa924066b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:37.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.614+0000 7fa922d9d700 1 -- 172.21.15.189:0/2329184265 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa924060310 msgr2=0x7fa9240ac050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:37.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.614+0000 7fa922d9d700 1 --2- 172.21.15.189:0/2329184265 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa924060310 0x7fa9240ac050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:37.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.614+0000 7fa922d9d700 1 -- 172.21.15.189:0/2329184265 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9240666c0 con 0x7fa9240a8450 2020-11-20T12:06:37.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.614+0000 7fa922d9d700 1 --2- 172.21.15.189:0/2329184265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9240a8450 0x7fa924067360 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fa91c00ec30 tx=0x7fa91c0091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:37.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.614+0000 7fa91a7fc700 1 -- 172.21.15.189:0/2329184265 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c00cb50 con 0x7fa9240a8450 2020-11-20T12:06:37.626 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.614+0000 7fa91a7fc700 1 -- 172.21.15.189:0/2329184265 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa91c00ccb0 con 0x7fa9240a8450 2020-11-20T12:06:37.626 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.614+0000 7fa92359e700 1 -- 172.21.15.189:0/2329184265 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa924067050 con 0x7fa9240a8450 2020-11-20T12:06:37.626 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.614+0000 7fa91a7fc700 1 -- 172.21.15.189:0/2329184265 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c012620 con 0x7fa9240a8450 2020-11-20T12:06:37.626 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.614+0000 7fa92359e700 1 -- 172.21.15.189:0/2329184265 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa924067850 con 0x7fa9240a8450 2020-11-20T12:06:37.626 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.614+0000 7fa92359e700 1 -- 172.21.15.189:0/2329184265 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9240606f0 con 0x7fa9240a8450 2020-11-20T12:06:37.627 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.614+0000 7fa91a7fc700 1 -- 172.21.15.189:0/2329184265 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fa91c022070 con 0x7fa9240a8450 2020-11-20T12:06:37.628 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.614+0000 7fa91a7fc700 1 --2- 172.21.15.189:0/2329184265 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa9000557f0 0x7fa900057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:37.628 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.614+0000 7fa91a7fc700 1 -- 172.21.15.189:0/2329184265 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa91c06ee00 con 0x7fa9240a8450 2020-11-20T12:06:37.628 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.614+0000 7fa92259c700 1 --2- 172.21.15.189:0/2329184265 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa9000557f0 0x7fa900057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:37.628 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.618+0000 7fa92259c700 1 --2- 172.21.15.189:0/2329184265 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa9000557f0 0x7fa900057cb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fa91400b150 tx=0x7fa914015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:37.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.622+0000 7fa91a7fc700 1 -- 172.21.15.189:0/2329184265 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa91c044ca0 con 0x7fa9240a8450 2020-11-20T12:06:37.861 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.854+0000 7fa92359e700 1 -- 172.21.15.189:0/2329184265 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9240446d0 con 0x7fa9240a8450 2020-11-20T12:06:37.862 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.858+0000 7fa91a7fc700 1 -- 172.21.15.189:0/2329184265 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2134 (secure 0 0 0) 0x7fa91c016070 con 0x7fa9240a8450 2020-11-20T12:06:37.862 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:06:37.862 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2660417436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2660417436},{"type":"v1","addr":"172.21.15.47:6833","nonce":2660417436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:04:56.025055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.189:6835/1042140684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1042140684},{"type":"v1","addr":"172.21.15.189:6835","nonce":1042140684}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:06:37.865 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.858+0000 7fa92359e700 1 -- 172.21.15.189:0/2329184265 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa9000557f0 msgr2=0x7fa900057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:37.865 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.858+0000 7fa92359e700 1 --2- 172.21.15.189:0/2329184265 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa9000557f0 0x7fa900057cb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fa91400b150 tx=0x7fa914015000).stop 2020-11-20T12:06:37.866 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.858+0000 7fa92359e700 1 -- 172.21.15.189:0/2329184265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9240a8450 msgr2=0x7fa924067360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:37.866 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.858+0000 7fa92359e700 1 --2- 172.21.15.189:0/2329184265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9240a8450 0x7fa924067360 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fa91c00ec30 tx=0x7fa91c0091d0).stop 2020-11-20T12:06:37.866 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.858+0000 7fa92359e700 1 -- 172.21.15.189:0/2329184265 shutdown_connections 2020-11-20T12:06:37.866 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.858+0000 7fa92359e700 1 --2- 172.21.15.189:0/2329184265 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa9000557f0 0x7fa900057cb0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:37.866 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.858+0000 7fa92359e700 1 --2- 172.21.15.189:0/2329184265 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa924060310 0x7fa9240ac050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:37.867 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.858+0000 7fa92359e700 1 --2- 172.21.15.189:0/2329184265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9240a8450 0x7fa924067360 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:37.867 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.858+0000 7fa92359e700 1 --2- 172.21.15.189:0/2329184265 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa9240a2440 0x7fa924066b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:37.867 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.858+0000 7fa92359e700 1 -- 172.21.15.189:0/2329184265 >> 172.21.15.189:0/2329184265 conn(0x7fa924063210 msgr2=0x7fa9240aa360 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:37.867 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.862+0000 7fa92359e700 1 -- 172.21.15.189:0/2329184265 shutdown_connections 2020-11-20T12:06:37.868 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:37.862+0000 7fa92359e700 1 -- 172.21.15.189:0/2329184265 wait complete. 2020-11-20T12:06:37.868 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 6 2020-11-20T12:06:37.880 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:06:39.881 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:06:40.014 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b6657c700 1 Processor -- start 2020-11-20T12:06:40.014 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b6657c700 1 -- start start 2020-11-20T12:06:40.016 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b6657c700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5b60066de0 0x7f5b6013ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:40.017 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b6657c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5b6013cea0 0x7f5b6013d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:40.017 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b6657c700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b6013b070 con 0x7f5b600671c0 2020-11-20T12:06:40.017 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b6657c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b6013b300 con 0x7f5b6013cea0 2020-11-20T12:06:40.017 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b6657c700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6013dd30 con 0x7f5b60066de0 2020-11-20T12:06:40.018 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b64b19700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5b6013cea0 0x7f5b6013d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:40.018 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b64b19700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5b6013cea0 0x7f5b6013d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49306/0 (socket says 172.21.15.189:49306) 2020-11-20T12:06:40.018 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b5ffff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5b60066de0 0x7f5b6013ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:40.018 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b64b19700 1 -- 172.21.15.189:0/2326070657 learned_addr learned my addr 172.21.15.189:0/2326070657 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:06:40.018 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b64b19700 1 -- 172.21.15.189:0/2326070657 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5b60066de0 msgr2=0x7f5b6013ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:40.019 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b64b19700 1 --2- 172.21.15.189:0/2326070657 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5b60066de0 0x7f5b6013ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:40.019 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b64b19700 1 -- 172.21.15.189:0/2326070657 >> v1:172.21.15.47:6789/0 conn(0x7f5b600671c0 legacy=0x7f5b6013e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:40.019 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b64b19700 1 -- 172.21.15.189:0/2326070657 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b60067620 con 0x7f5b6013cea0 2020-11-20T12:06:40.020 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b64b19700 1 --2- 172.21.15.189:0/2326070657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b6013cea0 0x7f5b6013d350 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f5b50006fd0 tx=0x7f5b50006eb0).ready entity=mon.1 client_cookie=180c789cd5538eb4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:40.020 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b5effd700 1 -- 172.21.15.189:0/2326070657 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b500098a0 con 0x7f5b6013cea0 2020-11-20T12:06:40.020 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b5effd700 1 -- 172.21.15.189:0/2326070657 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b50009a00 con 0x7f5b6013cea0 2020-11-20T12:06:40.020 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b5effd700 1 -- 172.21.15.189:0/2326070657 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b50009d80 con 0x7f5b6013cea0 2020-11-20T12:06:40.020 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b6657c700 1 -- 172.21.15.189:0/2326070657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b6013cea0 msgr2=0x7f5b6013d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:40.021 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b6657c700 1 --2- 172.21.15.189:0/2326070657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b6013cea0 0x7f5b6013d350 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f5b50006fd0 tx=0x7f5b50006eb0).stop 2020-11-20T12:06:40.021 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b6657c700 1 -- 172.21.15.189:0/2326070657 shutdown_connections 2020-11-20T12:06:40.021 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b6657c700 1 --2- 172.21.15.189:0/2326070657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b6013cea0 0x7f5b6013d350 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:40.021 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b6657c700 1 --2- 172.21.15.189:0/2326070657 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5b60066de0 0x7f5b6013ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:40.021 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b6657c700 1 -- 172.21.15.189:0/2326070657 >> 172.21.15.189:0/2326070657 conn(0x7f5b60061f60 msgr2=0x7f5b60062340 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:40.022 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b6657c700 1 -- 172.21.15.189:0/2326070657 shutdown_connections 2020-11-20T12:06:40.022 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.010+0000 7f5b6657c700 1 -- 172.21.15.189:0/2326070657 wait complete. 2020-11-20T12:06:40.025 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.014+0000 7f5b6657c700 1 Processor -- start 2020-11-20T12:06:40.025 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.014+0000 7f5b6657c700 1 -- start start 2020-11-20T12:06:40.025 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.014+0000 7f5b6657c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b60066de0 0x7f5b600a5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:40.026 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.014+0000 7f5b6657c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b600671c0 0x7f5b60136410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:40.026 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.014+0000 7f5b6657c700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5b6013cea0 0x7f5b600a3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:40.026 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.014+0000 7f5b6657c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6014c970 con 0x7f5b60066de0 2020-11-20T12:06:40.026 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.014+0000 7f5b6657c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b600a2fb0 con 0x7f5b600671c0 2020-11-20T12:06:40.027 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.014+0000 7f5b6657c700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b600a3250 con 0x7f5b6013cea0 2020-11-20T12:06:40.027 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.014+0000 7f5b5f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b600671c0 0x7f5b60136410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:40.027 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.014+0000 7f5b5f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b600671c0 0x7f5b60136410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49310/0 (socket says 172.21.15.189:49310) 2020-11-20T12:06:40.027 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.014+0000 7f5b5f7fe700 1 -- 172.21.15.189:0/4217833237 learned_addr learned my addr 172.21.15.189:0/4217833237 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:06:40.027 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.014+0000 7f5b5ffff700 1 --2- 172.21.15.189:0/4217833237 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b60066de0 0x7f5b600a5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:40.028 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.014+0000 7f5b64b19700 1 --2- 172.21.15.189:0/4217833237 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5b6013cea0 0x7f5b600a3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:40.028 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.014+0000 7f5b5f7fe700 1 -- 172.21.15.189:0/4217833237 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5b6013cea0 msgr2=0x7f5b600a3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:40.028 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.014+0000 7f5b5f7fe700 1 --2- 172.21.15.189:0/4217833237 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5b6013cea0 0x7f5b600a3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:40.028 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.014+0000 7f5b5f7fe700 1 -- 172.21.15.189:0/4217833237 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b60066de0 msgr2=0x7f5b600a5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:40.029 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.014+0000 7f5b5f7fe700 1 --2- 172.21.15.189:0/4217833237 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b60066de0 0x7f5b600a5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:40.029 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.014+0000 7f5b5f7fe700 1 -- 172.21.15.189:0/4217833237 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b600a1c40 con 0x7f5b600671c0 2020-11-20T12:06:40.030 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.014+0000 7f5b5f7fe700 1 --2- 172.21.15.189:0/4217833237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b600671c0 0x7f5b60136410 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f5b4800ca90 tx=0x7f5b48001460).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:40.030 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.014+0000 7f5b57fff700 1 -- 172.21.15.189:0/4217833237 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b48006870 con 0x7f5b600671c0 2020-11-20T12:06:40.030 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.014+0000 7f5b57fff700 1 -- 172.21.15.189:0/4217833237 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b480069d0 con 0x7f5b600671c0 2020-11-20T12:06:40.031 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.014+0000 7f5b6657c700 1 -- 172.21.15.189:0/4217833237 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b600a40b0 con 0x7f5b600671c0 2020-11-20T12:06:40.031 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.014+0000 7f5b57fff700 1 -- 172.21.15.189:0/4217833237 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b48006d70 con 0x7f5b600671c0 2020-11-20T12:06:40.032 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.014+0000 7f5b6657c700 1 -- 172.21.15.189:0/4217833237 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b600a2c40 con 0x7f5b600671c0 2020-11-20T12:06:40.032 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.014+0000 7f5b6657c700 1 -- 172.21.15.189:0/4217833237 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b60044f60 con 0x7f5b600671c0 2020-11-20T12:06:40.032 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.014+0000 7f5b57fff700 1 -- 172.21.15.189:0/4217833237 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f5b48018020 con 0x7f5b600671c0 2020-11-20T12:06:40.033 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.018+0000 7f5b57fff700 1 --2- 172.21.15.189:0/4217833237 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5b3c0557c0 0x7f5b3c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:40.033 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.018+0000 7f5b57fff700 1 -- 172.21.15.189:0/4217833237 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5b4806e1d0 con 0x7f5b600671c0 2020-11-20T12:06:40.033 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.018+0000 7f5b5ffff700 1 --2- 172.21.15.189:0/4217833237 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5b3c0557c0 0x7f5b3c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:40.034 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.018+0000 7f5b5ffff700 1 --2- 172.21.15.189:0/4217833237 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5b3c0557c0 0x7f5b3c057c80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f5b58006fd0 tx=0x7f5b58006df0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:40.035 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.022+0000 7f5b57fff700 1 -- 172.21.15.189:0/4217833237 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5b48044070 con 0x7f5b600671c0 2020-11-20T12:06:40.233 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.226+0000 7f5b6657c700 1 -- 172.21.15.189:0/4217833237 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b60061380 con 0x7f5b600671c0 2020-11-20T12:06:40.233 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.230+0000 7f5b57fff700 1 -- 172.21.15.189:0/4217833237 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2134 (secure 0 0 0) 0x7f5b48047690 con 0x7f5b600671c0 2020-11-20T12:06:40.234 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:06:40.467 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2660417436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2660417436},{"type":"v1","addr":"172.21.15.47:6833","nonce":2660417436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:04:56.025055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.189:6835/1042140684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1042140684},{"type":"v1","addr":"172.21.15.189:6835","nonce":1042140684}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:06:40.474 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.230+0000 7f5b6657c700 1 -- 172.21.15.189:0/4217833237 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5b3c0557c0 msgr2=0x7f5b3c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:40.474 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.230+0000 7f5b6657c700 1 --2- 172.21.15.189:0/4217833237 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5b3c0557c0 0x7f5b3c057c80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f5b58006fd0 tx=0x7f5b58006df0).stop 2020-11-20T12:06:40.474 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.230+0000 7f5b6657c700 1 -- 172.21.15.189:0/4217833237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b600671c0 msgr2=0x7f5b60136410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:40.475 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.230+0000 7f5b6657c700 1 --2- 172.21.15.189:0/4217833237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b600671c0 0x7f5b60136410 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f5b4800ca90 tx=0x7f5b48001460).stop 2020-11-20T12:06:40.475 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.230+0000 7f5b6657c700 1 -- 172.21.15.189:0/4217833237 shutdown_connections 2020-11-20T12:06:40.475 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.230+0000 7f5b6657c700 1 --2- 172.21.15.189:0/4217833237 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5b3c0557c0 0x7f5b3c057c80 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:40.475 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.230+0000 7f5b6657c700 1 --2- 172.21.15.189:0/4217833237 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b60066de0 0x7f5b600a5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:40.475 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.230+0000 7f5b6657c700 1 --2- 172.21.15.189:0/4217833237 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5b6013cea0 0x7f5b600a3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:40.476 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.230+0000 7f5b6657c700 1 --2- 172.21.15.189:0/4217833237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b600671c0 0x7f5b60136410 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:40.476 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.230+0000 7f5b6657c700 1 -- 172.21.15.189:0/4217833237 >> 172.21.15.189:0/4217833237 conn(0x7f5b60061f60 msgr2=0x7f5b600b1520 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:40.476 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.230+0000 7f5b6657c700 1 -- 172.21.15.189:0/4217833237 shutdown_connections 2020-11-20T12:06:40.476 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:40.230+0000 7f5b6657c700 1 -- 172.21.15.189:0/4217833237 wait complete. 2020-11-20T12:06:40.476 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 6 2020-11-20T12:06:40.477 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:06:42.478 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:06:42.615 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.610+0000 7f304f35d700 1 Processor -- start 2020-11-20T12:06:42.615 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.610+0000 7f304f35d700 1 -- start start 2020-11-20T12:06:42.616 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.610+0000 7f304f35d700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3048066de0 0x7f304813ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:42.617 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.610+0000 7f304f35d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f30480671c0 0x7f304813e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:42.617 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.610+0000 7f304f35d700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304813b070 con 0x7f304813e6c0 2020-11-20T12:06:42.617 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.610+0000 7f304f35d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f304813b300 con 0x7f30480671c0 2020-11-20T12:06:42.617 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.610+0000 7f304f35d700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f304813dd30 con 0x7f3048066de0 2020-11-20T12:06:42.618 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.610+0000 7f304c8f8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f30480671c0 0x7f304813e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:42.618 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.610+0000 7f304c8f8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f30480671c0 0x7f304813e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49322/0 (socket says 172.21.15.189:49322) 2020-11-20T12:06:42.618 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.610+0000 7f304c8f8700 1 -- 172.21.15.189:0/2283283206 learned_addr learned my addr 172.21.15.189:0/2283283206 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:06:42.618 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.610+0000 7f304d0f9700 1 --2- 172.21.15.189:0/2283283206 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3048066de0 0x7f304813ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:42.621 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.610+0000 7f304c8f8700 1 -- 172.21.15.189:0/2283283206 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3048066de0 msgr2=0x7f304813ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:42.621 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.610+0000 7f304c8f8700 1 --2- 172.21.15.189:0/2283283206 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3048066de0 0x7f304813ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:42.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.610+0000 7f304c8f8700 1 -- 172.21.15.189:0/2283283206 >> v1:172.21.15.47:6789/0 conn(0x7f304813e6c0 legacy=0x7f304813cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:42.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.610+0000 7f304c8f8700 1 -- 172.21.15.189:0/2283283206 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3048067620 con 0x7f30480671c0 2020-11-20T12:06:42.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.610+0000 7f304c8f8700 1 --2- 172.21.15.189:0/2283283206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30480671c0 0x7f304813e1d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f3038006fd0 tx=0x7f3038006eb0).ready entity=mon.1 client_cookie=2311edb94fd509bf server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:42.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.610+0000 7f303ffff700 1 -- 172.21.15.189:0/2283283206 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30380098a0 con 0x7f30480671c0 2020-11-20T12:06:42.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.614+0000 7f303ffff700 1 -- 172.21.15.189:0/2283283206 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3038009a00 con 0x7f30480671c0 2020-11-20T12:06:42.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.614+0000 7f303ffff700 1 -- 172.21.15.189:0/2283283206 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3038009d80 con 0x7f30480671c0 2020-11-20T12:06:42.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.614+0000 7f304f35d700 1 -- 172.21.15.189:0/2283283206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30480671c0 msgr2=0x7f304813e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:42.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.614+0000 7f304f35d700 1 --2- 172.21.15.189:0/2283283206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30480671c0 0x7f304813e1d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f3038006fd0 tx=0x7f3038006eb0).stop 2020-11-20T12:06:42.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.614+0000 7f304f35d700 1 -- 172.21.15.189:0/2283283206 shutdown_connections 2020-11-20T12:06:42.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.614+0000 7f304f35d700 1 --2- 172.21.15.189:0/2283283206 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3048066de0 0x7f304813ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:42.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.614+0000 7f304f35d700 1 --2- 172.21.15.189:0/2283283206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30480671c0 0x7f304813e1d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:42.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.614+0000 7f304f35d700 1 -- 172.21.15.189:0/2283283206 >> 172.21.15.189:0/2283283206 conn(0x7f3048061f60 msgr2=0x7f3048062340 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:42.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.614+0000 7f304f35d700 1 -- 172.21.15.189:0/2283283206 shutdown_connections 2020-11-20T12:06:42.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.614+0000 7f304f35d700 1 -- 172.21.15.189:0/2283283206 wait complete. 2020-11-20T12:06:42.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.614+0000 7f304f35d700 1 Processor -- start 2020-11-20T12:06:42.626 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.614+0000 7f304f35d700 1 -- start start 2020-11-20T12:06:42.627 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.618+0000 7f304f35d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3048066de0 0x7f30481409e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:42.627 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.618+0000 7f304f35d700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f30480671c0 0x7f30480a22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:42.627 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.618+0000 7f304f35d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f304813e6c0 0x7f3048149ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:42.627 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.618+0000 7f304f35d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30480a1ec0 con 0x7f304813e6c0 2020-11-20T12:06:42.627 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.618+0000 7f304f35d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30480a34c0 con 0x7f3048066de0 2020-11-20T12:06:42.628 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.618+0000 7f304f35d700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30481417b0 con 0x7f30480671c0 2020-11-20T12:06:42.628 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.618+0000 7f304d0f9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3048066de0 0x7f30481409e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:42.628 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.618+0000 7f304d0f9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3048066de0 0x7f30481409e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49324/0 (socket says 172.21.15.189:49324) 2020-11-20T12:06:42.628 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.618+0000 7f304d0f9700 1 -- 172.21.15.189:0/63954792 learned_addr learned my addr 172.21.15.189:0/63954792 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:06:42.628 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.618+0000 7f304c8f8700 1 --2- 172.21.15.189:0/63954792 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f30480671c0 0x7f30480a22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:42.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.618+0000 7f304d8fa700 1 --2- 172.21.15.189:0/63954792 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f304813e6c0 0x7f3048149ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:42.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.618+0000 7f304d0f9700 1 -- 172.21.15.189:0/63954792 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f30480671c0 msgr2=0x7f30480a22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:42.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.618+0000 7f304d0f9700 1 --2- 172.21.15.189:0/63954792 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f30480671c0 0x7f30480a22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:42.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.618+0000 7f304d0f9700 1 -- 172.21.15.189:0/63954792 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f304813e6c0 msgr2=0x7f3048149ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:42.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.618+0000 7f304d0f9700 1 --2- 172.21.15.189:0/63954792 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f304813e6c0 0x7f3048149ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:42.630 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.618+0000 7f304d0f9700 1 -- 172.21.15.189:0/63954792 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3048149880 con 0x7f3048066de0 2020-11-20T12:06:42.630 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.618+0000 7f304d0f9700 1 --2- 172.21.15.189:0/63954792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3048066de0 0x7f30481409e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f304400eba0 tx=0x7f3044006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:42.630 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.618+0000 7f303d7fa700 1 -- 172.21.15.189:0/63954792 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304400c8e0 con 0x7f3048066de0 2020-11-20T12:06:42.630 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.618+0000 7f303d7fa700 1 -- 172.21.15.189:0/63954792 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f304400ca40 con 0x7f3048066de0 2020-11-20T12:06:42.630 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.618+0000 7f304f35d700 1 -- 172.21.15.189:0/63954792 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30480a3760 con 0x7f3048066de0 2020-11-20T12:06:42.631 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.618+0000 7f303d7fa700 1 -- 172.21.15.189:0/63954792 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304400cdc0 con 0x7f3048066de0 2020-11-20T12:06:42.631 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.618+0000 7f304f35d700 1 -- 172.21.15.189:0/63954792 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30480a2a00 con 0x7f3048066de0 2020-11-20T12:06:42.631 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.622+0000 7f304f35d700 1 -- 172.21.15.189:0/63954792 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3048044f60 con 0x7f3048066de0 2020-11-20T12:06:42.632 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.622+0000 7f303d7fa700 1 -- 172.21.15.189:0/63954792 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f3044012600 con 0x7f3048066de0 2020-11-20T12:06:42.632 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.622+0000 7f303d7fa700 1 --2- 172.21.15.189:0/63954792 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f30200557c0 0x7f3020057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:42.632 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.622+0000 7f303d7fa700 1 -- 172.21.15.189:0/63954792 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f304406e9a0 con 0x7f3048066de0 2020-11-20T12:06:42.633 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.622+0000 7f304c8f8700 1 --2- 172.21.15.189:0/63954792 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f30200557c0 0x7f3020057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:42.633 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.622+0000 7f304c8f8700 1 --2- 172.21.15.189:0/63954792 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f30200557c0 0x7f3020057c80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f3038006ef0 tx=0x7f3038015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:42.634 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.630+0000 7f303d7fa700 1 -- 172.21.15.189:0/63954792 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3044044840 con 0x7f3048066de0 2020-11-20T12:06:42.834 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.830+0000 7f304f35d700 1 -- 172.21.15.189:0/63954792 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3048061380 con 0x7f3048066de0 2020-11-20T12:06:42.834 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.830+0000 7f303d7fa700 1 -- 172.21.15.189:0/63954792 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2134 (secure 0 0 0) 0x7f304406e030 con 0x7f3048066de0 2020-11-20T12:06:42.835 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:06:42.835 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2660417436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2660417436},{"type":"v1","addr":"172.21.15.47:6833","nonce":2660417436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:04:56.025055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.189:6835/1042140684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1042140684},{"type":"v1","addr":"172.21.15.189:6835","nonce":1042140684}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:06:42.836 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.830+0000 7f304f35d700 1 -- 172.21.15.189:0/63954792 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f30200557c0 msgr2=0x7f3020057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:42.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.830+0000 7f304f35d700 1 --2- 172.21.15.189:0/63954792 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f30200557c0 0x7f3020057c80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f3038006ef0 tx=0x7f3038015000).stop 2020-11-20T12:06:42.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.830+0000 7f304f35d700 1 -- 172.21.15.189:0/63954792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3048066de0 msgr2=0x7f30481409e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:42.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.830+0000 7f304f35d700 1 --2- 172.21.15.189:0/63954792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3048066de0 0x7f30481409e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f304400eba0 tx=0x7f3044006df0).stop 2020-11-20T12:06:42.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.830+0000 7f304f35d700 1 -- 172.21.15.189:0/63954792 shutdown_connections 2020-11-20T12:06:42.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.830+0000 7f304f35d700 1 --2- 172.21.15.189:0/63954792 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f30200557c0 0x7f3020057c80 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:42.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.830+0000 7f304f35d700 1 --2- 172.21.15.189:0/63954792 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f304813e6c0 0x7f3048149ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:42.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.830+0000 7f304f35d700 1 --2- 172.21.15.189:0/63954792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3048066de0 0x7f30481409e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:42.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.830+0000 7f304f35d700 1 --2- 172.21.15.189:0/63954792 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f30480671c0 0x7f30480a22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:42.839 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.830+0000 7f304f35d700 1 -- 172.21.15.189:0/63954792 >> 172.21.15.189:0/63954792 conn(0x7f3048061f60 msgr2=0x7f304813a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:42.839 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.830+0000 7f304f35d700 1 -- 172.21.15.189:0/63954792 shutdown_connections 2020-11-20T12:06:42.839 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:42.830+0000 7f304f35d700 1 -- 172.21.15.189:0/63954792 wait complete. 2020-11-20T12:06:42.840 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 6 2020-11-20T12:06:42.848 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:06:44.849 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:06:44.982 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a8f3d700 1 Processor -- start 2020-11-20T12:06:44.983 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a8f3d700 1 -- start start 2020-11-20T12:06:44.984 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a8f3d700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd4a40ad000 0x7fd4a413e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:44.984 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a8f3d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd4a40ad3e0 0x7fd4a4066d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:44.985 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a8f3d700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4a4067730 con 0x7fd4a413a6e0 2020-11-20T12:06:44.985 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a8f3d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4a4138190 con 0x7fd4a40ad3e0 2020-11-20T12:06:44.985 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a8f3d700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a413dd40 con 0x7fd4a40ad000 2020-11-20T12:06:44.985 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a2ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd4a40ad3e0 0x7fd4a4066d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:44.986 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a37fe700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd4a40ad000 0x7fd4a413e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:44.986 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a2ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd4a40ad3e0 0x7fd4a4066d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49336/0 (socket says 172.21.15.189:49336) 2020-11-20T12:06:44.986 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a2ffd700 1 -- 172.21.15.189:0/4098591464 learned_addr learned my addr 172.21.15.189:0/4098591464 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:06:44.986 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a37fe700 1 -- 172.21.15.189:0/4098591464 >> v2:172.21.15.189:3300/0 conn(0x7fd4a40ad3e0 msgr2=0x7fd4a4066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:44.987 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a37fe700 1 --2- 172.21.15.189:0/4098591464 >> v2:172.21.15.189:3300/0 conn(0x7fd4a40ad3e0 0x7fd4a4066d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:44.987 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a37fe700 1 -- 172.21.15.189:0/4098591464 >> v1:172.21.15.47:6789/0 conn(0x7fd4a413a6e0 legacy=0x7fd4a413ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:44.987 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a37fe700 1 -- 172.21.15.189:0/4098591464 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a413dfe0 con 0x7fd4a40ad000 2020-11-20T12:06:44.988 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a37fe700 1 --2- 172.21.15.189:0/4098591464 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd4a40ad000 0x7fd4a413e480 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fd48c00b9a0 tx=0x7fd48c006df0).ready entity=mon.2 client_cookie=9838784ae5e38e5d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:44.988 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a27fc700 1 -- 172.21.15.189:0/4098591464 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48c0097b0 con 0x7fd4a40ad000 2020-11-20T12:06:44.988 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a27fc700 1 -- 172.21.15.189:0/4098591464 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd48c009910 con 0x7fd4a40ad000 2020-11-20T12:06:44.988 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a27fc700 1 -- 172.21.15.189:0/4098591464 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48c009c90 con 0x7fd4a40ad000 2020-11-20T12:06:44.990 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a8f3d700 1 -- 172.21.15.189:0/4098591464 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd4a40ad000 msgr2=0x7fd4a413e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:44.991 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a8f3d700 1 --2- 172.21.15.189:0/4098591464 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd4a40ad000 0x7fd4a413e480 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fd48c00b9a0 tx=0x7fd48c006df0).stop 2020-11-20T12:06:44.991 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a8f3d700 1 -- 172.21.15.189:0/4098591464 shutdown_connections 2020-11-20T12:06:44.991 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a8f3d700 1 --2- 172.21.15.189:0/4098591464 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd4a40ad000 0x7fd4a413e480 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:44.992 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a8f3d700 1 --2- 172.21.15.189:0/4098591464 >> v2:172.21.15.189:3300/0 conn(0x7fd4a40ad3e0 0x7fd4a4066d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:44.992 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a8f3d700 1 -- 172.21.15.189:0/4098591464 >> 172.21.15.189:0/4098591464 conn(0x7fd4a4062080 msgr2=0x7fd4a4062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:44.992 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a8f3d700 1 -- 172.21.15.189:0/4098591464 shutdown_connections 2020-11-20T12:06:44.992 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.978+0000 7fd4a8f3d700 1 -- 172.21.15.189:0/4098591464 wait complete. 2020-11-20T12:06:44.992 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.982+0000 7fd4a8f3d700 1 Processor -- start 2020-11-20T12:06:44.993 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.982+0000 7fd4a8f3d700 1 -- start start 2020-11-20T12:06:44.993 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.982+0000 7fd4a8f3d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4a40ad000 0x7fd4a40a5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:44.993 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.982+0000 7fd4a8f3d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4a40ad3e0 0x7fd4a4149ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:44.993 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.982+0000 7fd4a8f3d700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd4a414a3e0 0x7fd4a40b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:44.994 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.982+0000 7fd4a8f3d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a40a89c0 con 0x7fd4a40ad3e0 2020-11-20T12:06:44.994 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.982+0000 7fd4a8f3d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a40af970 con 0x7fd4a40ad000 2020-11-20T12:06:44.994 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.982+0000 7fd4a8f3d700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a40afc10 con 0x7fd4a414a3e0 2020-11-20T12:06:44.994 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.982+0000 7fd4a37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4a40ad000 0x7fd4a40a5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:44.994 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.982+0000 7fd4a2ffd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4a40ad3e0 0x7fd4a4149ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:44.995 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.982+0000 7fd4a37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4a40ad000 0x7fd4a40a5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49342/0 (socket says 172.21.15.189:49342) 2020-11-20T12:06:44.995 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.982+0000 7fd4a3fff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd4a414a3e0 0x7fd4a40b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:44.995 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.982+0000 7fd4a37fe700 1 -- 172.21.15.189:0/885398228 learned_addr learned my addr 172.21.15.189:0/885398228 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:06:44.995 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.982+0000 7fd4a3fff700 1 -- 172.21.15.189:0/885398228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4a40ad000 msgr2=0x7fd4a40a5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:44.996 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.982+0000 7fd4a3fff700 1 --2- 172.21.15.189:0/885398228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4a40ad000 0x7fd4a40a5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:44.996 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.982+0000 7fd4a3fff700 1 -- 172.21.15.189:0/885398228 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4a40ad3e0 msgr2=0x7fd4a4149ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:44.996 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.982+0000 7fd4a3fff700 1 --2- 172.21.15.189:0/885398228 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4a40ad3e0 0x7fd4a4149ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:44.996 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.982+0000 7fd4a3fff700 1 -- 172.21.15.189:0/885398228 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a40b05a0 con 0x7fd4a414a3e0 2020-11-20T12:06:44.996 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.982+0000 7fd4a3fff700 1 --2- 172.21.15.189:0/885398228 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd4a414a3e0 0x7fd4a40b00b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fd4a4066c60 tx=0x7fd494007e90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:44.997 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.982+0000 7fd49b7fe700 1 -- 172.21.15.189:0/885398228 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49400e560 con 0x7fd4a414a3e0 2020-11-20T12:06:44.997 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.982+0000 7fd49b7fe700 1 -- 172.21.15.189:0/885398228 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd49400e6c0 con 0x7fd4a414a3e0 2020-11-20T12:06:44.997 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.982+0000 7fd49b7fe700 1 -- 172.21.15.189:0/885398228 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49400ea40 con 0x7fd4a414a3e0 2020-11-20T12:06:44.997 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.982+0000 7fd4a8f3d700 1 -- 172.21.15.189:0/885398228 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4a40b0850 con 0x7fd4a414a3e0 2020-11-20T12:06:44.998 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.982+0000 7fd4a8f3d700 1 -- 172.21.15.189:0/885398228 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a40aa7e0 con 0x7fd4a414a3e0 2020-11-20T12:06:44.999 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.986+0000 7fd49b7fe700 1 -- 172.21.15.189:0/885398228 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fd494019030 con 0x7fd4a414a3e0 2020-11-20T12:06:44.999 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.986+0000 7fd4a8f3d700 1 -- 172.21.15.189:0/885398228 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4a40afdd0 con 0x7fd4a414a3e0 2020-11-20T12:06:44.999 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.986+0000 7fd49b7fe700 1 --2- 172.21.15.189:0/885398228 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd4740557c0 0x7fd474057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:44.999 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.986+0000 7fd49b7fe700 1 -- 172.21.15.189:0/885398228 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd49406dad0 con 0x7fd4a414a3e0 2020-11-20T12:06:44.999 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.986+0000 7fd4a37fe700 1 --2- 172.21.15.189:0/885398228 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd4740557c0 0x7fd474057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:45.000 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.986+0000 7fd4a37fe700 1 --2- 172.21.15.189:0/885398228 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd4740557c0 0x7fd474057c80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fd48c00c010 tx=0x7fd48c015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:45.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:44.998+0000 7fd49b7fe700 1 -- 172.21.15.189:0/885398228 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd494043a20 con 0x7fd4a414a3e0 2020-11-20T12:06:45.202 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:45.198+0000 7fd4a8f3d700 1 -- 172.21.15.189:0/885398228 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4a40446d0 con 0x7fd4a414a3e0 2020-11-20T12:06:45.203 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:45.198+0000 7fd49b7fe700 1 -- 172.21.15.189:0/885398228 <== mon.2 v2:172.21.15.47:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2134 (secure 0 0 0) 0x7fd494017070 con 0x7fd4a414a3e0 2020-11-20T12:06:45.204 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:06:45.204 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2660417436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2660417436},{"type":"v1","addr":"172.21.15.47:6833","nonce":2660417436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:04:56.025055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.189:6835/1042140684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1042140684},{"type":"v1","addr":"172.21.15.189:6835","nonce":1042140684}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:06:45.205 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:45.202+0000 7fd4a8f3d700 1 -- 172.21.15.189:0/885398228 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd4740557c0 msgr2=0x7fd474057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:45.206 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:45.202+0000 7fd4a8f3d700 1 --2- 172.21.15.189:0/885398228 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd4740557c0 0x7fd474057c80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fd48c00c010 tx=0x7fd48c015000).stop 2020-11-20T12:06:45.206 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:45.202+0000 7fd4a8f3d700 1 -- 172.21.15.189:0/885398228 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd4a414a3e0 msgr2=0x7fd4a40b00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:45.206 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:45.202+0000 7fd4a8f3d700 1 --2- 172.21.15.189:0/885398228 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd4a414a3e0 0x7fd4a40b00b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fd4a4066c60 tx=0x7fd494007e90).stop 2020-11-20T12:06:45.206 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:45.202+0000 7fd4a8f3d700 1 -- 172.21.15.189:0/885398228 shutdown_connections 2020-11-20T12:06:45.207 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:45.202+0000 7fd4a8f3d700 1 --2- 172.21.15.189:0/885398228 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd4a414a3e0 0x7fd4a40b00b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:45.207 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:45.202+0000 7fd4a8f3d700 1 --2- 172.21.15.189:0/885398228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4a40ad000 0x7fd4a40a5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:45.207 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:45.202+0000 7fd4a8f3d700 1 --2- 172.21.15.189:0/885398228 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd4740557c0 0x7fd474057c80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:45.207 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:45.202+0000 7fd4a8f3d700 1 --2- 172.21.15.189:0/885398228 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4a40ad3e0 0x7fd4a4149ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:45.208 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:45.202+0000 7fd4a8f3d700 1 -- 172.21.15.189:0/885398228 >> 172.21.15.189:0/885398228 conn(0x7fd4a4062080 msgr2=0x7fd4a40ad7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:45.208 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:45.202+0000 7fd4a8f3d700 1 -- 172.21.15.189:0/885398228 shutdown_connections 2020-11-20T12:06:45.208 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:45.202+0000 7fd4a8f3d700 1 -- 172.21.15.189:0/885398228 wait complete. 2020-11-20T12:06:45.208 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 6 2020-11-20T12:06:45.218 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:06:47.219 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:06:47.350 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.346+0000 7f1f2f466700 1 Processor -- start 2020-11-20T12:06:47.351 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.346+0000 7f1f2f466700 1 -- start start 2020-11-20T12:06:47.353 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.346+0000 7f1f2f466700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1f280a9290 0x7f1f28134100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:47.353 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.346+0000 7f1f2f466700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f280663e0 con 0x7f1f280a9e20 2020-11-20T12:06:47.353 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.346+0000 7f1f2f466700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f28067b80 con 0x7f1f28068020 2020-11-20T12:06:47.353 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.346+0000 7f1f2f466700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f280a9970 con 0x7f1f280a9290 2020-11-20T12:06:47.354 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.346+0000 7f1f2ca01700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1f28068020 0x7f1f281339f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50972/0 (socket says 172.21.15.189:50972) 2020-11-20T12:06:47.354 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.346+0000 7f1f2ca01700 1 -- 172.21.15.189:0/3664156794 learned_addr learned my addr 172.21.15.189:0/3664156794 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:06:47.354 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.346+0000 7f1f2da03700 1 --2- 172.21.15.189:0/3664156794 >> v2:172.21.15.47:3300/0 conn(0x7f1f280a9290 0x7f1f28134100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:47.354 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.346+0000 7f1f17fff700 1 -- 172.21.15.189:0/3664156794 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228079864 0 0) 0x7f1f28067b80 con 0x7f1f28068020 2020-11-20T12:06:47.354 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.346+0000 7f1f17fff700 1 -- 172.21.15.189:0/3664156794 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f10003240 con 0x7f1f28068020 2020-11-20T12:06:47.356 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.350+0000 7f1f2da03700 1 -- 172.21.15.189:0/3664156794 >> v1:172.21.15.189:6789/0 conn(0x7f1f28068020 legacy=0x7f1f281339f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:47.356 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.350+0000 7f1f2da03700 1 -- 172.21.15.189:0/3664156794 >> v1:172.21.15.47:6789/0 conn(0x7f1f280a9e20 legacy=0x7f1f280a83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:47.356 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.350+0000 7f1f2da03700 1 -- 172.21.15.189:0/3664156794 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f28066680 con 0x7f1f280a9290 2020-11-20T12:06:47.357 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.350+0000 7f1f2da03700 1 --2- 172.21.15.189:0/3664156794 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f280a9290 0x7f1f28134100 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f1f2000b9a0 tx=0x7f1f20006df0).ready entity=mon.0 client_cookie=9eaf037aed6160de server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:47.357 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.350+0000 7f1f17fff700 1 -- 172.21.15.189:0/3664156794 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f200097b0 con 0x7f1f280a9290 2020-11-20T12:06:47.357 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.350+0000 7f1f17fff700 1 -- 172.21.15.189:0/3664156794 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f20009910 con 0x7f1f280a9290 2020-11-20T12:06:47.357 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.350+0000 7f1f17fff700 1 -- 172.21.15.189:0/3664156794 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f20009c90 con 0x7f1f280a9290 2020-11-20T12:06:47.357 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.350+0000 7f1f2f466700 1 -- 172.21.15.189:0/3664156794 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f280a9290 msgr2=0x7f1f28134100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:47.358 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.350+0000 7f1f2f466700 1 --2- 172.21.15.189:0/3664156794 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f280a9290 0x7f1f28134100 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f1f2000b9a0 tx=0x7f1f20006df0).stop 2020-11-20T12:06:47.358 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.350+0000 7f1f2f466700 1 -- 172.21.15.189:0/3664156794 shutdown_connections 2020-11-20T12:06:47.358 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.350+0000 7f1f2f466700 1 --2- 172.21.15.189:0/3664156794 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f280a9290 0x7f1f28134100 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:47.358 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.350+0000 7f1f2f466700 1 -- 172.21.15.189:0/3664156794 >> 172.21.15.189:0/3664156794 conn(0x7f1f280627f0 msgr2=0x7f1f28065870 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:47.358 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.350+0000 7f1f2f466700 1 -- 172.21.15.189:0/3664156794 shutdown_connections 2020-11-20T12:06:47.359 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.350+0000 7f1f2f466700 1 -- 172.21.15.189:0/3664156794 wait complete. 2020-11-20T12:06:47.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.354+0000 7f1f2f466700 1 Processor -- start 2020-11-20T12:06:47.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.354+0000 7f1f2f466700 1 -- start start 2020-11-20T12:06:47.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.354+0000 7f1f2f466700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1f28068020 0x7f1f280ac620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:47.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.354+0000 7f1f2f466700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f280a9290 0x7f1f2813a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:47.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.354+0000 7f1f2f466700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f280a9e20 0x7f1f2812e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:47.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.354+0000 7f1f2f466700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f281465a0 con 0x7f1f280a9290 2020-11-20T12:06:47.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.354+0000 7f1f2f466700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f280accc0 con 0x7f1f280a9e20 2020-11-20T12:06:47.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.354+0000 7f1f2f466700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f28146790 con 0x7f1f28068020 2020-11-20T12:06:47.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.354+0000 7f1f2da03700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f280a9e20 0x7f1f2812e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:47.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.354+0000 7f1f2da03700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f280a9e20 0x7f1f2812e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49354/0 (socket says 172.21.15.189:49354) 2020-11-20T12:06:47.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.354+0000 7f1f2da03700 1 -- 172.21.15.189:0/1270948367 learned_addr learned my addr 172.21.15.189:0/1270948367 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:06:47.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.354+0000 7f1f2d202700 1 --2- 172.21.15.189:0/1270948367 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1f28068020 0x7f1f280ac620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:47.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.354+0000 7f1f2ca01700 1 --2- 172.21.15.189:0/1270948367 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f280a9290 0x7f1f2813a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:47.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.354+0000 7f1f2da03700 1 -- 172.21.15.189:0/1270948367 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1f28068020 msgr2=0x7f1f280ac620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:47.364 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.354+0000 7f1f2da03700 1 --2- 172.21.15.189:0/1270948367 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1f28068020 0x7f1f280ac620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:47.364 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.354+0000 7f1f2da03700 1 -- 172.21.15.189:0/1270948367 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f280a9290 msgr2=0x7f1f2813a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:47.364 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.354+0000 7f1f2da03700 1 --2- 172.21.15.189:0/1270948367 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f280a9290 0x7f1f2813a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:47.364 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.354+0000 7f1f2da03700 1 -- 172.21.15.189:0/1270948367 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f28137450 con 0x7f1f280a9e20 2020-11-20T12:06:47.364 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.354+0000 7f1f2ca01700 1 --2- 172.21.15.189:0/1270948367 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f280a9290 0x7f1f2813a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:06:47.365 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.354+0000 7f1f2da03700 1 --2- 172.21.15.189:0/1270948367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f280a9e20 0x7f1f2812e650 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f1f28061a10 tx=0x7f1f20015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:47.365 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.354+0000 7f1f157fa700 1 -- 172.21.15.189:0/1270948367 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f20009e80 con 0x7f1f280a9e20 2020-11-20T12:06:47.365 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.354+0000 7f1f157fa700 1 -- 172.21.15.189:0/1270948367 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2001d5d0 con 0x7f1f280a9e20 2020-11-20T12:06:47.365 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.354+0000 7f1f2f466700 1 -- 172.21.15.189:0/1270948367 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f28137640 con 0x7f1f280a9e20 2020-11-20T12:06:47.366 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.354+0000 7f1f157fa700 1 -- 172.21.15.189:0/1270948367 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2001d950 con 0x7f1f280a9e20 2020-11-20T12:06:47.366 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.354+0000 7f1f2f466700 1 -- 172.21.15.189:0/1270948367 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f2812ed60 con 0x7f1f280a9e20 2020-11-20T12:06:47.367 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.358+0000 7f1f2f466700 1 -- 172.21.15.189:0/1270948367 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f280acb10 con 0x7f1f280a9e20 2020-11-20T12:06:47.367 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.358+0000 7f1f157fa700 1 -- 172.21.15.189:0/1270948367 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f1f20025030 con 0x7f1f280a9e20 2020-11-20T12:06:47.367 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.358+0000 7f1f157fa700 1 --2- 172.21.15.189:0/1270948367 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1f000557c0 0x7f1f00057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:47.368 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.358+0000 7f1f157fa700 1 -- 172.21.15.189:0/1270948367 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1f20070100 con 0x7f1f280a9e20 2020-11-20T12:06:47.368 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.358+0000 7f1f2d202700 1 --2- 172.21.15.189:0/1270948367 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1f000557c0 0x7f1f00057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:47.368 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.358+0000 7f1f2d202700 1 --2- 172.21.15.189:0/1270948367 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1f000557c0 0x7f1f00057c80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f1f28061690 tx=0x7f1f24009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:47.369 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.366+0000 7f1f157fa700 1 -- 172.21.15.189:0/1270948367 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1f20046050 con 0x7f1f280a9e20 2020-11-20T12:06:47.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.570+0000 7f1f2f466700 1 -- 172.21.15.189:0/1270948367 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f28146950 con 0x7f1f280a9e20 2020-11-20T12:06:47.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.574+0000 7f1f157fa700 1 -- 172.21.15.189:0/1270948367 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2134 (secure 0 0 0) 0x7f1f20023070 con 0x7f1f280a9e20 2020-11-20T12:06:47.578 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:06:47.578 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2660417436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2660417436},{"type":"v1","addr":"172.21.15.47:6833","nonce":2660417436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:04:56.025055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.189:6835/1042140684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1042140684},{"type":"v1","addr":"172.21.15.189:6835","nonce":1042140684}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:06:47.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.574+0000 7f1f2f466700 1 -- 172.21.15.189:0/1270948367 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1f000557c0 msgr2=0x7f1f00057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:47.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.574+0000 7f1f2f466700 1 --2- 172.21.15.189:0/1270948367 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1f000557c0 0x7f1f00057c80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f1f28061690 tx=0x7f1f24009040).stop 2020-11-20T12:06:47.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.574+0000 7f1f2f466700 1 -- 172.21.15.189:0/1270948367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f280a9e20 msgr2=0x7f1f2812e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:47.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.574+0000 7f1f2f466700 1 --2- 172.21.15.189:0/1270948367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f280a9e20 0x7f1f2812e650 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f1f28061a10 tx=0x7f1f20015000).stop 2020-11-20T12:06:47.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.574+0000 7f1f2f466700 1 -- 172.21.15.189:0/1270948367 shutdown_connections 2020-11-20T12:06:47.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.574+0000 7f1f2f466700 1 --2- 172.21.15.189:0/1270948367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f280a9e20 0x7f1f2812e650 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:47.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.574+0000 7f1f2f466700 1 --2- 172.21.15.189:0/1270948367 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1f28068020 0x7f1f280ac620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:47.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.574+0000 7f1f2f466700 1 --2- 172.21.15.189:0/1270948367 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1f000557c0 0x7f1f00057c80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:47.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.574+0000 7f1f2f466700 1 --2- 172.21.15.189:0/1270948367 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f280a9290 0x7f1f2813a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:47.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.574+0000 7f1f2f466700 1 -- 172.21.15.189:0/1270948367 >> 172.21.15.189:0/1270948367 conn(0x7f1f280627f0 msgr2=0x7f1f28131570 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:47.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.574+0000 7f1f2f466700 1 -- 172.21.15.189:0/1270948367 shutdown_connections 2020-11-20T12:06:47.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:47.574+0000 7f1f2f466700 1 -- 172.21.15.189:0/1270948367 wait complete. 2020-11-20T12:06:47.583 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 6 2020-11-20T12:06:47.593 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:06:49.593 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:06:49.731 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.726+0000 7f6daf59e700 1 Processor -- start 2020-11-20T12:06:49.731 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.726+0000 7f6daf59e700 1 -- start start 2020-11-20T12:06:49.733 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.726+0000 7f6daf59e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6db0066ed0 0x7f6db013b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:49.733 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.726+0000 7f6daf59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db0066800 con 0x7f6db00a8840 2020-11-20T12:06:49.733 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.726+0000 7f6daf59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db0066aa0 con 0x7f6db013ba60 2020-11-20T12:06:49.733 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.726+0000 7f6daf59e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db0067450 con 0x7f6db0066ed0 2020-11-20T12:06:49.734 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.726+0000 7f6daed9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6db013ba60 0x7f6db01495e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:50988/0 (socket says 172.21.15.189:50988) 2020-11-20T12:06:49.734 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.726+0000 7f6daed9d700 1 -- 172.21.15.189:0/3481585549 learned_addr learned my addr 172.21.15.189:0/3481585549 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:06:49.734 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.726+0000 7f6dadd9b700 1 --2- 172.21.15.189:0/3481585549 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6db0066ed0 0x7f6db013b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:49.734 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.726+0000 7f6dad59a700 1 -- 172.21.15.189:0/3481585549 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346398039 0 0) 0x7f6db0066aa0 con 0x7f6db013ba60 2020-11-20T12:06:49.734 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.726+0000 7f6dad59a700 1 -- 172.21.15.189:0/3481585549 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d94003240 con 0x7f6db013ba60 2020-11-20T12:06:49.735 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.726+0000 7f6dadd9b700 1 -- 172.21.15.189:0/3481585549 >> v1:172.21.15.189:6789/0 conn(0x7f6db013ba60 legacy=0x7f6db01495e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:49.735 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.726+0000 7f6dadd9b700 1 -- 172.21.15.189:0/3481585549 >> v1:172.21.15.47:6789/0 conn(0x7f6db00a8840 legacy=0x7f6db013eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:49.735 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.726+0000 7f6dad59a700 1 -- 172.21.15.189:0/3481585549 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2358855390 0 0) 0x7f6db0066800 con 0x7f6db00a8840 2020-11-20T12:06:49.735 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.726+0000 7f6dadd9b700 1 -- 172.21.15.189:0/3481585549 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db00676f0 con 0x7f6db0066ed0 2020-11-20T12:06:49.735 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6dadd9b700 1 --2- 172.21.15.189:0/3481585549 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6db0066ed0 0x7f6db013b570 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f6da400b9d0 tx=0x7f6da4006df0).ready entity=mon.2 client_cookie=eaf6c45331cefc24 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:49.738 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6dad59a700 1 -- 172.21.15.189:0/3481585549 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da40097b0 con 0x7f6db0066ed0 2020-11-20T12:06:49.738 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6dad59a700 1 -- 172.21.15.189:0/3481585549 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da4009910 con 0x7f6db0066ed0 2020-11-20T12:06:49.738 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6dad59a700 1 -- 172.21.15.189:0/3481585549 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da4009c90 con 0x7f6db0066ed0 2020-11-20T12:06:49.738 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6daf59e700 1 -- 172.21.15.189:0/3481585549 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6db0066ed0 msgr2=0x7f6db013b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:49.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6daf59e700 1 --2- 172.21.15.189:0/3481585549 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6db0066ed0 0x7f6db013b570 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f6da400b9d0 tx=0x7f6da4006df0).stop 2020-11-20T12:06:49.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6daf59e700 1 -- 172.21.15.189:0/3481585549 shutdown_connections 2020-11-20T12:06:49.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6daf59e700 1 --2- 172.21.15.189:0/3481585549 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6db0066ed0 0x7f6db013b570 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:49.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6daf59e700 1 -- 172.21.15.189:0/3481585549 >> 172.21.15.189:0/3481585549 conn(0x7f6db0060170 msgr2=0x7f6db0060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:49.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6daf59e700 1 -- 172.21.15.189:0/3481585549 shutdown_connections 2020-11-20T12:06:49.740 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6daf59e700 1 -- 172.21.15.189:0/3481585549 wait complete. 2020-11-20T12:06:49.740 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6daf59e700 1 Processor -- start 2020-11-20T12:06:49.740 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6daf59e700 1 -- start start 2020-11-20T12:06:49.740 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6daf59e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6db0066ed0 0x7f6db013cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:49.740 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6daf59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6db00a8840 0x7f6db00a3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:49.741 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6daf59e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6db013ba60 0x7f6db014de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:49.741 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6daf59e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db00a3840 con 0x7f6db0066ed0 2020-11-20T12:06:49.741 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6daf59e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db014e310 con 0x7f6db00a8840 2020-11-20T12:06:49.741 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6daf59e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db014e530 con 0x7f6db013ba60 2020-11-20T12:06:49.741 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6dadd9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6db00a8840 0x7f6db00a3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:49.742 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6dadd9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6db00a8840 0x7f6db00a3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49368/0 (socket says 172.21.15.189:49368) 2020-11-20T12:06:49.742 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6dadd9b700 1 -- 172.21.15.189:0/2030796735 learned_addr learned my addr 172.21.15.189:0/2030796735 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:06:49.742 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6dae59c700 1 --2- 172.21.15.189:0/2030796735 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6db0066ed0 0x7f6db013cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:49.742 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6daed9d700 1 --2- 172.21.15.189:0/2030796735 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6db013ba60 0x7f6db014de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:49.742 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6dadd9b700 1 -- 172.21.15.189:0/2030796735 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6db013ba60 msgr2=0x7f6db014de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:49.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6dadd9b700 1 --2- 172.21.15.189:0/2030796735 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6db013ba60 0x7f6db014de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:49.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6dadd9b700 1 -- 172.21.15.189:0/2030796735 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6db0066ed0 msgr2=0x7f6db013cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:49.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6dadd9b700 1 --2- 172.21.15.189:0/2030796735 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6db0066ed0 0x7f6db013cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:49.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6dadd9b700 1 -- 172.21.15.189:0/2030796735 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db014e7d0 con 0x7f6db00a8840 2020-11-20T12:06:49.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.730+0000 7f6dadd9b700 1 --2- 172.21.15.189:0/2030796735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6db00a8840 0x7f6db00a3350 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f6da4006e30 tx=0x7f6da4015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:49.744 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.734+0000 7f6d9e7fc700 1 -- 172.21.15.189:0/2030796735 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da4011d50 con 0x7f6db00a8840 2020-11-20T12:06:49.744 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.734+0000 7f6d9e7fc700 1 -- 172.21.15.189:0/2030796735 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da401e460 con 0x7f6db00a8840 2020-11-20T12:06:49.744 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.734+0000 7f6daf59e700 1 -- 172.21.15.189:0/2030796735 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db014ea70 con 0x7f6db00a8840 2020-11-20T12:06:49.744 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.734+0000 7f6d9e7fc700 1 -- 172.21.15.189:0/2030796735 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da401e7e0 con 0x7f6db00a8840 2020-11-20T12:06:49.744 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.734+0000 7f6daf59e700 1 -- 172.21.15.189:0/2030796735 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db014f560 con 0x7f6db00a8840 2020-11-20T12:06:49.745 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.734+0000 7f6daf59e700 1 -- 172.21.15.189:0/2030796735 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6db0066000 con 0x7f6db00a8840 2020-11-20T12:06:49.746 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.734+0000 7f6d9e7fc700 1 -- 172.21.15.189:0/2030796735 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f6da4025030 con 0x7f6db00a8840 2020-11-20T12:06:49.746 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.734+0000 7f6d9e7fc700 1 --2- 172.21.15.189:0/2030796735 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6d840557c0 0x7f6d84057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:49.746 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.734+0000 7f6dae59c700 1 --2- 172.21.15.189:0/2030796735 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6d840557c0 0x7f6d84057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:49.746 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.734+0000 7f6d9e7fc700 1 -- 172.21.15.189:0/2030796735 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6da406fc60 con 0x7f6db00a8840 2020-11-20T12:06:49.747 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.734+0000 7f6dae59c700 1 --2- 172.21.15.189:0/2030796735 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6d840557c0 0x7f6d84057c80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f6da0007fd0 tx=0x7f6da0007a00).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:49.747 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.742+0000 7f6d9e7fc700 1 -- 172.21.15.189:0/2030796735 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6da4045bb0 con 0x7f6db00a8840 2020-11-20T12:06:49.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.938+0000 7f6daf59e700 1 -- 172.21.15.189:0/2030796735 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6db00a3b90 con 0x7f6db00a8840 2020-11-20T12:06:49.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.942+0000 7f6d9e7fc700 1 -- 172.21.15.189:0/2030796735 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2134 (secure 0 0 0) 0x7f6da4023070 con 0x7f6db00a8840 2020-11-20T12:06:49.946 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:06:49.946 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2660417436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2660417436},{"type":"v1","addr":"172.21.15.47:6833","nonce":2660417436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:04:56.025055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.189:6835/1042140684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1042140684},{"type":"v1","addr":"172.21.15.189:6835","nonce":1042140684}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:06:49.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.942+0000 7f6daf59e700 1 -- 172.21.15.189:0/2030796735 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6d840557c0 msgr2=0x7f6d84057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:49.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.942+0000 7f6daf59e700 1 --2- 172.21.15.189:0/2030796735 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6d840557c0 0x7f6d84057c80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f6da0007fd0 tx=0x7f6da0007a00).stop 2020-11-20T12:06:49.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.942+0000 7f6daf59e700 1 -- 172.21.15.189:0/2030796735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6db00a8840 msgr2=0x7f6db00a3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:49.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.942+0000 7f6daf59e700 1 --2- 172.21.15.189:0/2030796735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6db00a8840 0x7f6db00a3350 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f6da4006e30 tx=0x7f6da4015000).stop 2020-11-20T12:06:49.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.942+0000 7f6daf59e700 1 -- 172.21.15.189:0/2030796735 shutdown_connections 2020-11-20T12:06:49.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.942+0000 7f6daf59e700 1 --2- 172.21.15.189:0/2030796735 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6d840557c0 0x7f6d84057c80 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:49.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.942+0000 7f6daf59e700 1 --2- 172.21.15.189:0/2030796735 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6db0066ed0 0x7f6db013cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:49.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.942+0000 7f6daf59e700 1 --2- 172.21.15.189:0/2030796735 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6db013ba60 0x7f6db014de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:49.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.942+0000 7f6daf59e700 1 --2- 172.21.15.189:0/2030796735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6db00a8840 0x7f6db00a3350 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:49.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.942+0000 7f6daf59e700 1 -- 172.21.15.189:0/2030796735 >> 172.21.15.189:0/2030796735 conn(0x7f6db0060170 msgr2=0x7f6db013a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:49.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.942+0000 7f6daf59e700 1 -- 172.21.15.189:0/2030796735 shutdown_connections 2020-11-20T12:06:49.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:49.942+0000 7f6daf59e700 1 -- 172.21.15.189:0/2030796735 wait complete. 2020-11-20T12:06:49.951 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 6 2020-11-20T12:06:49.967 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:06:51.932 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:06:51.935 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:06:51.968 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:06:52.110 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.106+0000 7f1638f69700 1 Processor -- start 2020-11-20T12:06:52.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.106+0000 7f1638f69700 1 -- start start 2020-11-20T12:06:52.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.106+0000 7f1638f69700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1634060310 0x7f16340a1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:52.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.106+0000 7f1638f69700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f16340a2440 0x7f16340a7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:52.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.106+0000 7f1638f69700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f16340a8230 0x7f1634135110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:52.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.106+0000 7f1638f69700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16340618b0 con 0x7f16340a2440 2020-11-20T12:06:52.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.106+0000 7f1638f69700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16340a8fa0 con 0x7f16340a8230 2020-11-20T12:06:52.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.106+0000 7f1638f69700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16340a9240 con 0x7f1634060310 2020-11-20T12:06:52.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.106+0000 7f1633fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f16340a8230 0x7f1634135110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:52.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.106+0000 7f16337fe700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1634060310 0x7f16340a1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:52.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.106+0000 7f1633fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f16340a8230 0x7f1634135110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49382/0 (socket says 172.21.15.189:49382) 2020-11-20T12:06:52.114 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.106+0000 7f16337fe700 1 --2- 172.21.15.189:0/2927428803 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1634060310 0x7f16340a1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.189:42412/0 (socket says 172.21.15.189:42412) 2020-11-20T12:06:52.114 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.106+0000 7f1633fff700 1 -- 172.21.15.189:0/2927428803 learned_addr learned my addr 172.21.15.189:0/2927428803 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:06:52.114 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.106+0000 7f1632ffd700 1 --2- 172.21.15.189:0/2927428803 >> v2:172.21.15.47:3300/0 conn(0x7f16340a2440 0x7f16340a7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:52.116 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.106+0000 7f1633fff700 1 -- 172.21.15.189:0/2927428803 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1634060310 msgr2=0x7f16340a1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:52.116 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.106+0000 7f1633fff700 1 --2- 172.21.15.189:0/2927428803 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1634060310 0x7f16340a1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:52.116 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.106+0000 7f1633fff700 1 -- 172.21.15.189:0/2927428803 >> v2:172.21.15.47:3300/0 conn(0x7f16340a2440 msgr2=0x7f16340a7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:52.116 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.106+0000 7f1633fff700 1 --2- 172.21.15.189:0/2927428803 >> v2:172.21.15.47:3300/0 conn(0x7f16340a2440 0x7f16340a7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:52.117 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.106+0000 7f1633fff700 1 -- 172.21.15.189:0/2927428803 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16340a94e0 con 0x7f16340a8230 2020-11-20T12:06:52.117 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.106+0000 7f1633fff700 1 --2- 172.21.15.189:0/2927428803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16340a8230 0x7f1634135110 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f16340a1cc0 tx=0x7f162c009290).ready entity=mon.1 client_cookie=7aaf36429f6bbe75 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:52.117 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.106+0000 7f16327fc700 1 -- 172.21.15.189:0/2927428803 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c00bb10 con 0x7f16340a8230 2020-11-20T12:06:52.117 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.106+0000 7f16327fc700 1 -- 172.21.15.189:0/2927428803 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162c00bc70 con 0x7f16340a8230 2020-11-20T12:06:52.117 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.110+0000 7f16327fc700 1 -- 172.21.15.189:0/2927428803 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c011400 con 0x7f16340a8230 2020-11-20T12:06:52.118 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.110+0000 7f1638f69700 1 -- 172.21.15.189:0/2927428803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16340a8230 msgr2=0x7f1634135110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:52.118 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.110+0000 7f1638f69700 1 --2- 172.21.15.189:0/2927428803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16340a8230 0x7f1634135110 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f16340a1cc0 tx=0x7f162c009290).stop 2020-11-20T12:06:52.118 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.110+0000 7f1638f69700 1 -- 172.21.15.189:0/2927428803 shutdown_connections 2020-11-20T12:06:52.118 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.110+0000 7f1638f69700 1 --2- 172.21.15.189:0/2927428803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16340a8230 0x7f1634135110 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:52.118 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.110+0000 7f1638f69700 1 --2- 172.21.15.189:0/2927428803 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1634060310 0x7f16340a1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:52.119 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.110+0000 7f1638f69700 1 --2- 172.21.15.189:0/2927428803 >> v2:172.21.15.47:3300/0 conn(0x7f16340a2440 0x7f16340a7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:52.119 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.110+0000 7f1638f69700 1 -- 172.21.15.189:0/2927428803 >> 172.21.15.189:0/2927428803 conn(0x7f1634063210 msgr2=0x7f1634060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:52.119 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.110+0000 7f1638f69700 1 -- 172.21.15.189:0/2927428803 shutdown_connections 2020-11-20T12:06:52.119 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.110+0000 7f1638f69700 1 -- 172.21.15.189:0/2927428803 wait complete. 2020-11-20T12:06:52.122 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.110+0000 7f1638f69700 1 Processor -- start 2020-11-20T12:06:52.122 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.110+0000 7f1638f69700 1 -- start start 2020-11-20T12:06:52.122 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.110+0000 7f1638f69700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1634060310 0x7f1634138de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:52.123 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.110+0000 7f1638f69700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f16340a2440 0x7f16341381e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:52.123 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.110+0000 7f1638f69700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16340a8230 0x7f1634130500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:52.123 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.110+0000 7f1638f69700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1634136120 con 0x7f1634060310 2020-11-20T12:06:52.123 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.110+0000 7f1638f69700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1634136350 con 0x7f16340a8230 2020-11-20T12:06:52.123 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.110+0000 7f1638f69700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16341365f0 con 0x7f16340a2440 2020-11-20T12:06:52.124 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.110+0000 7f1633fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16340a8230 0x7f1634130500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:52.124 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.110+0000 7f1633fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16340a8230 0x7f1634130500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49388/0 (socket says 172.21.15.189:49388) 2020-11-20T12:06:52.124 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.110+0000 7f1633fff700 1 -- 172.21.15.189:0/1250215400 learned_addr learned my addr 172.21.15.189:0/1250215400 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:06:52.124 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.110+0000 7f16337fe700 1 --2- 172.21.15.189:0/1250215400 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1634060310 0x7f1634138de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:52.124 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.110+0000 7f1632ffd700 1 --2- 172.21.15.189:0/1250215400 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f16340a2440 0x7f16341381e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:52.125 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.114+0000 7f1633fff700 1 -- 172.21.15.189:0/1250215400 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f16340a2440 msgr2=0x7f16341381e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:52.125 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.114+0000 7f1633fff700 1 --2- 172.21.15.189:0/1250215400 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f16340a2440 0x7f16341381e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:52.125 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.114+0000 7f1633fff700 1 -- 172.21.15.189:0/1250215400 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1634060310 msgr2=0x7f1634138de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:52.125 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.114+0000 7f1633fff700 1 --2- 172.21.15.189:0/1250215400 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1634060310 0x7f1634138de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:52.125 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.114+0000 7f1633fff700 1 -- 172.21.15.189:0/1250215400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16341386d0 con 0x7f16340a8230 2020-11-20T12:06:52.126 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.114+0000 7f1633fff700 1 --2- 172.21.15.189:0/1250215400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16340a8230 0x7f1634130500 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f162c0092d0 tx=0x7f162c016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:52.126 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.114+0000 7f16137fe700 1 -- 172.21.15.189:0/1250215400 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c01b5e0 con 0x7f16340a8230 2020-11-20T12:06:52.126 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.114+0000 7f16137fe700 1 -- 172.21.15.189:0/1250215400 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162c01b740 con 0x7f16340a8230 2020-11-20T12:06:52.126 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.114+0000 7f1638f69700 1 -- 172.21.15.189:0/1250215400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1634068180 con 0x7f16340a8230 2020-11-20T12:06:52.126 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.114+0000 7f16137fe700 1 -- 172.21.15.189:0/1250215400 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c01bb00 con 0x7f16340a8230 2020-11-20T12:06:52.127 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.114+0000 7f1638f69700 1 -- 172.21.15.189:0/1250215400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16340a7310 con 0x7f16340a8230 2020-11-20T12:06:52.127 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.114+0000 7f1638f69700 1 -- 172.21.15.189:0/1250215400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16340aab50 con 0x7f16340a8230 2020-11-20T12:06:52.127 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.114+0000 7f16137fe700 1 -- 172.21.15.189:0/1250215400 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f162c017070 con 0x7f16340a8230 2020-11-20T12:06:52.127 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.114+0000 7f16137fe700 1 --2- 172.21.15.189:0/1250215400 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f160c0557c0 0x7f160c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:52.128 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.114+0000 7f16337fe700 1 --2- 172.21.15.189:0/1250215400 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f160c0557c0 0x7f160c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:52.128 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.114+0000 7f16137fe700 1 -- 172.21.15.189:0/1250215400 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f162c070550 con 0x7f16340a8230 2020-11-20T12:06:52.128 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.114+0000 7f16337fe700 1 --2- 172.21.15.189:0/1250215400 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f160c0557c0 0x7f160c057c80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f1624006fd0 tx=0x7f1624006df0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:52.129 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.122+0000 7f16137fe700 1 -- 172.21.15.189:0/1250215400 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f16340aab50 con 0x7f16340a8230 2020-11-20T12:06:52.323 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.318+0000 7f1638f69700 1 -- 172.21.15.189:0/1250215400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16340aab50 con 0x7f16340a8230 2020-11-20T12:06:52.323 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.318+0000 7f16137fe700 1 -- 172.21.15.189:0/1250215400 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+2134 (secure 0 0 0) 0x7f16340aab50 con 0x7f16340a8230 2020-11-20T12:06:52.324 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:06:52.325 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/2660417436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2660417436},{"type":"v1","addr":"172.21.15.47:6833","nonce":2660417436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:04:56.025055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.189:6835/1042140684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1042140684},{"type":"v1","addr":"172.21.15.189:6835","nonce":1042140684}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:06:52.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.322+0000 7f1638f69700 1 -- 172.21.15.189:0/1250215400 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f160c0557c0 msgr2=0x7f160c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:52.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.322+0000 7f1638f69700 1 --2- 172.21.15.189:0/1250215400 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f160c0557c0 0x7f160c057c80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f1624006fd0 tx=0x7f1624006df0).stop 2020-11-20T12:06:52.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.322+0000 7f1638f69700 1 -- 172.21.15.189:0/1250215400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16340a8230 msgr2=0x7f1634130500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:52.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.322+0000 7f1638f69700 1 --2- 172.21.15.189:0/1250215400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16340a8230 0x7f1634130500 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f162c0092d0 tx=0x7f162c016000).stop 2020-11-20T12:06:52.328 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.322+0000 7f1638f69700 1 -- 172.21.15.189:0/1250215400 shutdown_connections 2020-11-20T12:06:52.328 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.322+0000 7f1638f69700 1 --2- 172.21.15.189:0/1250215400 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f160c0557c0 0x7f160c057c80 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:52.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.322+0000 7f1638f69700 1 --2- 172.21.15.189:0/1250215400 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1634060310 0x7f1634138de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:52.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.322+0000 7f1638f69700 1 --2- 172.21.15.189:0/1250215400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16340a8230 0x7f1634130500 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:52.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.322+0000 7f1638f69700 1 --2- 172.21.15.189:0/1250215400 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f16340a2440 0x7f16341381e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:52.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.322+0000 7f1638f69700 1 -- 172.21.15.189:0/1250215400 >> 172.21.15.189:0/1250215400 conn(0x7f1634063210 msgr2=0x7f16340ab760 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:52.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.322+0000 7f1638f69700 1 -- 172.21.15.189:0/1250215400 shutdown_connections 2020-11-20T12:06:52.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:52.322+0000 7f1638f69700 1 -- 172.21.15.189:0/1250215400 wait complete. 2020-11-20T12:06:52.331 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 6 2020-11-20T12:06:52.338 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:06:54.340 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:06:54.475 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.470+0000 7f3ea8f30700 1 Processor -- start 2020-11-20T12:06:54.475 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.470+0000 7f3ea8f30700 1 -- start start 2020-11-20T12:06:54.477 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.470+0000 7f3ea8f30700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3ea4066ed0 0x7f3ea413b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:54.477 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.470+0000 7f3ea8f30700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea4066800 con 0x7f3ea413ba60 2020-11-20T12:06:54.477 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.470+0000 7f3ea8f30700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea4066aa0 con 0x7f3ea40a8840 2020-11-20T12:06:54.477 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.470+0000 7f3ea8f30700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ea4067450 con 0x7f3ea4066ed0 2020-11-20T12:06:54.478 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.470+0000 7f3ea37fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3ea40a8840 0x7f3ea413eb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51018/0 (socket says 172.21.15.189:51018) 2020-11-20T12:06:54.478 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.470+0000 7f3ea2ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3ea4066ed0 0x7f3ea413b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:54.478 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.470+0000 7f3ea37fe700 1 -- 172.21.15.189:0/1636032947 learned_addr learned my addr 172.21.15.189:0/1636032947 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:06:54.478 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.470+0000 7f3ea27fc700 1 -- 172.21.15.189:0/1636032947 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2188716581 0 0) 0x7f3ea4066aa0 con 0x7f3ea40a8840 2020-11-20T12:06:54.479 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.470+0000 7f3ea27fc700 1 -- 172.21.15.189:0/1636032947 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e90003240 con 0x7f3ea40a8840 2020-11-20T12:06:54.479 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.470+0000 7f3ea2ffd700 1 -- 172.21.15.189:0/1636032947 >> v1:172.21.15.189:6789/0 conn(0x7f3ea40a8840 legacy=0x7f3ea413eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:54.482 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.470+0000 7f3ea2ffd700 1 -- 172.21.15.189:0/1636032947 >> v1:172.21.15.47:6789/0 conn(0x7f3ea413ba60 legacy=0x7f3ea41495e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:54.482 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.470+0000 7f3ea2ffd700 1 -- 172.21.15.189:0/1636032947 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea40676f0 con 0x7f3ea4066ed0 2020-11-20T12:06:54.482 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.470+0000 7f3ea2ffd700 1 --2- 172.21.15.189:0/1636032947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ea4066ed0 0x7f3ea413b570 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f3e9800b9a0 tx=0x7f3e98006df0).ready entity=mon.1 client_cookie=f02f34629577547a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:54.482 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea27fc700 1 -- 172.21.15.189:0/1636032947 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e980097b0 con 0x7f3ea4066ed0 2020-11-20T12:06:54.483 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea27fc700 1 -- 172.21.15.189:0/1636032947 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e98009910 con 0x7f3ea4066ed0 2020-11-20T12:06:54.483 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea27fc700 1 -- 172.21.15.189:0/1636032947 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e98009c90 con 0x7f3ea4066ed0 2020-11-20T12:06:54.483 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea8f30700 1 -- 172.21.15.189:0/1636032947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ea4066ed0 msgr2=0x7f3ea413b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:54.483 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea8f30700 1 --2- 172.21.15.189:0/1636032947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ea4066ed0 0x7f3ea413b570 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f3e9800b9a0 tx=0x7f3e98006df0).stop 2020-11-20T12:06:54.483 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea8f30700 1 -- 172.21.15.189:0/1636032947 shutdown_connections 2020-11-20T12:06:54.484 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea8f30700 1 --2- 172.21.15.189:0/1636032947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ea4066ed0 0x7f3ea413b570 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:54.484 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea8f30700 1 -- 172.21.15.189:0/1636032947 >> 172.21.15.189:0/1636032947 conn(0x7f3ea4060170 msgr2=0x7f3ea4060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:54.484 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea8f30700 1 -- 172.21.15.189:0/1636032947 shutdown_connections 2020-11-20T12:06:54.484 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea8f30700 1 -- 172.21.15.189:0/1636032947 wait complete. 2020-11-20T12:06:54.484 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea8f30700 1 Processor -- start 2020-11-20T12:06:54.485 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea8f30700 1 -- start start 2020-11-20T12:06:54.485 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea8f30700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ea4066ed0 0x7f3ea40a4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:54.485 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea8f30700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ea40a8840 0x7f3ea40a23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:54.485 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea8f30700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3ea413ba60 0x7f3ea413a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:54.485 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea8f30700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea40a31c0 con 0x7f3ea4066ed0 2020-11-20T12:06:54.486 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea8f30700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea40a5660 con 0x7f3ea40a8840 2020-11-20T12:06:54.486 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea8f30700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea40a5900 con 0x7f3ea413ba60 2020-11-20T12:06:54.486 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea2ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ea40a8840 0x7f3ea40a23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:54.486 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea2ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ea40a8840 0x7f3ea40a23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49402/0 (socket says 172.21.15.189:49402) 2020-11-20T12:06:54.486 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea2ffd700 1 -- 172.21.15.189:0/3710779116 learned_addr learned my addr 172.21.15.189:0/3710779116 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:06:54.487 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea37fe700 1 --2- 172.21.15.189:0/3710779116 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ea4066ed0 0x7f3ea40a4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:54.487 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea3fff700 1 --2- 172.21.15.189:0/3710779116 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3ea413ba60 0x7f3ea413a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:54.487 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea2ffd700 1 -- 172.21.15.189:0/3710779116 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3ea413ba60 msgr2=0x7f3ea413a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:54.487 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea2ffd700 1 --2- 172.21.15.189:0/3710779116 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3ea413ba60 0x7f3ea413a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:54.487 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea2ffd700 1 -- 172.21.15.189:0/3710779116 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ea4066ed0 msgr2=0x7f3ea40a4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:54.488 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea2ffd700 1 --2- 172.21.15.189:0/3710779116 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ea4066ed0 0x7f3ea40a4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:54.488 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea2ffd700 1 -- 172.21.15.189:0/3710779116 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea40a5ba0 con 0x7f3ea40a8840 2020-11-20T12:06:54.488 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.474+0000 7f3ea2ffd700 1 --2- 172.21.15.189:0/3710779116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ea40a8840 0x7f3ea40a23f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f3ea4064a70 tx=0x7f3e98015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:54.488 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.478+0000 7f3e837fe700 1 -- 172.21.15.189:0/3710779116 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e98011e00 con 0x7f3ea40a8840 2020-11-20T12:06:54.488 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.478+0000 7f3e837fe700 1 -- 172.21.15.189:0/3710779116 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e9801e460 con 0x7f3ea40a8840 2020-11-20T12:06:54.489 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.478+0000 7f3e837fe700 1 -- 172.21.15.189:0/3710779116 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e9801e7e0 con 0x7f3ea40a8840 2020-11-20T12:06:54.489 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.478+0000 7f3ea8f30700 1 -- 172.21.15.189:0/3710779116 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea41414f0 con 0x7f3ea40a8840 2020-11-20T12:06:54.489 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.478+0000 7f3ea8f30700 1 -- 172.21.15.189:0/3710779116 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea4141820 con 0x7f3ea40a8840 2020-11-20T12:06:54.490 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.478+0000 7f3ea8f30700 1 -- 172.21.15.189:0/3710779116 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea40a54b0 con 0x7f3ea40a8840 2020-11-20T12:06:54.491 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.478+0000 7f3e837fe700 1 -- 172.21.15.189:0/3710779116 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f3e98025030 con 0x7f3ea40a8840 2020-11-20T12:06:54.491 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.478+0000 7f3e837fe700 1 --2- 172.21.15.189:0/3710779116 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3e7c0557c0 0x7f3e7c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:54.491 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.478+0000 7f3e837fe700 1 -- 172.21.15.189:0/3710779116 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f3e9806fd50 con 0x7f3ea40a8840 2020-11-20T12:06:54.491 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.478+0000 7f3ea37fe700 1 --2- 172.21.15.189:0/3710779116 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3e7c0557c0 0x7f3e7c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:54.491 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.478+0000 7f3ea37fe700 1 --2- 172.21.15.189:0/3710779116 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3e7c0557c0 0x7f3e7c057c80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f3e94005920 tx=0x7f3e94009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:54.492 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.486+0000 7f3e837fe700 1 -- 172.21.15.189:0/3710779116 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3e98045cd0 con 0x7f3ea40a8840 2020-11-20T12:06:54.693 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.686+0000 7f3ea8f30700 1 -- 172.21.15.189:0/3710779116 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ea40a3b90 con 0x7f3ea40a8840 2020-11-20T12:06:54.694 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.690+0000 7f3e837fe700 1 -- 172.21.15.189:0/3710779116 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+1782 (secure 0 0 0) 0x7f3e98023070 con 0x7f3ea40a8840 2020-11-20T12:06:54.694 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:06:54.694 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:06:54.572507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4248},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"b","rank":0,"incarnation":7,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.47:6833/2660417436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2660417436},{"type":"v1","addr":"172.21.15.47:6833","nonce":2660417436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:06:54.695 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.690+0000 7f3ea8f30700 1 -- 172.21.15.189:0/3710779116 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3e7c0557c0 msgr2=0x7f3e7c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:54.696 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.690+0000 7f3ea8f30700 1 --2- 172.21.15.189:0/3710779116 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3e7c0557c0 0x7f3e7c057c80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f3e94005920 tx=0x7f3e94009040).stop 2020-11-20T12:06:54.696 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.690+0000 7f3ea8f30700 1 -- 172.21.15.189:0/3710779116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ea40a8840 msgr2=0x7f3ea40a23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:54.697 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.690+0000 7f3ea8f30700 1 --2- 172.21.15.189:0/3710779116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ea40a8840 0x7f3ea40a23f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f3ea4064a70 tx=0x7f3e98015000).stop 2020-11-20T12:06:54.698 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.690+0000 7f3ea8f30700 1 -- 172.21.15.189:0/3710779116 shutdown_connections 2020-11-20T12:06:54.699 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.690+0000 7f3ea8f30700 1 --2- 172.21.15.189:0/3710779116 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3e7c0557c0 0x7f3e7c057c80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:54.699 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.690+0000 7f3ea8f30700 1 --2- 172.21.15.189:0/3710779116 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3ea4066ed0 0x7f3ea40a4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:54.699 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.690+0000 7f3ea8f30700 1 --2- 172.21.15.189:0/3710779116 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3ea413ba60 0x7f3ea413a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:54.700 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.690+0000 7f3ea8f30700 1 --2- 172.21.15.189:0/3710779116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ea40a8840 0x7f3ea40a23f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:54.700 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.690+0000 7f3ea8f30700 1 -- 172.21.15.189:0/3710779116 >> 172.21.15.189:0/3710779116 conn(0x7f3ea4060170 msgr2=0x7f3ea4144040 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:54.700 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.690+0000 7f3ea8f30700 1 -- 172.21.15.189:0/3710779116 shutdown_connections 2020-11-20T12:06:54.701 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.690+0000 7f3ea8f30700 1 -- 172.21.15.189:0/3710779116 wait complete. 2020-11-20T12:06:54.701 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 9 2020-11-20T12:06:54.709 INFO:tasks.mds_thrash.fs.[None]:mds.a down, removed from mdsmap 2020-11-20T12:06:54.710 INFO:tasks.mds_thrash.fs.[None]:waiting for mds cluster to stabilize... 2020-11-20T12:06:54.711 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:06:54.833 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f894b27a700 1 Processor -- start 2020-11-20T12:06:54.834 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f894b27a700 1 -- start start 2020-11-20T12:06:54.835 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f894b27a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f89440acba0 0x7f8944067400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:54.836 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f894b27a700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f89440678f0 0x7f89440a73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:54.836 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f894b27a700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89440a8690 con 0x7f89440a78a0 2020-11-20T12:06:54.836 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f894b27a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f894413c660 con 0x7f89440acba0 2020-11-20T12:06:54.836 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f894b27a700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89440acf80 con 0x7f89440678f0 2020-11-20T12:06:54.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f8949016700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f89440acba0 0x7f8944067400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:54.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f8949016700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f89440acba0 0x7f8944067400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49408/0 (socket says 172.21.15.189:49408) 2020-11-20T12:06:54.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f8949016700 1 -- 172.21.15.189:0/3077638069 learned_addr learned my addr 172.21.15.189:0/3077638069 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:06:54.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f8948815700 1 --2- 172.21.15.189:0/3077638069 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f89440678f0 0x7f89440a73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:54.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f8949016700 1 -- 172.21.15.189:0/3077638069 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f89440678f0 msgr2=0x7f89440a73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:54.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f8949016700 1 --2- 172.21.15.189:0/3077638069 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f89440678f0 0x7f89440a73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:54.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f8949016700 1 -- 172.21.15.189:0/3077638069 >> v1:172.21.15.47:6789/0 conn(0x7f89440a78a0 legacy=0x7f89440a9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:54.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f8949016700 1 -- 172.21.15.189:0/3077638069 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89440a88c0 con 0x7f89440acba0 2020-11-20T12:06:54.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f8949016700 1 --2- 172.21.15.189:0/3077638069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89440acba0 0x7f8944067400 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f8940006fd0 tx=0x7f8940006eb0).ready entity=mon.1 client_cookie=956749530a630fc9 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:54.839 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f893bfff700 1 -- 172.21.15.189:0/3077638069 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89400098a0 con 0x7f89440acba0 2020-11-20T12:06:54.839 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f893bfff700 1 -- 172.21.15.189:0/3077638069 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8940009a00 con 0x7f89440acba0 2020-11-20T12:06:54.839 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f893bfff700 1 -- 172.21.15.189:0/3077638069 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8940009d80 con 0x7f89440acba0 2020-11-20T12:06:54.842 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f894b27a700 1 -- 172.21.15.189:0/3077638069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89440acba0 msgr2=0x7f8944067400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:54.842 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f894b27a700 1 --2- 172.21.15.189:0/3077638069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89440acba0 0x7f8944067400 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f8940006fd0 tx=0x7f8940006eb0).stop 2020-11-20T12:06:54.842 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f894b27a700 1 -- 172.21.15.189:0/3077638069 shutdown_connections 2020-11-20T12:06:54.842 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f894b27a700 1 --2- 172.21.15.189:0/3077638069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89440acba0 0x7f8944067400 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:54.843 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f894b27a700 1 --2- 172.21.15.189:0/3077638069 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f89440678f0 0x7f89440a73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:54.843 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f894b27a700 1 -- 172.21.15.189:0/3077638069 >> 172.21.15.189:0/3077638069 conn(0x7f8944065420 msgr2=0x7f8944062080 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:54.843 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f894b27a700 1 -- 172.21.15.189:0/3077638069 shutdown_connections 2020-11-20T12:06:54.843 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.830+0000 7f894b27a700 1 -- 172.21.15.189:0/3077638069 wait complete. 2020-11-20T12:06:54.844 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.834+0000 7f894b27a700 1 Processor -- start 2020-11-20T12:06:54.844 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.834+0000 7f894b27a700 1 -- start start 2020-11-20T12:06:54.844 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.834+0000 7f894b27a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89440678f0 0x7f8944114990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:54.844 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.834+0000 7f894b27a700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f89440a78a0 0x7f8944144910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:54.844 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.834+0000 7f894b27a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89440acba0 0x7f8944144e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:54.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.834+0000 7f894b27a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89441452f0 con 0x7f89440678f0 2020-11-20T12:06:54.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.834+0000 7f894b27a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894414e980 con 0x7f89440acba0 2020-11-20T12:06:54.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.834+0000 7f894b27a700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f894414ec20 con 0x7f89440a78a0 2020-11-20T12:06:54.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.834+0000 7f8949817700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89440acba0 0x7f8944144e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.834+0000 7f8949016700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89440678f0 0x7f8944114990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.834+0000 7f8948815700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f89440a78a0 0x7f8944144910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.834+0000 7f8949016700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89440678f0 0x7f8944114990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.189:55638/0 (socket says 172.21.15.189:55638) 2020-11-20T12:06:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.834+0000 7f8949016700 1 -- 172.21.15.189:0/317764007 learned_addr learned my addr 172.21.15.189:0/317764007 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:06:54.846 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.834+0000 7f8949817700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89440acba0 0x7f8944144e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49418/0 (socket says 172.21.15.189:49418) 2020-11-20T12:06:54.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.834+0000 7f8949016700 1 -- 172.21.15.189:0/317764007 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f89440a78a0 msgr2=0x7f8944144910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:54.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.834+0000 7f8949016700 1 --2- 172.21.15.189:0/317764007 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f89440a78a0 0x7f8944144910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:54.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.834+0000 7f8949016700 1 -- 172.21.15.189:0/317764007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89440acba0 msgr2=0x7f8944144e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:54.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.834+0000 7f8949016700 1 --2- 172.21.15.189:0/317764007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89440acba0 0x7f8944144e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:54.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.834+0000 7f8949016700 1 -- 172.21.15.189:0/317764007 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89440a3760 con 0x7f89440678f0 2020-11-20T12:06:54.848 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.834+0000 7f8948815700 1 --2- 172.21.15.189:0/317764007 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f89440a78a0 0x7f8944144910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:06:54.848 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.834+0000 7f8949817700 1 --2- 172.21.15.189:0/317764007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89440acba0 0x7f8944144e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:06:54.848 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.834+0000 7f8949016700 1 --2- 172.21.15.189:0/317764007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89440678f0 0x7f8944114990 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f8940006ef0 tx=0x7f8940015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:54.848 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.838+0000 7f89397fa700 1 -- 172.21.15.189:0/317764007 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8940011dd0 con 0x7f89440678f0 2020-11-20T12:06:54.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.838+0000 7f89397fa700 1 -- 172.21.15.189:0/317764007 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f894001e460 con 0x7f89440678f0 2020-11-20T12:06:54.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.838+0000 7f894b27a700 1 -- 172.21.15.189:0/317764007 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89441186c0 con 0x7f89440678f0 2020-11-20T12:06:54.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.838+0000 7f89397fa700 1 -- 172.21.15.189:0/317764007 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f894001e7e0 con 0x7f89440678f0 2020-11-20T12:06:54.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.838+0000 7f894b27a700 1 -- 172.21.15.189:0/317764007 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f894414e5a0 con 0x7f89440678f0 2020-11-20T12:06:54.850 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.838+0000 7f894b27a700 1 -- 172.21.15.189:0/317764007 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8944044f60 con 0x7f89440678f0 2020-11-20T12:06:54.850 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.838+0000 7f89397fa700 1 -- 172.21.15.189:0/317764007 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f8940025030 con 0x7f89440678f0 2020-11-20T12:06:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.838+0000 7f89397fa700 1 --2- 172.21.15.189:0/317764007 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f891c0557c0 0x7f891c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.842+0000 7f8948815700 1 --2- 172.21.15.189:0/317764007 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f891c0557c0 0x7f891c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.842+0000 7f89397fa700 1 -- 172.21.15.189:0/317764007 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f89400703b0 con 0x7f89440678f0 2020-11-20T12:06:54.851 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.842+0000 7f8948815700 1 --2- 172.21.15.189:0/317764007 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f891c0557c0 0x7f891c057c80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f8944118540 tx=0x7f8934006df0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:54.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:54.850+0000 7f89397fa700 1 -- 172.21.15.189:0/317764007 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8940046220 con 0x7f89440678f0 2020-11-20T12:06:55.062 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:55.058+0000 7f894b27a700 1 -- 172.21.15.189:0/317764007 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89440446d0 con 0x7f89440678f0 2020-11-20T12:06:55.063 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:55.058+0000 7f89397fa700 1 -- 172.21.15.189:0/317764007 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+1782 (secure 0 0 0) 0x7f8940023030 con 0x7f89440678f0 2020-11-20T12:06:55.064 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:06:55.064 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:06:54.572507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4248},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"b","rank":0,"incarnation":7,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.47:6833/2660417436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2660417436},{"type":"v1","addr":"172.21.15.47:6833","nonce":2660417436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:06:55.065 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:55.062+0000 7f894b27a700 1 -- 172.21.15.189:0/317764007 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f891c0557c0 msgr2=0x7f891c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:55.066 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:55.062+0000 7f894b27a700 1 --2- 172.21.15.189:0/317764007 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f891c0557c0 0x7f891c057c80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f8944118540 tx=0x7f8934006df0).stop 2020-11-20T12:06:55.066 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:55.062+0000 7f894b27a700 1 -- 172.21.15.189:0/317764007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89440678f0 msgr2=0x7f8944114990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:55.066 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:55.062+0000 7f894b27a700 1 --2- 172.21.15.189:0/317764007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89440678f0 0x7f8944114990 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f8940006ef0 tx=0x7f8940015000).stop 2020-11-20T12:06:55.066 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:55.062+0000 7f894b27a700 1 -- 172.21.15.189:0/317764007 shutdown_connections 2020-11-20T12:06:55.066 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:55.062+0000 7f894b27a700 1 --2- 172.21.15.189:0/317764007 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f891c0557c0 0x7f891c057c80 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:55.067 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:55.062+0000 7f894b27a700 1 --2- 172.21.15.189:0/317764007 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f89440678f0 0x7f8944114990 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:55.067 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:55.062+0000 7f894b27a700 1 --2- 172.21.15.189:0/317764007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89440acba0 0x7f8944144e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:55.067 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:55.062+0000 7f894b27a700 1 --2- 172.21.15.189:0/317764007 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f89440a78a0 0x7f8944144910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:55.067 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:55.062+0000 7f894b27a700 1 -- 172.21.15.189:0/317764007 >> 172.21.15.189:0/317764007 conn(0x7f8944065420 msgr2=0x7f89440a9230 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:55.067 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:55.062+0000 7f894b27a700 1 -- 172.21.15.189:0/317764007 shutdown_connections 2020-11-20T12:06:55.068 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:55.062+0000 7f894b27a700 1 -- 172.21.15.189:0/317764007 wait complete. 2020-11-20T12:06:55.068 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 9 2020-11-20T12:06:55.079 INFO:tasks.mds_thrash.fs.[None]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 9, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2020-11-20T12:04:54.012110+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 9, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4248": { "addr": "172.21.15.47:6833/2660417436", "addrs": { "addrvec": [ { "addr": "172.21.15.47:6832", "nonce": 2660417436, "type": "v2" }, { "addr": "172.21.15.47:6833", "nonce": 2660417436, "type": "v1" } ] }, "export_targets": [], "features": 4540138297136906239, "flags": 0, "gid": 4248, "incarnation": 7, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:rejoin", "state_seq": 33 } }, "last_failure": 0, "last_failure_osd_epoch": 23, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2020-11-20T12:06:54.572507+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4248 } } } ], "standbys": [] } 2020-11-20T12:06:55.633 INFO:tasks.ceph.mds.b.smithi047.stderr:2020-11-20T12:06:55.631+0000 7ff130355700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:06:57.080 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:06:57.212 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.202+0000 7fb9915cf700 1 Processor -- start 2020-11-20T12:06:57.212 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.202+0000 7fb9915cf700 1 -- start start 2020-11-20T12:06:57.212 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.202+0000 7fb9915cf700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb98c1347f0 0x7fb98c0677f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:57.213 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.202+0000 7fb9915cf700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb98c0a9c60 con 0x7fb98c0623c0 2020-11-20T12:06:57.213 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.202+0000 7fb9915cf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb98c067e90 con 0x7fb98c13b1d0 2020-11-20T12:06:57.214 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.202+0000 7fb9915cf700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb98c0a8e60 con 0x7fb98c1347f0 2020-11-20T12:06:57.214 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.202+0000 7fb98b7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb98c13b1d0 0x7fb98c1340e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51046/0 (socket says 172.21.15.189:51046) 2020-11-20T12:06:57.214 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.202+0000 7fb98b7fe700 1 -- 172.21.15.189:0/1346503018 learned_addr learned my addr 172.21.15.189:0/1346503018 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:06:57.215 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb990dce700 1 --2- 172.21.15.189:0/1346503018 >> v2:172.21.15.47:3300/0 conn(0x7fb98c1347f0 0x7fb98c0677f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:57.215 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb98affd700 1 -- 172.21.15.189:0/1346503018 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 203313032 0 0) 0x7fb98c067e90 con 0x7fb98c13b1d0 2020-11-20T12:06:57.215 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb98affd700 1 -- 172.21.15.189:0/1346503018 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb978003240 con 0x7fb98c13b1d0 2020-11-20T12:06:57.215 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb98affd700 1 -- 172.21.15.189:0/1346503018 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2138977125 0 0) 0x7fb978003240 con 0x7fb98c13b1d0 2020-11-20T12:06:57.216 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb98affd700 1 -- 172.21.15.189:0/1346503018 >> v2:172.21.15.47:3300/0 conn(0x7fb98c1347f0 msgr2=0x7fb98c0677f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:57.216 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb98affd700 1 --2- 172.21.15.189:0/1346503018 >> v2:172.21.15.47:3300/0 conn(0x7fb98c1347f0 0x7fb98c0677f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:57.216 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb98affd700 1 -- 172.21.15.189:0/1346503018 >> v1:172.21.15.47:6789/0 conn(0x7fb98c0623c0 legacy=0x7fb98c13aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:57.216 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb98affd700 1 -- 172.21.15.189:0/1346503018 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb98c068130 con 0x7fb98c13b1d0 2020-11-20T12:06:57.217 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb98affd700 1 -- 172.21.15.189:0/1346503018 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7fb9800014e0 con 0x7fb98c13b1d0 2020-11-20T12:06:57.217 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb98affd700 1 -- 172.21.15.189:0/1346503018 >> v1:172.21.15.189:6789/0 conn(0x7fb98c13b1d0 legacy=0x7fb98c1340e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:57.217 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb98affd700 1 --2- 172.21.15.189:0/1346503018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb978003730 0x7fb978003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:57.218 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb98affd700 1 --2- 172.21.15.189:0/1346503018 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb9780044e0 0x7fb978004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:57.218 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb98affd700 1 --2- 172.21.15.189:0/1346503018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb978004e50 0x7fb97800b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:57.218 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb98affd700 1 -- 172.21.15.189:0/1346503018 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb978004260 con 0x7fb978004e50 2020-11-20T12:06:57.218 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb98affd700 1 -- 172.21.15.189:0/1346503018 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97800bc20 con 0x7fb978003730 2020-11-20T12:06:57.219 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb98affd700 1 -- 172.21.15.189:0/1346503018 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb97800bde0 con 0x7fb9780044e0 2020-11-20T12:06:57.219 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb98bfff700 1 --2- 172.21.15.189:0/1346503018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb978003730 0x7fb978003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:57.219 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb990dce700 1 --2- 172.21.15.189:0/1346503018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb978004e50 0x7fb97800b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:57.220 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb98b7fe700 1 --2- 172.21.15.189:0/1346503018 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb9780044e0 0x7fb978004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:57.220 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb98bfff700 1 -- 172.21.15.189:0/1346503018 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb9780044e0 msgr2=0x7fb978004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:57.220 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb98bfff700 1 --2- 172.21.15.189:0/1346503018 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb9780044e0 0x7fb978004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:57.220 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb98bfff700 1 -- 172.21.15.189:0/1346503018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb978004e50 msgr2=0x7fb97800b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:06:57.220 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb98bfff700 1 --2- 172.21.15.189:0/1346503018 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb978004e50 0x7fb97800b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:57.221 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb98bfff700 1 -- 172.21.15.189:0/1346503018 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb978003240 con 0x7fb978003730 2020-11-20T12:06:57.221 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb990dce700 1 -- 172.21.15.189:0/1346503018 reap_dead start 2020-11-20T12:06:57.221 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb98bfff700 1 --2- 172.21.15.189:0/1346503018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb978003730 0x7fb978003b50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fb97c001920 tx=0x7fb97c007b90).ready entity=mon.1 client_cookie=35c23dfe6fdfe182 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:57.222 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb98affd700 1 -- 172.21.15.189:0/1346503018 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c007020 con 0x7fb978003730 2020-11-20T12:06:57.222 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb98affd700 1 -- 172.21.15.189:0/1346503018 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97c007180 con 0x7fb978003730 2020-11-20T12:06:57.223 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb98affd700 1 -- 172.21.15.189:0/1346503018 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c00c830 con 0x7fb978003730 2020-11-20T12:06:57.223 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb9915cf700 1 -- 172.21.15.189:0/1346503018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb978003730 msgr2=0x7fb978003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:57.223 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb9915cf700 1 --2- 172.21.15.189:0/1346503018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb978003730 0x7fb978003b50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fb97c001920 tx=0x7fb97c007b90).stop 2020-11-20T12:06:57.223 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb9915cf700 1 -- 172.21.15.189:0/1346503018 shutdown_connections 2020-11-20T12:06:57.224 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb9915cf700 1 --2- 172.21.15.189:0/1346503018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb978003730 0x7fb978003b50 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:57.224 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb9915cf700 1 -- 172.21.15.189:0/1346503018 >> 172.21.15.189:0/1346503018 conn(0x7fb98c065fe0 msgr2=0x7fb98c061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:57.224 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb9915cf700 1 -- 172.21.15.189:0/1346503018 shutdown_connections 2020-11-20T12:06:57.225 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.206+0000 7fb9915cf700 1 -- 172.21.15.189:0/1346503018 wait complete. 2020-11-20T12:06:57.229 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.210+0000 7fb9915cf700 1 Processor -- start 2020-11-20T12:06:57.229 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.210+0000 7fb9915cf700 1 -- start start 2020-11-20T12:06:57.229 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.210+0000 7fb9915cf700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb978003730 0x7fb98c0a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:57.229 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.210+0000 7fb9915cf700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb98c1397b0 0x7fb98c0a7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:57.230 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.210+0000 7fb9915cf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb98c131c60 0x7fb98c1320c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:57.230 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.210+0000 7fb9915cf700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98c13fb50 con 0x7fb978003730 2020-11-20T12:06:57.230 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.210+0000 7fb9915cf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98c139c30 con 0x7fb98c131c60 2020-11-20T12:06:57.231 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.210+0000 7fb9915cf700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb98c0a6000 con 0x7fb98c1397b0 2020-11-20T12:06:57.231 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.210+0000 7fb990dce700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb98c131c60 0x7fb98c1320c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:57.231 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.210+0000 7fb990dce700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb98c131c60 0x7fb98c1320c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49438/0 (socket says 172.21.15.189:49438) 2020-11-20T12:06:57.231 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.210+0000 7fb990dce700 1 -- 172.21.15.189:0/1042944034 learned_addr learned my addr 172.21.15.189:0/1042944034 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:06:57.232 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.210+0000 7fb98bfff700 1 --2- 172.21.15.189:0/1042944034 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb978003730 0x7fb98c0a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:57.232 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.210+0000 7fb98b7fe700 1 --2- 172.21.15.189:0/1042944034 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb98c1397b0 0x7fb98c0a7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:57.232 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.210+0000 7fb990dce700 1 -- 172.21.15.189:0/1042944034 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb98c1397b0 msgr2=0x7fb98c0a7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:57.233 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.210+0000 7fb990dce700 1 --2- 172.21.15.189:0/1042944034 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb98c1397b0 0x7fb98c0a7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:57.233 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.210+0000 7fb990dce700 1 -- 172.21.15.189:0/1042944034 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb978003730 msgr2=0x7fb98c0a53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:57.233 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.210+0000 7fb990dce700 1 --2- 172.21.15.189:0/1042944034 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb978003730 0x7fb98c0a53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:57.233 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.210+0000 7fb990dce700 1 -- 172.21.15.189:0/1042944034 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb98c0a5d60 con 0x7fb98c131c60 2020-11-20T12:06:57.234 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.210+0000 7fb98b7fe700 1 --2- 172.21.15.189:0/1042944034 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb98c1397b0 0x7fb98c0a7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:06:57.234 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.210+0000 7fb990dce700 1 --2- 172.21.15.189:0/1042944034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb98c131c60 0x7fb98c1320c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fb98c13b5b0 tx=0x7fb9840091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:57.235 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.210+0000 7fb96bfff700 1 -- 172.21.15.189:0/1042944034 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb98400fb00 con 0x7fb98c131c60 2020-11-20T12:06:57.235 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.210+0000 7fb96bfff700 1 -- 172.21.15.189:0/1042944034 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb98400fc60 con 0x7fb98c131c60 2020-11-20T12:06:57.235 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.210+0000 7fb96bfff700 1 -- 172.21.15.189:0/1042944034 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb984014440 con 0x7fb98c131c60 2020-11-20T12:06:57.236 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.210+0000 7fb9915cf700 1 -- 172.21.15.189:0/1042944034 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb98c133510 con 0x7fb98c131c60 2020-11-20T12:06:57.236 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.210+0000 7fb9915cf700 1 -- 172.21.15.189:0/1042944034 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb98c0a7800 con 0x7fb98c131c60 2020-11-20T12:06:57.237 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.210+0000 7fb9915cf700 1 -- 172.21.15.189:0/1042944034 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb98c068310 con 0x7fb98c131c60 2020-11-20T12:06:57.237 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.214+0000 7fb96bfff700 1 -- 172.21.15.189:0/1042944034 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fb984014830 con 0x7fb98c131c60 2020-11-20T12:06:57.237 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.214+0000 7fb96bfff700 1 --2- 172.21.15.189:0/1042944034 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb9640557f0 0x7fb964057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:06:57.238 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.214+0000 7fb96bfff700 1 -- 172.21.15.189:0/1042944034 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fb98406f320 con 0x7fb98c131c60 2020-11-20T12:06:57.238 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.214+0000 7fb98bfff700 1 --2- 172.21.15.189:0/1042944034 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb9640557f0 0x7fb964057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:06:57.239 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.214+0000 7fb98bfff700 1 --2- 172.21.15.189:0/1042944034 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb9640557f0 0x7fb964057cb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fb97c0077d0 tx=0x7fb97c015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:06:57.239 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.218+0000 7fb96bfff700 1 -- 172.21.15.189:0/1042944034 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb984045170 con 0x7fb98c131c60 2020-11-20T12:06:57.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.446+0000 7fb9915cf700 1 -- 172.21.15.189:0/1042944034 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb98c1336a0 con 0x7fb98c131c60 2020-11-20T12:06:57.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.446+0000 7fb96bfff700 1 -- 172.21.15.189:0/1042944034 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+1784 (secure 0 0 0) 0x7fb984048790 con 0x7fb98c131c60 2020-11-20T12:06:57.451 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:06:57.452 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:06:55.583738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4248},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"b","rank":0,"incarnation":7,"state":"up:active","state_seq":34,"addr":"172.21.15.47:6833/2660417436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2660417436},{"type":"v1","addr":"172.21.15.47:6833","nonce":2660417436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:06:57.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.446+0000 7fb9915cf700 1 -- 172.21.15.189:0/1042944034 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb9640557f0 msgr2=0x7fb964057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:57.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.450+0000 7fb9915cf700 1 --2- 172.21.15.189:0/1042944034 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb9640557f0 0x7fb964057cb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fb97c0077d0 tx=0x7fb97c015000).stop 2020-11-20T12:06:57.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.450+0000 7fb9915cf700 1 -- 172.21.15.189:0/1042944034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb98c131c60 msgr2=0x7fb98c1320c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:06:57.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.450+0000 7fb9915cf700 1 --2- 172.21.15.189:0/1042944034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb98c131c60 0x7fb98c1320c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fb98c13b5b0 tx=0x7fb9840091d0).stop 2020-11-20T12:06:57.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.450+0000 7fb9915cf700 1 -- 172.21.15.189:0/1042944034 shutdown_connections 2020-11-20T12:06:57.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.450+0000 7fb9915cf700 1 --2- 172.21.15.189:0/1042944034 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb9640557f0 0x7fb964057cb0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:57.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.450+0000 7fb9915cf700 1 --2- 172.21.15.189:0/1042944034 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb978003730 0x7fb98c0a53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:57.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.450+0000 7fb9915cf700 1 --2- 172.21.15.189:0/1042944034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb98c131c60 0x7fb98c1320c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:57.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.450+0000 7fb9915cf700 1 --2- 172.21.15.189:0/1042944034 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb98c1397b0 0x7fb98c0a7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:06:57.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.450+0000 7fb9915cf700 1 -- 172.21.15.189:0/1042944034 >> 172.21.15.189:0/1042944034 conn(0x7fb98c065fe0 msgr2=0x7fb98c05ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:06:57.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.450+0000 7fb9915cf700 1 -- 172.21.15.189:0/1042944034 shutdown_connections 2020-11-20T12:06:57.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:06:57.450+0000 7fb9915cf700 1 -- 172.21.15.189:0/1042944034 wait complete. 2020-11-20T12:06:57.456 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 10 2020-11-20T12:06:57.468 INFO:tasks.mds_thrash.fs.[None]:mds.b has gained rank=0, replacing gid=4273 2020-11-20T12:06:57.468 INFO:tasks.mds_thrash.fs.[None]:waiting for 6 secs before reviving mds.a 2020-11-20T12:07:03.469 INFO:tasks.mds_thrash.fs.[None]:reviving mds.a 2020-11-20T12:07:03.470 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:07:03.470 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:07:03.473 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:07:03.474 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:07:03.550 INFO:tasks.ceph.mds.a.smithi189.stdout:starting mds.a at 2020-11-20T12:07:03.603 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.598+0000 7f0c42bee700 1 Processor -- start 2020-11-20T12:07:03.603 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.598+0000 7f0c42bee700 1 -- start start 2020-11-20T12:07:03.611 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.598+0000 7f0c42bee700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0c3c0ad000 0x7f0c3c13e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:03.611 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.598+0000 7f0c42bee700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0c3c13a6e0 0x7f0c3c13ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:03.612 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.598+0000 7f0c42bee700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c3c067730 con 0x7f0c3c0ad3e0 2020-11-20T12:07:03.612 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.598+0000 7f0c42bee700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c3c138190 con 0x7f0c3c13a6e0 2020-11-20T12:07:03.612 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.598+0000 7f0c42bee700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c13dd40 con 0x7f0c3c0ad000 2020-11-20T12:07:03.613 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.598+0000 7f0c423ed700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0c3c13a6e0 0x7f0c3c13ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:03.613 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.598+0000 7f0c423ed700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0c3c13a6e0 0x7f0c3c13ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49456/0 (socket says 172.21.15.189:49456) 2020-11-20T12:07:03.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.598+0000 7f0c423ed700 1 -- 172.21.15.189:0/3999617191 learned_addr learned my addr 172.21.15.189:0/3999617191 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:07:03.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.598+0000 7f0c413eb700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0c3c0ad3e0 0x7f0c3c066d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51080/0 (socket says 172.21.15.189:51080) 2020-11-20T12:07:03.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.598+0000 7f0c41bec700 1 --2- 172.21.15.189:0/3999617191 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0c3c0ad000 0x7f0c3c13e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:03.615 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.598+0000 7f0c423ed700 1 -- 172.21.15.189:0/3999617191 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0c3c0ad000 msgr2=0x7f0c3c13e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:03.615 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.598+0000 7f0c423ed700 1 --2- 172.21.15.189:0/3999617191 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0c3c0ad000 0x7f0c3c13e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.616 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.598+0000 7f0c423ed700 1 -- 172.21.15.189:0/3999617191 >> v1:172.21.15.189:6789/0 conn(0x7f0c3c0ad3e0 legacy=0x7f0c3c066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:03.616 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.598+0000 7f0c40bea700 1 -- 172.21.15.189:0/3999617191 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1029196584 0 0) 0x7f0c3c067730 con 0x7f0c3c0ad3e0 2020-11-20T12:07:03.616 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.598+0000 7f0c423ed700 1 -- 172.21.15.189:0/3999617191 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c3c13dfe0 con 0x7f0c3c13a6e0 2020-11-20T12:07:03.617 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.598+0000 7f0c41bec700 1 --2- 172.21.15.189:0/3999617191 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0c3c0ad000 0x7f0c3c13e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:07:03.617 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.598+0000 7f0c423ed700 1 --2- 172.21.15.189:0/3999617191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3c13a6e0 0x7f0c3c13ad00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f0c34009d50 tx=0x7f0c34009290).ready entity=mon.1 client_cookie=cfe52e3cdeebfa5f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:03.618 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.598+0000 7f0c40bea700 1 -- 172.21.15.189:0/3999617191 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3400bc80 con 0x7f0c3c13a6e0 2020-11-20T12:07:03.618 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.598+0000 7f0c40bea700 1 -- 172.21.15.189:0/3999617191 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c3400bde0 con 0x7f0c3c13a6e0 2020-11-20T12:07:03.618 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.598+0000 7f0c40bea700 1 -- 172.21.15.189:0/3999617191 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c34011610 con 0x7f0c3c13a6e0 2020-11-20T12:07:03.619 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c42bee700 1 -- 172.21.15.189:0/3999617191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3c13a6e0 msgr2=0x7f0c3c13ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:03.619 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c42bee700 1 --2- 172.21.15.189:0/3999617191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3c13a6e0 0x7f0c3c13ad00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f0c34009d50 tx=0x7f0c34009290).stop 2020-11-20T12:07:03.619 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c42bee700 1 -- 172.21.15.189:0/3999617191 shutdown_connections 2020-11-20T12:07:03.620 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c42bee700 1 --2- 172.21.15.189:0/3999617191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3c13a6e0 0x7f0c3c13ad00 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.620 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c42bee700 1 --2- 172.21.15.189:0/3999617191 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0c3c0ad000 0x7f0c3c13e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.621 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c42bee700 1 -- 172.21.15.189:0/3999617191 >> 172.21.15.189:0/3999617191 conn(0x7f0c3c062080 msgr2=0x7f0c3c062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:03.621 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c42bee700 1 -- 172.21.15.189:0/3999617191 shutdown_connections 2020-11-20T12:07:03.621 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c42bee700 1 -- 172.21.15.189:0/3999617191 wait complete. 2020-11-20T12:07:03.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c42bee700 1 Processor -- start 2020-11-20T12:07:03.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c42bee700 1 -- start start 2020-11-20T12:07:03.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c42bee700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0c3c0ad000 0x7f0c3c13cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:03.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c42bee700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c3c0ad3e0 0x7f0c3c0a1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:03.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c42bee700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3c0a51b0 0x7f0c3c0a23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:03.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c42bee700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c0a3320 con 0x7f0c3c0ad3e0 2020-11-20T12:07:03.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c42bee700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c0a35c0 con 0x7f0c3c0a51b0 2020-11-20T12:07:03.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c42bee700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c1413d0 con 0x7f0c3c0ad000 2020-11-20T12:07:03.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c423ed700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3c0a51b0 0x7f0c3c0a23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:03.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c423ed700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3c0a51b0 0x7f0c3c0a23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49464/0 (socket says 172.21.15.189:49464) 2020-11-20T12:07:03.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c423ed700 1 -- 172.21.15.189:0/911008562 learned_addr learned my addr 172.21.15.189:0/911008562 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:07:03.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c41bec700 1 --2- 172.21.15.189:0/911008562 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0c3c0ad000 0x7f0c3c13cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:03.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c413eb700 1 --2- 172.21.15.189:0/911008562 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c3c0ad3e0 0x7f0c3c0a1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:03.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c423ed700 1 -- 172.21.15.189:0/911008562 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0c3c0ad000 msgr2=0x7f0c3c13cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:03.626 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c423ed700 1 --2- 172.21.15.189:0/911008562 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0c3c0ad000 0x7f0c3c13cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.626 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c423ed700 1 -- 172.21.15.189:0/911008562 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c3c0ad3e0 msgr2=0x7f0c3c0a1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:03.626 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c423ed700 1 --2- 172.21.15.189:0/911008562 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c3c0ad3e0 0x7f0c3c0a1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.627 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c423ed700 1 -- 172.21.15.189:0/911008562 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c3c0a2da0 con 0x7f0c3c0a51b0 2020-11-20T12:07:03.627 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.602+0000 7f0c41bec700 1 --2- 172.21.15.189:0/911008562 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0c3c0ad000 0x7f0c3c13cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:07:03.628 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.606+0000 7f0c423ed700 1 --2- 172.21.15.189:0/911008562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3c0a51b0 0x7f0c3c0a23b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f0c340092d0 tx=0x7f0c34016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:03.628 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.606+0000 7f0c29ffb700 1 -- 172.21.15.189:0/911008562 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c34020070 con 0x7f0c3c0a51b0 2020-11-20T12:07:03.628 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.606+0000 7f0c29ffb700 1 -- 172.21.15.189:0/911008562 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c34009e90 con 0x7f0c3c0a51b0 2020-11-20T12:07:03.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.606+0000 7f0c42bee700 1 -- 172.21.15.189:0/911008562 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c3c0a2ff0 con 0x7f0c3c0a51b0 2020-11-20T12:07:03.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.606+0000 7f0c29ffb700 1 -- 172.21.15.189:0/911008562 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c34011650 con 0x7f0c3c0a51b0 2020-11-20T12:07:03.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.606+0000 7f0c42bee700 1 -- 172.21.15.189:0/911008562 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c3c14d2f0 con 0x7f0c3c0a51b0 2020-11-20T12:07:03.630 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.606+0000 7f0c42bee700 1 -- 172.21.15.189:0/911008562 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c3c0a3780 con 0x7f0c3c0a51b0 2020-11-20T12:07:03.630 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.606+0000 7f0c29ffb700 1 -- 172.21.15.189:0/911008562 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f0c34017070 con 0x7f0c3c0a51b0 2020-11-20T12:07:03.632 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.606+0000 7f0c29ffb700 1 --2- 172.21.15.189:0/911008562 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0c140557c0 0x7f0c14057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:03.632 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.606+0000 7f0c29ffb700 1 -- 172.21.15.189:0/911008562 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f0c34070220 con 0x7f0c3c0a51b0 2020-11-20T12:07:03.632 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.606+0000 7f0c41bec700 1 --2- 172.21.15.189:0/911008562 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0c140557c0 0x7f0c14057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:03.633 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.606+0000 7f0c41bec700 1 --2- 172.21.15.189:0/911008562 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0c140557c0 0x7f0c14057c80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f0c3c141920 tx=0x7f0c2c008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:03.633 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.614+0000 7f0c29ffb700 1 -- 172.21.15.189:0/911008562 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0c340460b0 con 0x7f0c3c0a51b0 2020-11-20T12:07:03.840 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.834+0000 7f0c42bee700 1 -- 172.21.15.189:0/911008562 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c3c0446d0 con 0x7f0c3c0a51b0 2020-11-20T12:07:03.841 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.838+0000 7f0c29ffb700 1 -- 172.21.15.189:0/911008562 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+1784 (secure 0 0 0) 0x7f0c34015020 con 0x7f0c3c0a51b0 2020-11-20T12:07:03.842 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:07:03.842 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:06:55.583738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4248},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"b","rank":0,"incarnation":7,"state":"up:active","state_seq":34,"addr":"172.21.15.47:6833/2660417436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2660417436},{"type":"v1","addr":"172.21.15.47:6833","nonce":2660417436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:07:03.843 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.838+0000 7f0c42bee700 1 -- 172.21.15.189:0/911008562 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0c140557c0 msgr2=0x7f0c14057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:03.843 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.838+0000 7f0c42bee700 1 --2- 172.21.15.189:0/911008562 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0c140557c0 0x7f0c14057c80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f0c3c141920 tx=0x7f0c2c008040).stop 2020-11-20T12:07:03.843 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.838+0000 7f0c42bee700 1 -- 172.21.15.189:0/911008562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3c0a51b0 msgr2=0x7f0c3c0a23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:03.844 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.838+0000 7f0c42bee700 1 --2- 172.21.15.189:0/911008562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3c0a51b0 0x7f0c3c0a23b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f0c340092d0 tx=0x7f0c34016000).stop 2020-11-20T12:07:03.844 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.838+0000 7f0c42bee700 1 -- 172.21.15.189:0/911008562 shutdown_connections 2020-11-20T12:07:03.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.838+0000 7f0c42bee700 1 --2- 172.21.15.189:0/911008562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c3c0a51b0 0x7f0c3c0a23b0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.838+0000 7f0c42bee700 1 --2- 172.21.15.189:0/911008562 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0c3c0ad000 0x7f0c3c13cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.838+0000 7f0c42bee700 1 --2- 172.21.15.189:0/911008562 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0c140557c0 0x7f0c14057c80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.838+0000 7f0c42bee700 1 --2- 172.21.15.189:0/911008562 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c3c0ad3e0 0x7f0c3c0a1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:03.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.838+0000 7f0c42bee700 1 -- 172.21.15.189:0/911008562 >> 172.21.15.189:0/911008562 conn(0x7f0c3c062080 msgr2=0x7f0c3c0ac380 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:03.846 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.838+0000 7f0c42bee700 1 -- 172.21.15.189:0/911008562 shutdown_connections 2020-11-20T12:07:03.846 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:03.838+0000 7f0c42bee700 1 -- 172.21.15.189:0/911008562 wait complete. 2020-11-20T12:07:03.846 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 10 2020-11-20T12:07:03.860 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2020-11-20T12:07:05.861 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:07:05.998 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.994+0000 7ff889d67700 1 Processor -- start 2020-11-20T12:07:05.998 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.994+0000 7ff889d67700 1 -- start start 2020-11-20T12:07:05.999 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.994+0000 7ff889d67700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff8840623c0 0x7ff8840677f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:05.999 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.994+0000 7ff889d67700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff88413aac0 0x7ff88413af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:06.000 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.994+0000 7ff889d67700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff88413b430 con 0x7ff8841340e0 2020-11-20T12:07:06.000 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.994+0000 7ff889d67700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8840a8e60 con 0x7ff88413aac0 2020-11-20T12:07:06.000 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.994+0000 7ff889d67700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8840a9c60 con 0x7ff8840623c0 2020-11-20T12:07:06.001 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.994+0000 7ff888d65700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff8840623c0 0x7ff8840677f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:06.001 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.994+0000 7ff883fff700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff88413aac0 0x7ff88413af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:06.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.994+0000 7ff888d65700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff8840623c0 0x7ff8840677f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.189:42504/0 (socket says 172.21.15.189:42504) 2020-11-20T12:07:06.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.994+0000 7ff888d65700 1 -- 172.21.15.189:0/154599707 learned_addr learned my addr 172.21.15.189:0/154599707 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:07:06.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.994+0000 7ff888d65700 1 -- 172.21.15.189:0/154599707 >> v2:172.21.15.47:3300/0 conn(0x7ff88413aac0 msgr2=0x7ff88413af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:06.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.994+0000 7ff888d65700 1 --2- 172.21.15.189:0/154599707 >> v2:172.21.15.47:3300/0 conn(0x7ff88413aac0 0x7ff88413af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:06.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.994+0000 7ff888d65700 1 -- 172.21.15.189:0/154599707 >> v1:172.21.15.47:6789/0 conn(0x7ff8841340e0 legacy=0x7ff884134590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:06.003 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.994+0000 7ff888d65700 1 -- 172.21.15.189:0/154599707 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff884067e90 con 0x7ff8840623c0 2020-11-20T12:07:06.003 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.994+0000 7ff888d65700 1 --2- 172.21.15.189:0/154599707 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff8840623c0 0x7ff8840677f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7ff878006fd0 tx=0x7ff878006eb0).ready entity=mon.2 client_cookie=4e82e8c42ef4ebc9 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:06.003 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.998+0000 7ff8837fe700 1 -- 172.21.15.189:0/154599707 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff878009810 con 0x7ff8840623c0 2020-11-20T12:07:06.003 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.998+0000 7ff8837fe700 1 -- 172.21.15.189:0/154599707 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff878009970 con 0x7ff8840623c0 2020-11-20T12:07:06.004 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.998+0000 7ff8837fe700 1 -- 172.21.15.189:0/154599707 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff878009cf0 con 0x7ff8840623c0 2020-11-20T12:07:06.005 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.998+0000 7ff889d67700 1 -- 172.21.15.189:0/154599707 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff8840623c0 msgr2=0x7ff8840677f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:06.005 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.998+0000 7ff889d67700 1 --2- 172.21.15.189:0/154599707 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff8840623c0 0x7ff8840677f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7ff878006fd0 tx=0x7ff878006eb0).stop 2020-11-20T12:07:06.005 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.998+0000 7ff889d67700 1 -- 172.21.15.189:0/154599707 shutdown_connections 2020-11-20T12:07:06.006 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.998+0000 7ff889d67700 1 --2- 172.21.15.189:0/154599707 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff8840623c0 0x7ff8840677f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:06.006 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.998+0000 7ff889d67700 1 --2- 172.21.15.189:0/154599707 >> v2:172.21.15.47:3300/0 conn(0x7ff88413aac0 0x7ff88413af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:06.006 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.998+0000 7ff889d67700 1 -- 172.21.15.189:0/154599707 >> 172.21.15.189:0/154599707 conn(0x7ff884065fe0 msgr2=0x7ff884061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:06.006 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.998+0000 7ff889d67700 1 -- 172.21.15.189:0/154599707 shutdown_connections 2020-11-20T12:07:06.006 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.998+0000 7ff889d67700 1 -- 172.21.15.189:0/154599707 wait complete. 2020-11-20T12:07:06.007 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.998+0000 7ff889d67700 1 Processor -- start 2020-11-20T12:07:06.007 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.998+0000 7ff889d67700 1 -- start start 2020-11-20T12:07:06.009 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.998+0000 7ff889d67700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff8840623c0 0x7ff8840a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:06.009 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.998+0000 7ff889d67700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8841340e0 0x7ff88414a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:06.009 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:05.998+0000 7ff889d67700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff88413aac0 0x7ff884129c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:06.009 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.002+0000 7ff889d67700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8840ade00 con 0x7ff8841340e0 2020-11-20T12:07:06.010 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.002+0000 7ff889d67700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff884149c20 con 0x7ff88413aac0 2020-11-20T12:07:06.010 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.002+0000 7ff889d67700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8840ac8b0 con 0x7ff8840623c0 2020-11-20T12:07:06.010 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.002+0000 7ff889566700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff88413aac0 0x7ff884129c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:06.010 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.002+0000 7ff889566700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff88413aac0 0x7ff884129c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49480/0 (socket says 172.21.15.189:49480) 2020-11-20T12:07:06.010 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.002+0000 7ff889566700 1 -- 172.21.15.189:0/1901297844 learned_addr learned my addr 172.21.15.189:0/1901297844 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:07:06.011 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.002+0000 7ff888d65700 1 --2- 172.21.15.189:0/1901297844 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff8840623c0 0x7ff8840a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:06.011 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.002+0000 7ff889566700 1 -- 172.21.15.189:0/1901297844 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff8840623c0 msgr2=0x7ff8840a4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:06.011 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.002+0000 7ff889566700 1 --2- 172.21.15.189:0/1901297844 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff8840623c0 0x7ff8840a4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:06.011 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.002+0000 7ff889566700 1 -- 172.21.15.189:0/1901297844 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8841340e0 msgr2=0x7ff88414a0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2020-11-20T12:07:06.011 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.002+0000 7ff889566700 1 --2- 172.21.15.189:0/1901297844 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8841340e0 0x7ff88414a0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:06.012 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.002+0000 7ff889566700 1 -- 172.21.15.189:0/1901297844 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff884149910 con 0x7ff88413aac0 2020-11-20T12:07:06.012 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.002+0000 7ff889566700 1 --2- 172.21.15.189:0/1901297844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff88413aac0 0x7ff884129c00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7ff87c00ca60 tx=0x7ff87c007f40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:06.012 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.002+0000 7ff880ff9700 1 -- 172.21.15.189:0/1901297844 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c00a710 con 0x7ff88413aac0 2020-11-20T12:07:06.012 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.002+0000 7ff889d67700 1 -- 172.21.15.189:0/1901297844 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8840a6630 con 0x7ff88413aac0 2020-11-20T12:07:06.012 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.002+0000 7ff880ff9700 1 -- 172.21.15.189:0/1901297844 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87c00a870 con 0x7ff88413aac0 2020-11-20T12:07:06.013 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.002+0000 7ff880ff9700 1 -- 172.21.15.189:0/1901297844 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c00abf0 con 0x7ff88413aac0 2020-11-20T12:07:06.013 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.002+0000 7ff889d67700 1 -- 172.21.15.189:0/1901297844 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff88414a7d0 con 0x7ff88413aac0 2020-11-20T12:07:06.013 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.002+0000 7ff889d67700 1 -- 172.21.15.189:0/1901297844 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff884149de0 con 0x7ff88413aac0 2020-11-20T12:07:06.013 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.002+0000 7ff880ff9700 1 -- 172.21.15.189:0/1901297844 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7ff87c019030 con 0x7ff88413aac0 2020-11-20T12:07:06.013 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.002+0000 7ff880ff9700 1 --2- 172.21.15.189:0/1901297844 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff85c0557c0 0x7ff85c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:06.014 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.002+0000 7ff880ff9700 1 -- 172.21.15.189:0/1901297844 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6159+0+0 (secure 0 0 0) 0x7ff87c06de50 con 0x7ff88413aac0 2020-11-20T12:07:06.014 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.002+0000 7ff888d65700 1 --2- 172.21.15.189:0/1901297844 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff85c0557c0 0x7ff85c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:06.015 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.006+0000 7ff888d65700 1 --2- 172.21.15.189:0/1901297844 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff85c0557c0 0x7ff85c057c80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7ff884130790 tx=0x7ff878015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:06.015 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.010+0000 7ff880ff9700 1 -- 172.21.15.189:0/1901297844 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff87c043d50 con 0x7ff88413aac0 2020-11-20T12:07:06.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.254+0000 7ff889d67700 1 -- 172.21.15.189:0/1901297844 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8840663c0 con 0x7ff88413aac0 2020-11-20T12:07:06.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.254+0000 7ff880ff9700 1 -- 172.21.15.189:0/1901297844 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+2139 (secure 0 0 0) 0x7ff87c047370 con 0x7ff88413aac0 2020-11-20T12:07:06.259 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:07:06.260 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4849,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1375880901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1375880901},{"type":"v1","addr":"172.21.15.189:6835","nonce":1375880901}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:06:55.583738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4248},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"b","rank":0,"incarnation":7,"state":"up:active","state_seq":34,"addr":"172.21.15.47:6833/2660417436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2660417436},{"type":"v1","addr":"172.21.15.47:6833","nonce":2660417436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:07:06.261 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.254+0000 7ff889d67700 1 -- 172.21.15.189:0/1901297844 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff85c0557c0 msgr2=0x7ff85c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:06.261 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.258+0000 7ff889d67700 1 --2- 172.21.15.189:0/1901297844 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff85c0557c0 0x7ff85c057c80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7ff884130790 tx=0x7ff878015000).stop 2020-11-20T12:07:06.262 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.258+0000 7ff889d67700 1 -- 172.21.15.189:0/1901297844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff88413aac0 msgr2=0x7ff884129c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:06.262 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.258+0000 7ff889d67700 1 --2- 172.21.15.189:0/1901297844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff88413aac0 0x7ff884129c00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7ff87c00ca60 tx=0x7ff87c007f40).stop 2020-11-20T12:07:06.262 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.258+0000 7ff889d67700 1 -- 172.21.15.189:0/1901297844 shutdown_connections 2020-11-20T12:07:06.262 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.258+0000 7ff889d67700 1 --2- 172.21.15.189:0/1901297844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff88413aac0 0x7ff884129c00 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:06.262 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.258+0000 7ff889d67700 1 --2- 172.21.15.189:0/1901297844 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff8840623c0 0x7ff8840a4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:06.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.258+0000 7ff889d67700 1 --2- 172.21.15.189:0/1901297844 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff85c0557c0 0x7ff85c057c80 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:06.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.258+0000 7ff889d67700 1 --2- 172.21.15.189:0/1901297844 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff8841340e0 0x7ff88414a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:06.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.258+0000 7ff889d67700 1 -- 172.21.15.189:0/1901297844 >> 172.21.15.189:0/1901297844 conn(0x7ff884065fe0 msgr2=0x7ff884130d90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:06.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.258+0000 7ff889d67700 1 -- 172.21.15.189:0/1901297844 shutdown_connections 2020-11-20T12:07:06.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:06.258+0000 7ff889d67700 1 -- 172.21.15.189:0/1901297844 wait complete. 2020-11-20T12:07:06.264 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 11 2020-11-20T12:07:06.273 INFO:tasks.mds_thrash.fs.[None]:mds.a reported in up:standby state 2020-11-20T12:07:06.274 INFO:tasks.mds_thrash.fs.[None]:waiting for 52 secs before thrashing 2020-11-20T12:07:22.505 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:07:22.508 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:07:52.551 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:07:52.556 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:07:58.275 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:07:58.405 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.402+0000 7f9e2f59e700 1 Processor -- start 2020-11-20T12:07:58.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.402+0000 7f9e2f59e700 1 -- start start 2020-11-20T12:07:58.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.402+0000 7f9e2f59e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9e300ad3e0 0x7f9e30066d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:58.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.402+0000 7f9e2f59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e30067730 con 0x7f9e3013a6e0 2020-11-20T12:07:58.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.402+0000 7f9e2f59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e3013dd40 con 0x7f9e300ad000 2020-11-20T12:07:58.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.402+0000 7f9e2f59e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e30066050 con 0x7f9e300ad3e0 2020-11-20T12:07:58.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.402+0000 7f9e2e59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9e300ad000 0x7f9e3013e480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51106/0 (socket says 172.21.15.189:51106) 2020-11-20T12:07:58.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.402+0000 7f9e2e59c700 1 -- 172.21.15.189:0/3747929139 learned_addr learned my addr 172.21.15.189:0/3747929139 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:07:58.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.402+0000 7f9e27fff700 1 --2- 172.21.15.189:0/3747929139 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9e300ad3e0 0x7f9e30066d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:58.409 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.402+0000 7f9e2dd9b700 1 -- 172.21.15.189:0/3747929139 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 521929739 0 0) 0x7f9e3013dd40 con 0x7f9e300ad000 2020-11-20T12:07:58.409 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.402+0000 7f9e2dd9b700 1 -- 172.21.15.189:0/3747929139 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e10003240 con 0x7f9e300ad000 2020-11-20T12:07:58.409 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.402+0000 7f9e2dd9b700 1 -- 172.21.15.189:0/3747929139 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1260767913 0 0) 0x7f9e10003240 con 0x7f9e300ad000 2020-11-20T12:07:58.409 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.402+0000 7f9e27fff700 1 -- 172.21.15.189:0/3747929139 >> v1:172.21.15.189:6789/0 conn(0x7f9e300ad000 legacy=0x7f9e3013e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:58.409 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.402+0000 7f9e27fff700 1 -- 172.21.15.189:0/3747929139 >> v1:172.21.15.47:6789/0 conn(0x7f9e3013a6e0 legacy=0x7f9e3013ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:58.410 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.402+0000 7f9e27fff700 1 -- 172.21.15.189:0/3747929139 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e3013dfe0 con 0x7f9e300ad3e0 2020-11-20T12:07:58.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.402+0000 7f9e27fff700 1 --2- 172.21.15.189:0/3747929139 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9e300ad3e0 0x7f9e30066d40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f9e1800b980 tx=0x7f9e18006df0).ready entity=mon.2 client_cookie=17843917f57022b6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:58.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.406+0000 7f9e2dd9b700 1 -- 172.21.15.189:0/3747929139 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e200014e0 con 0x7f9e300ad3e0 2020-11-20T12:07:58.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.406+0000 7f9e2dd9b700 1 -- 172.21.15.189:0/3747929139 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e18012460 con 0x7f9e300ad3e0 2020-11-20T12:07:58.412 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.406+0000 7f9e2dd9b700 1 -- 172.21.15.189:0/3747929139 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e180127e0 con 0x7f9e300ad3e0 2020-11-20T12:07:58.412 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.406+0000 7f9e2f59e700 1 -- 172.21.15.189:0/3747929139 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9e300ad3e0 msgr2=0x7f9e30066d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:58.412 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.406+0000 7f9e2f59e700 1 --2- 172.21.15.189:0/3747929139 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9e300ad3e0 0x7f9e30066d40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f9e1800b980 tx=0x7f9e18006df0).stop 2020-11-20T12:07:58.412 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.406+0000 7f9e2f59e700 1 -- 172.21.15.189:0/3747929139 shutdown_connections 2020-11-20T12:07:58.413 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.406+0000 7f9e2f59e700 1 --2- 172.21.15.189:0/3747929139 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9e300ad3e0 0x7f9e30066d40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:58.413 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.406+0000 7f9e2f59e700 1 -- 172.21.15.189:0/3747929139 >> 172.21.15.189:0/3747929139 conn(0x7f9e30062080 msgr2=0x7f9e30062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:58.413 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.406+0000 7f9e2f59e700 1 -- 172.21.15.189:0/3747929139 shutdown_connections 2020-11-20T12:07:58.413 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.406+0000 7f9e2f59e700 1 -- 172.21.15.189:0/3747929139 wait complete. 2020-11-20T12:07:58.413 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.406+0000 7f9e2f59e700 1 Processor -- start 2020-11-20T12:07:58.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.406+0000 7f9e2f59e700 1 -- start start 2020-11-20T12:07:58.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.410+0000 7f9e2f59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e300ad000 0x7f9e301409b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:58.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.410+0000 7f9e2f59e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9e300ad3e0 0x7f9e300a97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:58.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.410+0000 7f9e2f59e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9e300a2180 0x7f9e300a9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:58.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.410+0000 7f9e2f59e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e300aa1b0 con 0x7f9e300ad3e0 2020-11-20T12:07:58.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.410+0000 7f9e2f59e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e300aab20 con 0x7f9e300ad000 2020-11-20T12:07:58.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.410+0000 7f9e2f59e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e300aadc0 con 0x7f9e300a2180 2020-11-20T12:07:58.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.410+0000 7f9e2e59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e300ad000 0x7f9e301409b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:58.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.410+0000 7f9e2e59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e300ad000 0x7f9e301409b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49490/0 (socket says 172.21.15.189:49490) 2020-11-20T12:07:58.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.410+0000 7f9e2e59c700 1 -- 172.21.15.189:0/2896979545 learned_addr learned my addr 172.21.15.189:0/2896979545 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:07:58.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.410+0000 7f9e27fff700 1 --2- 172.21.15.189:0/2896979545 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9e300ad3e0 0x7f9e300a97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:58.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.410+0000 7f9e2ed9d700 1 --2- 172.21.15.189:0/2896979545 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9e300a2180 0x7f9e300a9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:58.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.410+0000 7f9e2e59c700 1 -- 172.21.15.189:0/2896979545 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9e300a2180 msgr2=0x7f9e300a9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:58.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.410+0000 7f9e2e59c700 1 --2- 172.21.15.189:0/2896979545 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9e300a2180 0x7f9e300a9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:58.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.410+0000 7f9e2e59c700 1 -- 172.21.15.189:0/2896979545 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9e300ad3e0 msgr2=0x7f9e300a97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:58.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.410+0000 7f9e2e59c700 1 --2- 172.21.15.189:0/2896979545 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9e300ad3e0 0x7f9e300a97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:58.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.410+0000 7f9e2e59c700 1 -- 172.21.15.189:0/2896979545 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e300ab060 con 0x7f9e300ad000 2020-11-20T12:07:58.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.410+0000 7f9e2e59c700 1 --2- 172.21.15.189:0/2896979545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e300ad000 0x7f9e301409b0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f9e2000cae0 tx=0x7f9e20001280).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:58.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.410+0000 7f9e267fc700 1 -- 172.21.15.189:0/2896979545 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e20007170 con 0x7f9e300ad000 2020-11-20T12:07:58.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.410+0000 7f9e267fc700 1 -- 172.21.15.189:0/2896979545 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e200072f0 con 0x7f9e300ad000 2020-11-20T12:07:58.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.410+0000 7f9e2f59e700 1 -- 172.21.15.189:0/2896979545 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e300aa7a0 con 0x7f9e300ad000 2020-11-20T12:07:58.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.410+0000 7f9e267fc700 1 -- 172.21.15.189:0/2896979545 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2000a9c0 con 0x7f9e300ad000 2020-11-20T12:07:58.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.410+0000 7f9e2f59e700 1 -- 172.21.15.189:0/2896979545 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e30136fe0 con 0x7f9e300ad000 2020-11-20T12:07:58.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.410+0000 7f9e2f59e700 1 -- 172.21.15.189:0/2896979545 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e300aa990 con 0x7f9e300ad000 2020-11-20T12:07:58.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.410+0000 7f9e267fc700 1 -- 172.21.15.189:0/2896979545 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f9e2001c030 con 0x7f9e300ad000 2020-11-20T12:07:58.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.414+0000 7f9e267fc700 1 --2- 172.21.15.189:0/2896979545 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9e0c0557c0 0x7f9e0c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:58.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.414+0000 7f9e267fc700 1 -- 172.21.15.189:0/2896979545 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f9e2006e7c0 con 0x7f9e300ad000 2020-11-20T12:07:58.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.414+0000 7f9e27fff700 1 --2- 172.21.15.189:0/2896979545 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9e0c0557c0 0x7f9e0c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:58.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.414+0000 7f9e27fff700 1 --2- 172.21.15.189:0/2896979545 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9e0c0557c0 0x7f9e0c057c80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f9e1800b920 tx=0x7f9e3013e170).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:58.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.418+0000 7f9e267fc700 1 -- 172.21.15.189:0/2896979545 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9e20044610 con 0x7f9e300ad000 2020-11-20T12:07:58.646 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.642+0000 7f9e2f59e700 1 -- 172.21.15.189:0/2896979545 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e300ab1f0 con 0x7f9e300ad000 2020-11-20T12:07:58.647 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.642+0000 7f9e267fc700 1 -- 172.21.15.189:0/2896979545 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+2139 (secure 0 0 0) 0x7f9e300ab1f0 con 0x7f9e300ad000 2020-11-20T12:07:58.648 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:07:58.648 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4849,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1375880901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1375880901},{"type":"v1","addr":"172.21.15.189:6835","nonce":1375880901}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:06:55.583738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4248},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"b","rank":0,"incarnation":7,"state":"up:active","state_seq":34,"addr":"172.21.15.47:6833/2660417436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2660417436},{"type":"v1","addr":"172.21.15.47:6833","nonce":2660417436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:07:58.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.646+0000 7f9e2f59e700 1 -- 172.21.15.189:0/2896979545 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9e0c0557c0 msgr2=0x7f9e0c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:58.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.646+0000 7f9e2f59e700 1 --2- 172.21.15.189:0/2896979545 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9e0c0557c0 0x7f9e0c057c80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f9e1800b920 tx=0x7f9e3013e170).stop 2020-11-20T12:07:58.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.646+0000 7f9e2f59e700 1 -- 172.21.15.189:0/2896979545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e300ad000 msgr2=0x7f9e301409b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:58.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.646+0000 7f9e2f59e700 1 --2- 172.21.15.189:0/2896979545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e300ad000 0x7f9e301409b0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f9e2000cae0 tx=0x7f9e20001280).stop 2020-11-20T12:07:58.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.646+0000 7f9e2f59e700 1 -- 172.21.15.189:0/2896979545 shutdown_connections 2020-11-20T12:07:58.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.646+0000 7f9e2f59e700 1 --2- 172.21.15.189:0/2896979545 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9e300a2180 0x7f9e300a9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:58.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.646+0000 7f9e2f59e700 1 --2- 172.21.15.189:0/2896979545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e300ad000 0x7f9e301409b0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:58.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.646+0000 7f9e2f59e700 1 --2- 172.21.15.189:0/2896979545 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9e0c0557c0 0x7f9e0c057c80 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:58.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.646+0000 7f9e2f59e700 1 --2- 172.21.15.189:0/2896979545 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9e300ad3e0 0x7f9e300a97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:58.652 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.646+0000 7f9e2f59e700 1 -- 172.21.15.189:0/2896979545 >> 172.21.15.189:0/2896979545 conn(0x7f9e30062080 msgr2=0x7f9e3013aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:58.652 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.646+0000 7f9e2f59e700 1 -- 172.21.15.189:0/2896979545 shutdown_connections 2020-11-20T12:07:58.652 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.646+0000 7f9e2f59e700 1 -- 172.21.15.189:0/2896979545 wait complete. 2020-11-20T12:07:58.652 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 11 2020-11-20T12:07:58.664 INFO:tasks.mds_thrash.fs.[None]:kill mds.b (rank=0) 2020-11-20T12:07:58.664 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:07:58.664 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:07:58.681 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:07:58.682 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:07:58.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.794+0000 7f9a93fff700 1 Processor -- start 2020-11-20T12:07:58.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a93fff700 1 -- start start 2020-11-20T12:07:58.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a93fff700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f9a9413dc20 0x7f9a94061180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:58.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a93fff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9a940a9cc0 0x7f9a940aa150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:58.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a93fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a94066ee0 con 0x7f9a940a0a90 2020-11-20T12:07:58.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a93fff700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a940a96e0 con 0x7f9a9413dc20 2020-11-20T12:07:58.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a93fff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a94067180 con 0x7f9a940a9cc0 2020-11-20T12:07:58.806 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a92ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9a940a0a90 0x7f9a940a8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51122/0 (socket says 172.21.15.189:51122) 2020-11-20T12:07:58.806 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a92ffd700 1 -- 172.21.15.189:0/218005189 learned_addr learned my addr 172.21.15.189:0/218005189 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:07:58.806 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a937fe700 1 --2- 172.21.15.189:0/218005189 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9a940a9cc0 0x7f9a940aa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:58.806 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a927fc700 1 --2- 172.21.15.189:0/218005189 >> v2:172.21.15.47:3300/0 conn(0x7f9a9413dc20 0x7f9a94061180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:58.807 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a91ffb700 1 -- 172.21.15.189:0/218005189 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759966093 0 0) 0x7f9a94066ee0 con 0x7f9a940a0a90 2020-11-20T12:07:58.807 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a91ffb700 1 -- 172.21.15.189:0/218005189 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a80003240 con 0x7f9a940a0a90 2020-11-20T12:07:58.807 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a91ffb700 1 -- 172.21.15.189:0/218005189 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2702697950 0 0) 0x7f9a80003240 con 0x7f9a940a0a90 2020-11-20T12:07:58.807 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a91ffb700 1 -- 172.21.15.189:0/218005189 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9a940a9cc0 msgr2=0x7f9a940aa150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:58.807 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a91ffb700 1 --2- 172.21.15.189:0/218005189 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9a940a9cc0 0x7f9a940aa150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:58.808 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a91ffb700 1 -- 172.21.15.189:0/218005189 >> v2:172.21.15.47:3300/0 conn(0x7f9a9413dc20 msgr2=0x7f9a94061180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:58.808 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a91ffb700 1 --2- 172.21.15.189:0/218005189 >> v2:172.21.15.47:3300/0 conn(0x7f9a9413dc20 0x7f9a94061180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:58.808 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a91ffb700 1 -- 172.21.15.189:0/218005189 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a940a0e70 con 0x7f9a940a0a90 2020-11-20T12:07:58.808 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a91ffb700 1 -- 172.21.15.189:0/218005189 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f9a840014e0 con 0x7f9a940a0a90 2020-11-20T12:07:58.808 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a937fe700 1 --2- 172.21.15.189:0/218005189 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9a940a9cc0 0x7f9a940aa150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:07:58.809 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a91ffb700 1 -- 172.21.15.189:0/218005189 >> v1:172.21.15.189:6789/0 conn(0x7f9a940a0a90 legacy=0x7f9a940a8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:58.809 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a91ffb700 1 --2- 172.21.15.189:0/218005189 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a80003730 0x7f9a80003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:58.809 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a91ffb700 1 --2- 172.21.15.189:0/218005189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a800044e0 0x7f9a80004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:58.809 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a91ffb700 1 --2- 172.21.15.189:0/218005189 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9a940a9cc0 0x7f9a80004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:58.809 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a91ffb700 1 -- 172.21.15.189:0/218005189 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a940a96e0 con 0x7f9a80003730 2020-11-20T12:07:58.810 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a91ffb700 1 -- 172.21.15.189:0/218005189 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a94067180 con 0x7f9a800044e0 2020-11-20T12:07:58.810 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a91ffb700 1 -- 172.21.15.189:0/218005189 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a80004230 con 0x7f9a940a9cc0 2020-11-20T12:07:58.810 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a927fc700 1 --2- 172.21.15.189:0/218005189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a800044e0 0x7f9a80004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:58.810 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a937fe700 1 --2- 172.21.15.189:0/218005189 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9a940a9cc0 0x7f9a80004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:58.810 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a927fc700 1 -- 172.21.15.189:0/218005189 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9a940a9cc0 msgr2=0x7f9a80004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:58.811 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a927fc700 1 --2- 172.21.15.189:0/218005189 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9a940a9cc0 0x7f9a80004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:58.811 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a927fc700 1 -- 172.21.15.189:0/218005189 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a80003730 msgr2=0x7f9a80003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:07:58.811 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a927fc700 1 --2- 172.21.15.189:0/218005189 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a80003730 0x7f9a80003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:07:58.811 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a927fc700 1 -- 172.21.15.189:0/218005189 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a80003240 con 0x7f9a800044e0 2020-11-20T12:07:58.811 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a937fe700 1 --2- 172.21.15.189:0/218005189 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9a940a9cc0 0x7f9a80004e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:07:58.812 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a927fc700 1 --2- 172.21.15.189:0/218005189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a800044e0 0x7f9a80004960 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f9a88006fd0 tx=0x7f9a88006df0).ready entity=mon.1 client_cookie=18eab84cd0290878 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:58.812 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a91ffb700 1 -- 172.21.15.189:0/218005189 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8800b310 con 0x7f9a800044e0 2020-11-20T12:07:58.812 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a91ffb700 1 -- 172.21.15.189:0/218005189 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a8800eae0 con 0x7f9a800044e0 2020-11-20T12:07:58.812 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.798+0000 7f9a91ffb700 1 -- 172.21.15.189:0/218005189 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8800ee60 con 0x7f9a800044e0 2020-11-20T12:07:58.812 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a93fff700 1 -- 172.21.15.189:0/218005189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a800044e0 msgr2=0x7f9a80004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:58.813 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a93fff700 1 --2- 172.21.15.189:0/218005189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a800044e0 0x7f9a80004960 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f9a88006fd0 tx=0x7f9a88006df0).stop 2020-11-20T12:07:58.813 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a937fe700 1 -- 172.21.15.189:0/218005189 reap_dead start 2020-11-20T12:07:58.813 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a93fff700 1 -- 172.21.15.189:0/218005189 shutdown_connections 2020-11-20T12:07:58.813 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a93fff700 1 -- 172.21.15.189:0/218005189 >> 172.21.15.189:0/218005189 conn(0x7f9a94063210 msgr2=0x7f9a94060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:58.814 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a93fff700 1 -- 172.21.15.189:0/218005189 shutdown_connections 2020-11-20T12:07:58.814 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a93fff700 1 -- 172.21.15.189:0/218005189 wait complete. 2020-11-20T12:07:58.816 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a93fff700 1 Processor -- start 2020-11-20T12:07:58.816 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a93fff700 1 -- start start 2020-11-20T12:07:58.817 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a93fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a94141140 0x7f9a94149c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:58.817 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a93fff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9a94130890 0x7f9a9412f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:58.817 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a93fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a94130d10 0x7f9a9412f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:58.817 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a93fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9413cc00 con 0x7f9a94141140 2020-11-20T12:07:58.818 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a93fff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a941419a0 con 0x7f9a94130d10 2020-11-20T12:07:58.818 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a93fff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9413cdf0 con 0x7f9a94130890 2020-11-20T12:07:58.818 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a937fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a94130d10 0x7f9a9412f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:58.818 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a937fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a94130d10 0x7f9a9412f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49516/0 (socket says 172.21.15.189:49516) 2020-11-20T12:07:58.818 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a937fe700 1 -- 172.21.15.189:0/2687929969 learned_addr learned my addr 172.21.15.189:0/2687929969 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:07:58.819 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a927fc700 1 --2- 172.21.15.189:0/2687929969 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9a94130890 0x7f9a9412f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:58.819 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a92ffd700 1 --2- 172.21.15.189:0/2687929969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a94141140 0x7f9a94149c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:58.819 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a937fe700 1 -- 172.21.15.189:0/2687929969 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9a94130890 msgr2=0x7f9a9412f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:58.819 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a937fe700 1 --2- 172.21.15.189:0/2687929969 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9a94130890 0x7f9a9412f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:58.820 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a937fe700 1 -- 172.21.15.189:0/2687929969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a94141140 msgr2=0x7f9a94149c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:58.820 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a937fe700 1 --2- 172.21.15.189:0/2687929969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a94141140 0x7f9a94149c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:58.820 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a937fe700 1 -- 172.21.15.189:0/2687929969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a9413d090 con 0x7f9a94130d10 2020-11-20T12:07:58.820 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a937fe700 1 --2- 172.21.15.189:0/2687929969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a94130d10 0x7f9a9412f990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f9a9413e000 tx=0x7f9a8c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:58.820 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a7affd700 1 -- 172.21.15.189:0/2687929969 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c018070 con 0x7f9a94130d10 2020-11-20T12:07:58.821 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a7affd700 1 -- 172.21.15.189:0/2687929969 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a8c011660 con 0x7f9a94130d10 2020-11-20T12:07:58.821 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.802+0000 7f9a7affd700 1 -- 172.21.15.189:0/2687929969 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c011a00 con 0x7f9a94130d10 2020-11-20T12:07:58.821 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.806+0000 7f9a93fff700 1 -- 172.21.15.189:0/2687929969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a9413d330 con 0x7f9a94130d10 2020-11-20T12:07:58.821 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.806+0000 7f9a93fff700 1 -- 172.21.15.189:0/2687929969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a9412fe80 con 0x7f9a94130d10 2020-11-20T12:07:58.821 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.806+0000 7f9a93fff700 1 -- 172.21.15.189:0/2687929969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a9413d4c0 con 0x7f9a94130d10 2020-11-20T12:07:58.822 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.806+0000 7f9a7affd700 1 -- 172.21.15.189:0/2687929969 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f9a8c01d410 con 0x7f9a94130d10 2020-11-20T12:07:58.822 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.806+0000 7f9a7affd700 1 --2- 172.21.15.189:0/2687929969 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9a700557c0 0x7f9a70057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:07:58.822 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.806+0000 7f9a7affd700 1 -- 172.21.15.189:0/2687929969 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f9a8c06e510 con 0x7f9a94130d10 2020-11-20T12:07:58.822 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.806+0000 7f9a92ffd700 1 --2- 172.21.15.189:0/2687929969 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9a700557c0 0x7f9a70057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:07:58.822 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.806+0000 7f9a92ffd700 1 --2- 172.21.15.189:0/2687929969 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9a700557c0 0x7f9a70057c80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f9a84004950 tx=0x7f9a94067070).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:07:58.823 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:58.810+0000 7f9a7affd700 1 -- 172.21.15.189:0/2687929969 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9a8c044300 con 0x7f9a94130d10 2020-11-20T12:07:59.040 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:59.038+0000 7f9a93fff700 1 -- 172.21.15.189:0/2687929969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a94066af0 con 0x7f9a94130d10 2020-11-20T12:07:59.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:59.038+0000 7f9a7affd700 1 -- 172.21.15.189:0/2687929969 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+2139 (secure 0 0 0) 0x7f9a8c047920 con 0x7f9a94130d10 2020-11-20T12:07:59.042 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:07:59.042 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4849,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1375880901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1375880901},{"type":"v1","addr":"172.21.15.189:6835","nonce":1375880901}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:06:55.583738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4248},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"b","rank":0,"incarnation":7,"state":"up:active","state_seq":34,"addr":"172.21.15.47:6833/2660417436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2660417436},{"type":"v1","addr":"172.21.15.47:6833","nonce":2660417436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:07:59.044 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:59.038+0000 7f9a93fff700 1 -- 172.21.15.189:0/2687929969 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9a700557c0 msgr2=0x7f9a70057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:59.044 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:59.038+0000 7f9a93fff700 1 --2- 172.21.15.189:0/2687929969 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9a700557c0 0x7f9a70057c80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f9a84004950 tx=0x7f9a94067070).stop 2020-11-20T12:07:59.044 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:59.038+0000 7f9a93fff700 1 -- 172.21.15.189:0/2687929969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a94130d10 msgr2=0x7f9a9412f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:07:59.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:59.038+0000 7f9a93fff700 1 --2- 172.21.15.189:0/2687929969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a94130d10 0x7f9a9412f990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f9a9413e000 tx=0x7f9a8c008040).stop 2020-11-20T12:07:59.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:59.038+0000 7f9a93fff700 1 -- 172.21.15.189:0/2687929969 shutdown_connections 2020-11-20T12:07:59.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:59.038+0000 7f9a93fff700 1 --2- 172.21.15.189:0/2687929969 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9a700557c0 0x7f9a70057c80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:59.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:59.038+0000 7f9a93fff700 1 --2- 172.21.15.189:0/2687929969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9a94141140 0x7f9a94149c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:59.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:59.038+0000 7f9a93fff700 1 --2- 172.21.15.189:0/2687929969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a94130d10 0x7f9a9412f990 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:59.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:59.038+0000 7f9a93fff700 1 --2- 172.21.15.189:0/2687929969 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9a94130890 0x7f9a9412f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:07:59.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:59.038+0000 7f9a93fff700 1 -- 172.21.15.189:0/2687929969 >> 172.21.15.189:0/2687929969 conn(0x7f9a94063210 msgr2=0x7f9a941385c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:07:59.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:59.042+0000 7f9a93fff700 1 -- 172.21.15.189:0/2687929969 shutdown_connections 2020-11-20T12:07:59.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:07:59.042+0000 7f9a93fff700 1 -- 172.21.15.189:0/2687929969 wait complete. 2020-11-20T12:07:59.047 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 11 2020-11-20T12:07:59.063 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:08:01.064 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:08:01.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.194+0000 7f9954c39700 1 Processor -- start 2020-11-20T12:08:01.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.194+0000 7f9954c39700 1 -- start start 2020-11-20T12:08:01.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.198+0000 7f9954c39700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f9950061040 0x7f99501354f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:01.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.198+0000 7f9954c39700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9950065ca0 con 0x7f9950060bc0 2020-11-20T12:08:01.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.198+0000 7f9954c39700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9950138ba0 con 0x7f99500623c0 2020-11-20T12:08:01.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.198+0000 7f9954c39700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99500687e0 con 0x7f9950061040 2020-11-20T12:08:01.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.198+0000 7f994f7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f99500623c0 0x7f9950138290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51148/0 (socket says 172.21.15.189:51148) 2020-11-20T12:08:01.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.198+0000 7f994f7fe700 1 -- 172.21.15.189:0/2198346038 learned_addr learned my addr 172.21.15.189:0/2198346038 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:08:01.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.198+0000 7f994ffff700 1 --2- 172.21.15.189:0/2198346038 >> v2:172.21.15.47:3300/0 conn(0x7f9950061040 0x7f99501354f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:01.203 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.198+0000 7f994e7fc700 1 -- 172.21.15.189:0/2198346038 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3631115852 0 0) 0x7f9950138ba0 con 0x7f99500623c0 2020-11-20T12:08:01.203 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.198+0000 7f994e7fc700 1 -- 172.21.15.189:0/2198346038 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f993c003240 con 0x7f99500623c0 2020-11-20T12:08:01.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.198+0000 7f994e7fc700 1 -- 172.21.15.189:0/2198346038 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3362673122 0 0) 0x7f9950065ca0 con 0x7f9950060bc0 2020-11-20T12:08:01.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.198+0000 7f994e7fc700 1 -- 172.21.15.189:0/2198346038 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9950138ba0 con 0x7f9950060bc0 2020-11-20T12:08:01.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.198+0000 7f994e7fc700 1 -- 172.21.15.189:0/2198346038 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3466578862 0 0) 0x7f993c003240 con 0x7f99500623c0 2020-11-20T12:08:01.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.198+0000 7f994ffff700 1 -- 172.21.15.189:0/2198346038 >> v1:172.21.15.189:6789/0 conn(0x7f99500623c0 legacy=0x7f9950138290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:01.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.198+0000 7f994ffff700 1 -- 172.21.15.189:0/2198346038 >> v1:172.21.15.47:6789/0 conn(0x7f9950060bc0 legacy=0x7f9950134de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:01.205 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.198+0000 7f994ffff700 1 -- 172.21.15.189:0/2198346038 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9950138e40 con 0x7f9950061040 2020-11-20T12:08:01.205 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.198+0000 7f994ffff700 1 --2- 172.21.15.189:0/2198346038 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9950061040 0x7f99501354f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f9948009c10 tx=0x7f99480091d0).ready entity=mon.0 client_cookie=3ab0d43ff3bada00 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:01.205 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.198+0000 7f994e7fc700 1 -- 172.21.15.189:0/2198346038 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9940001520 con 0x7f9950061040 2020-11-20T12:08:01.205 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.198+0000 7f994e7fc700 1 -- 172.21.15.189:0/2198346038 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f994800b7c0 con 0x7f9950061040 2020-11-20T12:08:01.206 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.198+0000 7f994e7fc700 1 -- 172.21.15.189:0/2198346038 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994800bb40 con 0x7f9950061040 2020-11-20T12:08:01.206 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.198+0000 7f9954c39700 1 -- 172.21.15.189:0/2198346038 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9950061040 msgr2=0x7f99501354f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:01.206 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.198+0000 7f9954c39700 1 --2- 172.21.15.189:0/2198346038 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9950061040 0x7f99501354f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f9948009c10 tx=0x7f99480091d0).stop 2020-11-20T12:08:01.206 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.198+0000 7f9954c39700 1 -- 172.21.15.189:0/2198346038 shutdown_connections 2020-11-20T12:08:01.206 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.198+0000 7f9954c39700 1 --2- 172.21.15.189:0/2198346038 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9950061040 0x7f99501354f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:01.206 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.198+0000 7f9954c39700 1 -- 172.21.15.189:0/2198346038 >> 172.21.15.189:0/2198346038 conn(0x7f9950065fe0 msgr2=0x7f9950061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:01.207 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.198+0000 7f9954c39700 1 -- 172.21.15.189:0/2198346038 shutdown_connections 2020-11-20T12:08:01.207 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.198+0000 7f9954c39700 1 -- 172.21.15.189:0/2198346038 wait complete. 2020-11-20T12:08:01.209 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.202+0000 7f9954c39700 1 Processor -- start 2020-11-20T12:08:01.209 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.202+0000 7f9954c39700 1 -- start start 2020-11-20T12:08:01.210 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.202+0000 7f9954c39700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9950060bc0 0x7f99500a8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:01.210 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.202+0000 7f9954c39700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9950061040 0x7f995013bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:01.210 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.202+0000 7f9954c39700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99500623c0 0x7f995013c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:01.210 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.202+0000 7f9954c39700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99500aa240 con 0x7f9950060bc0 2020-11-20T12:08:01.210 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.202+0000 7f9954c39700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99500a8f00 con 0x7f99500623c0 2020-11-20T12:08:01.211 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.202+0000 7f9954c39700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99500aa430 con 0x7f9950061040 2020-11-20T12:08:01.211 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.202+0000 7f994ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99500623c0 0x7f995013c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:01.211 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.202+0000 7f994ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99500623c0 0x7f995013c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49536/0 (socket says 172.21.15.189:49536) 2020-11-20T12:08:01.211 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.202+0000 7f994ffff700 1 -- 172.21.15.189:0/3781053530 learned_addr learned my addr 172.21.15.189:0/3781053530 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:08:01.211 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.202+0000 7f994f7fe700 1 --2- 172.21.15.189:0/3781053530 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9950060bc0 0x7f99500a8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:01.212 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.202+0000 7f994effd700 1 --2- 172.21.15.189:0/3781053530 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9950061040 0x7f995013bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:01.212 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.202+0000 7f994ffff700 1 -- 172.21.15.189:0/3781053530 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9950061040 msgr2=0x7f995013bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:01.212 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.202+0000 7f994ffff700 1 --2- 172.21.15.189:0/3781053530 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9950061040 0x7f995013bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:01.212 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.202+0000 7f994ffff700 1 -- 172.21.15.189:0/3781053530 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9950060bc0 msgr2=0x7f99500a8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:01.212 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.202+0000 7f994ffff700 1 --2- 172.21.15.189:0/3781053530 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9950060bc0 0x7f99500a8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:01.213 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.202+0000 7f994ffff700 1 -- 172.21.15.189:0/3781053530 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99500aa6d0 con 0x7f99500623c0 2020-11-20T12:08:01.213 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.202+0000 7f994ffff700 1 --2- 172.21.15.189:0/3781053530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99500623c0 0x7f995013c070 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f99500ab3f0 tx=0x7f9948016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:01.213 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.202+0000 7f992f7fe700 1 -- 172.21.15.189:0/3781053530 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9948011e60 con 0x7f99500623c0 2020-11-20T12:08:01.213 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.202+0000 7f992f7fe700 1 -- 172.21.15.189:0/3781053530 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f994801d660 con 0x7f99500623c0 2020-11-20T12:08:01.214 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.202+0000 7f9954c39700 1 -- 172.21.15.189:0/3781053530 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99500a4970 con 0x7f99500623c0 2020-11-20T12:08:01.214 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.202+0000 7f992f7fe700 1 -- 172.21.15.189:0/3781053530 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994801d9e0 con 0x7f99500623c0 2020-11-20T12:08:01.214 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.202+0000 7f9954c39700 1 -- 172.21.15.189:0/3781053530 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f995013b160 con 0x7f99500623c0 2020-11-20T12:08:01.214 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.206+0000 7f992f7fe700 1 -- 172.21.15.189:0/3781053530 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f994802d070 con 0x7f99500623c0 2020-11-20T12:08:01.214 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.206+0000 7f9954c39700 1 -- 172.21.15.189:0/3781053530 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99500aa860 con 0x7f99500623c0 2020-11-20T12:08:01.215 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.206+0000 7f992f7fe700 1 --2- 172.21.15.189:0/3781053530 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f99280557c0 0x7f9928057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:01.215 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.206+0000 7f992f7fe700 1 -- 172.21.15.189:0/3781053530 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f9948070c20 con 0x7f99500623c0 2020-11-20T12:08:01.215 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.206+0000 7f994f7fe700 1 --2- 172.21.15.189:0/3781053530 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f99280557c0 0x7f9928057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:01.215 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.206+0000 7f994f7fe700 1 --2- 172.21.15.189:0/3781053530 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f99280557c0 0x7f9928057c80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f995013c870 tx=0x7f9940000b80).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:01.216 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.210+0000 7f992f7fe700 1 -- 172.21.15.189:0/3781053530 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9948046af0 con 0x7f99500623c0 2020-11-20T12:08:01.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.422+0000 7f9954c39700 1 -- 172.21.15.189:0/3781053530 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99500663c0 con 0x7f99500623c0 2020-11-20T12:08:01.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.426+0000 7f992f7fe700 1 -- 172.21.15.189:0/3781053530 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+2139 (secure 0 0 0) 0x7f9948021090 con 0x7f99500623c0 2020-11-20T12:08:01.428 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:08:01.428 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4849,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1375880901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1375880901},{"type":"v1","addr":"172.21.15.189:6835","nonce":1375880901}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:06:55.583738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4248},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"b","rank":0,"incarnation":7,"state":"up:active","state_seq":34,"addr":"172.21.15.47:6833/2660417436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2660417436},{"type":"v1","addr":"172.21.15.47:6833","nonce":2660417436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:08:01.430 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.426+0000 7f9954c39700 1 -- 172.21.15.189:0/3781053530 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f99280557c0 msgr2=0x7f9928057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:01.430 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.426+0000 7f9954c39700 1 --2- 172.21.15.189:0/3781053530 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f99280557c0 0x7f9928057c80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f995013c870 tx=0x7f9940000b80).stop 2020-11-20T12:08:01.430 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.426+0000 7f9954c39700 1 -- 172.21.15.189:0/3781053530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99500623c0 msgr2=0x7f995013c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:01.430 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.426+0000 7f9954c39700 1 --2- 172.21.15.189:0/3781053530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99500623c0 0x7f995013c070 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f99500ab3f0 tx=0x7f9948016000).stop 2020-11-20T12:08:01.431 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.426+0000 7f9954c39700 1 -- 172.21.15.189:0/3781053530 shutdown_connections 2020-11-20T12:08:01.431 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.426+0000 7f9954c39700 1 --2- 172.21.15.189:0/3781053530 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f99280557c0 0x7f9928057c80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:01.431 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.426+0000 7f9954c39700 1 --2- 172.21.15.189:0/3781053530 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9950060bc0 0x7f99500a8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:01.431 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.426+0000 7f9954c39700 1 --2- 172.21.15.189:0/3781053530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99500623c0 0x7f995013c070 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:01.431 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.426+0000 7f9954c39700 1 --2- 172.21.15.189:0/3781053530 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9950061040 0x7f995013bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:01.432 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.426+0000 7f9954c39700 1 -- 172.21.15.189:0/3781053530 >> 172.21.15.189:0/3781053530 conn(0x7f9950065fe0 msgr2=0x7f9950061420 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:01.432 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.426+0000 7f9954c39700 1 -- 172.21.15.189:0/3781053530 shutdown_connections 2020-11-20T12:08:01.432 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:01.426+0000 7f9954c39700 1 -- 172.21.15.189:0/3781053530 wait complete. 2020-11-20T12:08:01.432 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 11 2020-11-20T12:08:01.442 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:08:03.443 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:08:03.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.574+0000 7f5e08ee5700 1 Processor -- start 2020-11-20T12:08:03.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.574+0000 7f5e08ee5700 1 -- start start 2020-11-20T12:08:03.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.574+0000 7f5e08ee5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5e0413c370 0x7f5e0414a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:03.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.574+0000 7f5e08ee5700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e040a0eb0 con 0x7f5e040a8b00 2020-11-20T12:08:03.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.574+0000 7f5e08ee5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e040675a0 con 0x7f5e04060310 2020-11-20T12:08:03.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.574+0000 7f5e08ee5700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e040a9910 con 0x7f5e0413c370 2020-11-20T12:08:03.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.574+0000 7f5e03fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5e0413c370 0x7f5e0414a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:03.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.574+0000 7f5e03fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5e0413c370 0x7f5e0414a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49546/0 (socket says 172.21.15.189:49546) 2020-11-20T12:08:03.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.574+0000 7f5e03fff700 1 -- 172.21.15.189:0/289667697 learned_addr learned my addr 172.21.15.189:0/289667697 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:08:03.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.574+0000 7f5e03fff700 1 -- 172.21.15.189:0/289667697 >> v1:172.21.15.189:6789/0 conn(0x7f5e04060310 legacy=0x7f5e040a9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:03.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.574+0000 7f5e03fff700 1 -- 172.21.15.189:0/289667697 >> v1:172.21.15.47:6789/0 conn(0x7f5e040a8b00 legacy=0x7f5e0413bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:03.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.574+0000 7f5e03fff700 1 -- 172.21.15.189:0/289667697 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e04067840 con 0x7f5e0413c370 2020-11-20T12:08:03.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e03fff700 1 --2- 172.21.15.189:0/289667697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e0413c370 0x7f5e0414a450 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f5dfc009d50 tx=0x7f5dfc009290).ready entity=mon.1 client_cookie=8206deed1a12086d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:03.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e027fc700 1 -- 172.21.15.189:0/289667697 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc00bc80 con 0x7f5e0413c370 2020-11-20T12:08:03.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e027fc700 1 -- 172.21.15.189:0/289667697 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dfc00bde0 con 0x7f5e0413c370 2020-11-20T12:08:03.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e027fc700 1 -- 172.21.15.189:0/289667697 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc011610 con 0x7f5e0413c370 2020-11-20T12:08:03.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e08ee5700 1 -- 172.21.15.189:0/289667697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e0413c370 msgr2=0x7f5e0414a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:03.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e08ee5700 1 --2- 172.21.15.189:0/289667697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e0413c370 0x7f5e0414a450 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f5dfc009d50 tx=0x7f5dfc009290).stop 2020-11-20T12:08:03.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e08ee5700 1 -- 172.21.15.189:0/289667697 shutdown_connections 2020-11-20T12:08:03.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e08ee5700 1 --2- 172.21.15.189:0/289667697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e0413c370 0x7f5e0414a450 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:03.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e08ee5700 1 -- 172.21.15.189:0/289667697 >> 172.21.15.189:0/289667697 conn(0x7f5e04063210 msgr2=0x7f5e04060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:03.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e08ee5700 1 -- 172.21.15.189:0/289667697 shutdown_connections 2020-11-20T12:08:03.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e08ee5700 1 -- 172.21.15.189:0/289667697 wait complete. 2020-11-20T12:08:03.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e08ee5700 1 Processor -- start 2020-11-20T12:08:03.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e08ee5700 1 -- start start 2020-11-20T12:08:03.589 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e08ee5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e04060310 0x7f5e040a1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:03.589 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e08ee5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e040a8b00 0x7f5e0414e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:03.589 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e08ee5700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5e0413c370 0x7f5e041415f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:03.589 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e08ee5700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0414e730 con 0x7f5e040a8b00 2020-11-20T12:08:03.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e08ee5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0414e9c0 con 0x7f5e04060310 2020-11-20T12:08:03.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e08ee5700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e04141150 con 0x7f5e0413c370 2020-11-20T12:08:03.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e037fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e04060310 0x7f5e040a1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:03.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e037fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e04060310 0x7f5e040a1370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49548/0 (socket says 172.21.15.189:49548) 2020-11-20T12:08:03.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e037fe700 1 -- 172.21.15.189:0/2693722514 learned_addr learned my addr 172.21.15.189:0/2693722514 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:08:03.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e03fff700 1 --2- 172.21.15.189:0/2693722514 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5e0413c370 0x7f5e041415f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:03.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e02ffd700 1 --2- 172.21.15.189:0/2693722514 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e040a8b00 0x7f5e0414e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:03.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e037fe700 1 -- 172.21.15.189:0/2693722514 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5e0413c370 msgr2=0x7f5e041415f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:03.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e037fe700 1 --2- 172.21.15.189:0/2693722514 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5e0413c370 0x7f5e041415f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:03.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e037fe700 1 -- 172.21.15.189:0/2693722514 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e040a8b00 msgr2=0x7f5e0414e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:03.592 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e037fe700 1 --2- 172.21.15.189:0/2693722514 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e040a8b00 0x7f5e0414e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:03.592 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e037fe700 1 -- 172.21.15.189:0/2693722514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e04142830 con 0x7f5e04060310 2020-11-20T12:08:03.592 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.578+0000 7f5e037fe700 1 --2- 172.21.15.189:0/2693722514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e04060310 0x7f5e040a1370 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f5df4001e70 tx=0x7f5df4007c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:03.592 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.582+0000 7f5dfb7fe700 1 -- 172.21.15.189:0/2693722514 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df4007020 con 0x7f5e04060310 2020-11-20T12:08:03.592 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.582+0000 7f5e08ee5700 1 -- 172.21.15.189:0/2693722514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e04142a80 con 0x7f5e04060310 2020-11-20T12:08:03.593 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.582+0000 7f5dfb7fe700 1 -- 172.21.15.189:0/2693722514 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5df4007180 con 0x7f5e04060310 2020-11-20T12:08:03.593 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.582+0000 7f5dfb7fe700 1 -- 172.21.15.189:0/2693722514 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df40064f0 con 0x7f5e04060310 2020-11-20T12:08:03.593 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.582+0000 7f5e08ee5700 1 -- 172.21.15.189:0/2693722514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e04141c80 con 0x7f5e04060310 2020-11-20T12:08:03.593 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.582+0000 7f5e08ee5700 1 -- 172.21.15.189:0/2693722514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e040446d0 con 0x7f5e04060310 2020-11-20T12:08:03.593 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.582+0000 7f5dfb7fe700 1 -- 172.21.15.189:0/2693722514 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f5df4010070 con 0x7f5e04060310 2020-11-20T12:08:03.594 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.582+0000 7f5dfb7fe700 1 --2- 172.21.15.189:0/2693722514 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5de00557c0 0x7f5de0057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:03.594 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.582+0000 7f5dfb7fe700 1 -- 172.21.15.189:0/2693722514 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f5df406da10 con 0x7f5e04060310 2020-11-20T12:08:03.594 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.582+0000 7f5e02ffd700 1 --2- 172.21.15.189:0/2693722514 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5de00557c0 0x7f5de0057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:03.594 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.582+0000 7f5e02ffd700 1 --2- 172.21.15.189:0/2693722514 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5de00557c0 0x7f5de0057c80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f5e0414b8a0 tx=0x7f5dec007b90).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:03.595 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.586+0000 7f5dfb7fe700 1 -- 172.21.15.189:0/2693722514 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5df4042fd0 con 0x7f5e04060310 2020-11-20T12:08:03.790 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.786+0000 7f5e08ee5700 1 -- 172.21.15.189:0/2693722514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e0405cdd0 con 0x7f5e04060310 2020-11-20T12:08:03.791 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.786+0000 7f5dfb7fe700 1 -- 172.21.15.189:0/2693722514 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+2139 (secure 0 0 0) 0x7f5df4043860 con 0x7f5e04060310 2020-11-20T12:08:03.791 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:08:03.791 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4849,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1375880901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1375880901},{"type":"v1","addr":"172.21.15.189:6835","nonce":1375880901}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:06:55.583738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4248},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"b","rank":0,"incarnation":7,"state":"up:active","state_seq":34,"addr":"172.21.15.47:6833/2660417436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2660417436},{"type":"v1","addr":"172.21.15.47:6833","nonce":2660417436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:08:03.793 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.790+0000 7f5e08ee5700 1 -- 172.21.15.189:0/2693722514 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5de00557c0 msgr2=0x7f5de0057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:03.793 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.790+0000 7f5e08ee5700 1 --2- 172.21.15.189:0/2693722514 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5de00557c0 0x7f5de0057c80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f5e0414b8a0 tx=0x7f5dec007b90).stop 2020-11-20T12:08:03.793 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.790+0000 7f5e08ee5700 1 -- 172.21.15.189:0/2693722514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e04060310 msgr2=0x7f5e040a1370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:03.793 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.790+0000 7f5e08ee5700 1 --2- 172.21.15.189:0/2693722514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e04060310 0x7f5e040a1370 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f5df4001e70 tx=0x7f5df4007c50).stop 2020-11-20T12:08:03.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.790+0000 7f5e08ee5700 1 -- 172.21.15.189:0/2693722514 shutdown_connections 2020-11-20T12:08:03.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.790+0000 7f5e08ee5700 1 --2- 172.21.15.189:0/2693722514 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5e0413c370 0x7f5e041415f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:03.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.790+0000 7f5e08ee5700 1 --2- 172.21.15.189:0/2693722514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e04060310 0x7f5e040a1370 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:03.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.790+0000 7f5e08ee5700 1 --2- 172.21.15.189:0/2693722514 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5de00557c0 0x7f5de0057c80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:03.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.790+0000 7f5e08ee5700 1 --2- 172.21.15.189:0/2693722514 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5e040a8b00 0x7f5e0414e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:03.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.790+0000 7f5e08ee5700 1 -- 172.21.15.189:0/2693722514 >> 172.21.15.189:0/2693722514 conn(0x7f5e04063210 msgr2=0x7f5e040acb40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:03.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.790+0000 7f5e08ee5700 1 -- 172.21.15.189:0/2693722514 shutdown_connections 2020-11-20T12:08:03.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:03.790+0000 7f5e08ee5700 1 -- 172.21.15.189:0/2693722514 wait complete. 2020-11-20T12:08:03.796 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 11 2020-11-20T12:08:03.806 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:08:05.807 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:08:05.938 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.934+0000 7fb551adf700 1 Processor -- start 2020-11-20T12:08:05.938 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.934+0000 7fb551adf700 1 -- start start 2020-11-20T12:08:05.941 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.934+0000 7fb551adf700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb54c0ad000 0x7fb54c13e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:05.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.934+0000 7fb551adf700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb54c13a6e0 0x7fb54c13ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:05.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.934+0000 7fb551adf700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb54c067730 con 0x7fb54c0ad3e0 2020-11-20T12:08:05.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.934+0000 7fb551adf700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb54c066050 con 0x7fb54c0ad000 2020-11-20T12:08:05.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.934+0000 7fb551adf700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c138190 con 0x7fb54c13a6e0 2020-11-20T12:08:05.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.934+0000 7fb550add700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb54c0ad000 0x7fb54c13e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:05.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.934+0000 7fb550add700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb54c0ad000 0x7fb54c13e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49558/0 (socket says 172.21.15.189:49558) 2020-11-20T12:08:05.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.934+0000 7fb550add700 1 -- 172.21.15.189:0/4072694982 learned_addr learned my addr 172.21.15.189:0/4072694982 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:08:05.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.934+0000 7fb5512de700 1 --2- 172.21.15.189:0/4072694982 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb54c13a6e0 0x7fb54c13ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:05.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.938+0000 7fb54b7fe700 1 -- 172.21.15.189:0/4072694982 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3407420038 0 0) 0x7fb54c067730 con 0x7fb54c0ad3e0 2020-11-20T12:08:05.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.938+0000 7fb54b7fe700 1 -- 172.21.15.189:0/4072694982 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb538003240 con 0x7fb54c0ad3e0 2020-11-20T12:08:05.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.938+0000 7fb550add700 1 -- 172.21.15.189:0/4072694982 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb54c13a6e0 msgr2=0x7fb54c13ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:05.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.938+0000 7fb550add700 1 --2- 172.21.15.189:0/4072694982 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb54c13a6e0 0x7fb54c13ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:05.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.938+0000 7fb550add700 1 -- 172.21.15.189:0/4072694982 >> v1:172.21.15.47:6789/0 conn(0x7fb54c0ad3e0 legacy=0x7fb54c066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:05.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.938+0000 7fb550add700 1 -- 172.21.15.189:0/4072694982 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54c13dd40 con 0x7fb54c0ad000 2020-11-20T12:08:05.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.938+0000 7fb5512de700 1 --2- 172.21.15.189:0/4072694982 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb54c13a6e0 0x7fb54c13ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:08:05.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.938+0000 7fb550add700 1 --2- 172.21.15.189:0/4072694982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb54c0ad000 0x7fb54c13e480 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fb53c00ba20 tx=0x7fb53c006eb0).ready entity=mon.1 client_cookie=70540366f6086c3d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:05.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.938+0000 7fb54b7fe700 1 -- 172.21.15.189:0/4072694982 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53c009860 con 0x7fb54c0ad000 2020-11-20T12:08:05.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.938+0000 7fb54b7fe700 1 -- 172.21.15.189:0/4072694982 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53c0099c0 con 0x7fb54c0ad000 2020-11-20T12:08:05.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.938+0000 7fb54b7fe700 1 -- 172.21.15.189:0/4072694982 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53c009d40 con 0x7fb54c0ad000 2020-11-20T12:08:05.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.938+0000 7fb551adf700 1 -- 172.21.15.189:0/4072694982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb54c0ad000 msgr2=0x7fb54c13e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:05.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.938+0000 7fb551adf700 1 --2- 172.21.15.189:0/4072694982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb54c0ad000 0x7fb54c13e480 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fb53c00ba20 tx=0x7fb53c006eb0).stop 2020-11-20T12:08:05.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.938+0000 7fb551adf700 1 -- 172.21.15.189:0/4072694982 shutdown_connections 2020-11-20T12:08:05.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.938+0000 7fb551adf700 1 --2- 172.21.15.189:0/4072694982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb54c0ad000 0x7fb54c13e480 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:05.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.938+0000 7fb551adf700 1 --2- 172.21.15.189:0/4072694982 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb54c13a6e0 0x7fb54c13ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:05.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.938+0000 7fb551adf700 1 -- 172.21.15.189:0/4072694982 >> 172.21.15.189:0/4072694982 conn(0x7fb54c062080 msgr2=0x7fb54c062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:05.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.938+0000 7fb551adf700 1 -- 172.21.15.189:0/4072694982 shutdown_connections 2020-11-20T12:08:05.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.938+0000 7fb551adf700 1 -- 172.21.15.189:0/4072694982 wait complete. 2020-11-20T12:08:05.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.938+0000 7fb551adf700 1 Processor -- start 2020-11-20T12:08:05.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.938+0000 7fb551adf700 1 -- start start 2020-11-20T12:08:05.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.942+0000 7fb551adf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb54c0ad3e0 0x7fb54c13ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:05.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.942+0000 7fb551adf700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb54c0a5650 0x7fb54c1499b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:05.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.942+0000 7fb551adf700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb54c0a9880 0x7fb54c0a9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:05.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.942+0000 7fb551adf700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c14d300 con 0x7fb54c0a5650 2020-11-20T12:08:05.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.942+0000 7fb551adf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c14d5a0 con 0x7fb54c0ad3e0 2020-11-20T12:08:05.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.942+0000 7fb551adf700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c14d840 con 0x7fb54c0a9880 2020-11-20T12:08:05.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.942+0000 7fb550add700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb54c0ad3e0 0x7fb54c13ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:05.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.942+0000 7fb550add700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb54c0ad3e0 0x7fb54c13ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49566/0 (socket says 172.21.15.189:49566) 2020-11-20T12:08:05.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.942+0000 7fb550add700 1 -- 172.21.15.189:0/2262983401 learned_addr learned my addr 172.21.15.189:0/2262983401 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:08:05.954 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.942+0000 7fb54bfff700 1 --2- 172.21.15.189:0/2262983401 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb54c0a5650 0x7fb54c1499b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:05.954 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.942+0000 7fb5512de700 1 --2- 172.21.15.189:0/2262983401 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb54c0a9880 0x7fb54c0a9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:05.954 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.942+0000 7fb550add700 1 -- 172.21.15.189:0/2262983401 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb54c0a9880 msgr2=0x7fb54c0a9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:05.954 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.942+0000 7fb550add700 1 --2- 172.21.15.189:0/2262983401 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb54c0a9880 0x7fb54c0a9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:05.955 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.942+0000 7fb550add700 1 -- 172.21.15.189:0/2262983401 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb54c0a5650 msgr2=0x7fb54c1499b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:05.955 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.942+0000 7fb550add700 1 --2- 172.21.15.189:0/2262983401 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb54c0a5650 0x7fb54c1499b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:05.955 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.942+0000 7fb550add700 1 -- 172.21.15.189:0/2262983401 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54c0aa1d0 con 0x7fb54c0ad3e0 2020-11-20T12:08:05.955 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.942+0000 7fb550add700 1 --2- 172.21.15.189:0/2262983401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb54c0ad3e0 0x7fb54c13ff00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fb53c006ef0 tx=0x7fb53c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:05.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.942+0000 7fb548ff9700 1 -- 172.21.15.189:0/2262983401 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53c011d80 con 0x7fb54c0ad3e0 2020-11-20T12:08:05.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.942+0000 7fb548ff9700 1 -- 172.21.15.189:0/2262983401 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53c01e460 con 0x7fb54c0ad3e0 2020-11-20T12:08:05.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.942+0000 7fb548ff9700 1 -- 172.21.15.189:0/2262983401 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53c01e820 con 0x7fb54c0ad3e0 2020-11-20T12:08:05.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.942+0000 7fb551adf700 1 -- 172.21.15.189:0/2262983401 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb54c14cf80 con 0x7fb54c0ad3e0 2020-11-20T12:08:05.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.942+0000 7fb551adf700 1 -- 172.21.15.189:0/2262983401 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb54c1406e0 con 0x7fb54c0ad3e0 2020-11-20T12:08:05.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.942+0000 7fb551adf700 1 -- 172.21.15.189:0/2262983401 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb54c14da00 con 0x7fb54c0ad3e0 2020-11-20T12:08:05.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.946+0000 7fb548ff9700 1 -- 172.21.15.189:0/2262983401 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fb53c01a040 con 0x7fb54c0ad3e0 2020-11-20T12:08:05.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.946+0000 7fb548ff9700 1 --2- 172.21.15.189:0/2262983401 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb5240557c0 0x7fb524057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:05.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.946+0000 7fb548ff9700 1 -- 172.21.15.189:0/2262983401 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fb53c06f8f0 con 0x7fb54c0ad3e0 2020-11-20T12:08:05.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.946+0000 7fb54bfff700 1 --2- 172.21.15.189:0/2262983401 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb5240557c0 0x7fb524057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:05.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.946+0000 7fb54bfff700 1 --2- 172.21.15.189:0/2262983401 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb5240557c0 0x7fb524057c80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fb540007fd0 tx=0x7fb540007a20).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:05.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:05.954+0000 7fb548ff9700 1 -- 172.21.15.189:0/2262983401 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb53c045740 con 0x7fb54c0ad3e0 2020-11-20T12:08:06.162 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:06.154+0000 7fb551adf700 1 -- 172.21.15.189:0/2262983401 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb54c0446d0 con 0x7fb54c0ad3e0 2020-11-20T12:08:06.162 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:06.154+0000 7fb548ff9700 1 -- 172.21.15.189:0/2262983401 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+2139 (secure 0 0 0) 0x7fb53c023070 con 0x7fb54c0ad3e0 2020-11-20T12:08:06.163 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:06.158+0000 7fb551adf700 1 -- 172.21.15.189:0/2262983401 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb5240557c0 msgr2=0x7fb524057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:06.163 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:06.158+0000 7fb551adf700 1 --2- 172.21.15.189:0/2262983401 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb5240557c0 0x7fb524057c80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fb540007fd0 tx=0x7fb540007a20).stop 2020-11-20T12:08:06.163 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:06.158+0000 7fb551adf700 1 -- 172.21.15.189:0/2262983401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb54c0ad3e0 msgr2=0x7fb54c13ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:06.163 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:06.158+0000 7fb551adf700 1 --2- 172.21.15.189:0/2262983401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb54c0ad3e0 0x7fb54c13ff00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fb53c006ef0 tx=0x7fb53c015000).stop 2020-11-20T12:08:06.164 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:06.158+0000 7fb551adf700 1 -- 172.21.15.189:0/2262983401 shutdown_connections 2020-11-20T12:08:06.164 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:06.158+0000 7fb551adf700 1 --2- 172.21.15.189:0/2262983401 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb54c0a9880 0x7fb54c0a9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:06.164 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:06.158+0000 7fb551adf700 1 --2- 172.21.15.189:0/2262983401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb54c0ad3e0 0x7fb54c13ff00 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:06.164 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:06.158+0000 7fb551adf700 1 --2- 172.21.15.189:0/2262983401 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb5240557c0 0x7fb524057c80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:06.164 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:06.158+0000 7fb551adf700 1 --2- 172.21.15.189:0/2262983401 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb54c0a5650 0x7fb54c1499b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:06.165 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:06.158+0000 7fb551adf700 1 -- 172.21.15.189:0/2262983401 >> 172.21.15.189:0/2262983401 conn(0x7fb54c062080 msgr2=0x7fb54c1373e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:06.166 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:06.158+0000 7fb551adf700 1 -- 172.21.15.189:0/2262983401 shutdown_connections 2020-11-20T12:08:06.166 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:06.158+0000 7fb551adf700 1 -- 172.21.15.189:0/2262983401 wait complete. 2020-11-20T12:08:06.166 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 11 2020-11-20T12:08:06.166 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:08:06.167 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4849,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1375880901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1375880901},{"type":"v1","addr":"172.21.15.189:6835","nonce":1375880901}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:06:55.583738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4248},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"b","rank":0,"incarnation":7,"state":"up:active","state_seq":34,"addr":"172.21.15.47:6833/2660417436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2660417436},{"type":"v1","addr":"172.21.15.47:6833","nonce":2660417436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:08:06.174 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:08:08.175 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:08:08.308 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.306+0000 7f88fd9cd700 1 Processor -- start 2020-11-20T12:08:08.309 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.306+0000 7f88fd9cd700 1 -- start start 2020-11-20T12:08:08.310 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.306+0000 7f88fd9cd700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f88f813f480 0x7f88f813b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:08.310 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.306+0000 7f88fd9cd700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88f80676a0 con 0x7f88f813be50 2020-11-20T12:08:08.310 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.306+0000 7f88fd9cd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88f8067c50 con 0x7f88f80a9800 2020-11-20T12:08:08.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.306+0000 7f88fd9cd700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88f8065ec0 con 0x7f88f813f480 2020-11-20T12:08:08.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.306+0000 7f88f6ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f88f80a9800 0x7f88f813ed70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51194/0 (socket says 172.21.15.189:51194) 2020-11-20T12:08:08.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.306+0000 7f88f6ffd700 1 -- 172.21.15.189:0/2504835819 learned_addr learned my addr 172.21.15.189:0/2504835819 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:08:08.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.306+0000 7f88f67fc700 1 --2- 172.21.15.189:0/2504835819 >> v2:172.21.15.47:3300/0 conn(0x7f88f813f480 0x7f88f813b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:08.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.306+0000 7f88f5ffb700 1 -- 172.21.15.189:0/2504835819 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2556792152 0 0) 0x7f88f8067c50 con 0x7f88f80a9800 2020-11-20T12:08:08.312 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.306+0000 7f88f5ffb700 1 -- 172.21.15.189:0/2504835819 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88d8003240 con 0x7f88f80a9800 2020-11-20T12:08:08.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.306+0000 7f88f67fc700 1 -- 172.21.15.189:0/2504835819 >> v1:172.21.15.189:6789/0 conn(0x7f88f80a9800 legacy=0x7f88f813ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:08.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.306+0000 7f88f67fc700 1 -- 172.21.15.189:0/2504835819 >> v1:172.21.15.47:6789/0 conn(0x7f88f813be50 legacy=0x7f88f81492c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:08.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.306+0000 7f88f67fc700 1 -- 172.21.15.189:0/2504835819 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f8067160 con 0x7f88f813f480 2020-11-20T12:08:08.315 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.306+0000 7f88f67fc700 1 --2- 172.21.15.189:0/2504835819 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88f813f480 0x7f88f813b960 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f88e000b9a0 tx=0x7f88e0006df0).ready entity=mon.0 client_cookie=bf9b4701583b1caa server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:08.315 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.306+0000 7f88f5ffb700 1 -- 172.21.15.189:0/2504835819 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e00097b0 con 0x7f88f813f480 2020-11-20T12:08:08.315 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.306+0000 7f88f5ffb700 1 -- 172.21.15.189:0/2504835819 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e0009910 con 0x7f88f813f480 2020-11-20T12:08:08.315 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.306+0000 7f88f5ffb700 1 -- 172.21.15.189:0/2504835819 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e0009c90 con 0x7f88f813f480 2020-11-20T12:08:08.315 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.306+0000 7f88fd9cd700 1 -- 172.21.15.189:0/2504835819 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88f813f480 msgr2=0x7f88f813b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:08.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.306+0000 7f88fd9cd700 1 --2- 172.21.15.189:0/2504835819 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88f813f480 0x7f88f813b960 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f88e000b9a0 tx=0x7f88e0006df0).stop 2020-11-20T12:08:08.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.306+0000 7f88fd9cd700 1 -- 172.21.15.189:0/2504835819 shutdown_connections 2020-11-20T12:08:08.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.306+0000 7f88fd9cd700 1 --2- 172.21.15.189:0/2504835819 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88f813f480 0x7f88f813b960 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:08.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.306+0000 7f88fd9cd700 1 -- 172.21.15.189:0/2504835819 >> 172.21.15.189:0/2504835819 conn(0x7f88f8061f60 msgr2=0x7f88f8062370 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:08.317 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.306+0000 7f88fd9cd700 1 -- 172.21.15.189:0/2504835819 shutdown_connections 2020-11-20T12:08:08.317 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.310+0000 7f88fd9cd700 1 -- 172.21.15.189:0/2504835819 wait complete. 2020-11-20T12:08:08.317 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.310+0000 7f88fd9cd700 1 Processor -- start 2020-11-20T12:08:08.317 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.310+0000 7f88fd9cd700 1 -- start start 2020-11-20T12:08:08.317 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.310+0000 7f88fd9cd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88f80a9800 0x7f88f8116600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:08.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.310+0000 7f88fd9cd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88f813be50 0x7f88f8114b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:08.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.310+0000 7f88fd9cd700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f88f813f480 0x7f88f8117e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:08.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.310+0000 7f88fd9cd700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f80a2a90 con 0x7f88f80a9800 2020-11-20T12:08:08.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.310+0000 7f88fd9cd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f80a2d30 con 0x7f88f813be50 2020-11-20T12:08:08.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.310+0000 7f88fd9cd700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f80a2fd0 con 0x7f88f813f480 2020-11-20T12:08:08.319 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.310+0000 7f88f67fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88f813be50 0x7f88f8114b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:08.319 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.310+0000 7f88f67fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88f813be50 0x7f88f8114b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49580/0 (socket says 172.21.15.189:49580) 2020-11-20T12:08:08.319 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.310+0000 7f88f67fc700 1 -- 172.21.15.189:0/1568323781 learned_addr learned my addr 172.21.15.189:0/1568323781 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:08:08.319 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.310+0000 7f88f6ffd700 1 --2- 172.21.15.189:0/1568323781 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88f80a9800 0x7f88f8116600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:08.319 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.310+0000 7f88f77fe700 1 --2- 172.21.15.189:0/1568323781 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f88f813f480 0x7f88f8117e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:08.321 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.310+0000 7f88f67fc700 1 -- 172.21.15.189:0/1568323781 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f88f813f480 msgr2=0x7f88f8117e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:08.321 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.310+0000 7f88f67fc700 1 --2- 172.21.15.189:0/1568323781 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f88f813f480 0x7f88f8117e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:08.322 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.310+0000 7f88f67fc700 1 -- 172.21.15.189:0/1568323781 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88f80a9800 msgr2=0x7f88f8116600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:08.322 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.310+0000 7f88f67fc700 1 --2- 172.21.15.189:0/1568323781 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88f80a9800 0x7f88f8116600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:08.322 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.310+0000 7f88f67fc700 1 -- 172.21.15.189:0/1568323781 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f8115010 con 0x7f88f813be50 2020-11-20T12:08:08.322 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.310+0000 7f88f6ffd700 1 --2- 172.21.15.189:0/1568323781 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88f80a9800 0x7f88f8116600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:08:08.322 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.310+0000 7f88f67fc700 1 --2- 172.21.15.189:0/1568323781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88f813be50 0x7f88f8114b20 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f88e0010000 tx=0x7f88e0015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:08.323 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.310+0000 7f88eeffd700 1 -- 172.21.15.189:0/1568323781 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e0011e50 con 0x7f88f813be50 2020-11-20T12:08:08.323 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.310+0000 7f88eeffd700 1 -- 172.21.15.189:0/1568323781 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e001d3c0 con 0x7f88f813be50 2020-11-20T12:08:08.323 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.310+0000 7f88fd9cd700 1 -- 172.21.15.189:0/1568323781 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f80a3820 con 0x7f88f813be50 2020-11-20T12:08:08.323 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.310+0000 7f88eeffd700 1 -- 172.21.15.189:0/1568323781 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e001d780 con 0x7f88f813be50 2020-11-20T12:08:08.323 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.310+0000 7f88fd9cd700 1 -- 172.21.15.189:0/1568323781 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f81184f0 con 0x7f88f813be50 2020-11-20T12:08:08.324 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.314+0000 7f88eeffd700 1 -- 172.21.15.189:0/1568323781 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f88e001a040 con 0x7f88f813be50 2020-11-20T12:08:08.324 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.314+0000 7f88fd9cd700 1 -- 172.21.15.189:0/1568323781 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88f81186e0 con 0x7f88f813be50 2020-11-20T12:08:08.324 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.314+0000 7f88eeffd700 1 --2- 172.21.15.189:0/1568323781 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f88cc0557f0 0x7f88cc057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:08.324 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.314+0000 7f88eeffd700 1 -- 172.21.15.189:0/1568323781 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f88e006fbe0 con 0x7f88f813be50 2020-11-20T12:08:08.324 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.314+0000 7f88f6ffd700 1 --2- 172.21.15.189:0/1568323781 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f88cc0557f0 0x7f88cc057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:08.325 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.314+0000 7f88f6ffd700 1 --2- 172.21.15.189:0/1568323781 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f88cc0557f0 0x7f88cc057cb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f88f0005920 tx=0x7f88f0009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:08.325 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.322+0000 7f88eeffd700 1 -- 172.21.15.189:0/1568323781 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f88f81186e0 con 0x7f88f813be50 2020-11-20T12:08:08.524 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.522+0000 7f88fd9cd700 1 -- 172.21.15.189:0/1568323781 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88f81186e0 con 0x7f88f813be50 2020-11-20T12:08:08.525 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.522+0000 7f88eeffd700 1 -- 172.21.15.189:0/1568323781 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+2139 (secure 0 0 0) 0x7f88f81186e0 con 0x7f88f813be50 2020-11-20T12:08:08.525 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:08:08.526 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4849,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1375880901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1375880901},{"type":"v1","addr":"172.21.15.189:6835","nonce":1375880901}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:06:55.583738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4248},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"b","rank":0,"incarnation":7,"state":"up:active","state_seq":34,"addr":"172.21.15.47:6833/2660417436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2660417436},{"type":"v1","addr":"172.21.15.47:6833","nonce":2660417436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:08:08.527 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.522+0000 7f88fd9cd700 1 -- 172.21.15.189:0/1568323781 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f88cc0557f0 msgr2=0x7f88cc057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:08.528 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.522+0000 7f88fd9cd700 1 --2- 172.21.15.189:0/1568323781 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f88cc0557f0 0x7f88cc057cb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f88f0005920 tx=0x7f88f0009040).stop 2020-11-20T12:08:08.528 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.522+0000 7f88fd9cd700 1 -- 172.21.15.189:0/1568323781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88f813be50 msgr2=0x7f88f8114b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:08.528 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.522+0000 7f88fd9cd700 1 --2- 172.21.15.189:0/1568323781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88f813be50 0x7f88f8114b20 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f88e0010000 tx=0x7f88e0015000).stop 2020-11-20T12:08:08.528 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.522+0000 7f88fd9cd700 1 -- 172.21.15.189:0/1568323781 shutdown_connections 2020-11-20T12:08:08.528 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.522+0000 7f88fd9cd700 1 --2- 172.21.15.189:0/1568323781 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f88cc0557f0 0x7f88cc057cb0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:08.529 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.522+0000 7f88fd9cd700 1 --2- 172.21.15.189:0/1568323781 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88f80a9800 0x7f88f8116600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:08.529 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.522+0000 7f88fd9cd700 1 --2- 172.21.15.189:0/1568323781 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f88f813f480 0x7f88f8117e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:08.529 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.522+0000 7f88fd9cd700 1 --2- 172.21.15.189:0/1568323781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88f813be50 0x7f88f8114b20 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:08.529 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.522+0000 7f88fd9cd700 1 -- 172.21.15.189:0/1568323781 >> 172.21.15.189:0/1568323781 conn(0x7f88f8061f60 msgr2=0x7f88f81450a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:08.529 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.522+0000 7f88fd9cd700 1 -- 172.21.15.189:0/1568323781 shutdown_connections 2020-11-20T12:08:08.530 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:08.522+0000 7f88fd9cd700 1 -- 172.21.15.189:0/1568323781 wait complete. 2020-11-20T12:08:08.530 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 11 2020-11-20T12:08:08.539 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:08:10.541 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:08:10.668 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f4581263700 1 Processor -- start 2020-11-20T12:08:10.669 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f4581263700 1 -- start start 2020-11-20T12:08:10.670 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f4581263700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f457c0612b0 0x7f457c0a0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:10.670 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f4581263700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f457c067b50 0x7f457c136fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:10.671 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f4581263700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457c068020 con 0x7f457c1374a0 2020-11-20T12:08:10.671 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f4581263700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f457c0682c0 con 0x7f457c067b50 2020-11-20T12:08:10.671 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f4581263700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f457c137900 con 0x7f457c0612b0 2020-11-20T12:08:10.671 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f457bfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f457c0612b0 0x7f457c0a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:10.671 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f4580a62700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f457c1374a0 0x7f457c133a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51212/0 (socket says 172.21.15.189:51212) 2020-11-20T12:08:10.672 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f4580a62700 1 -- 172.21.15.189:0/1309731576 learned_addr learned my addr 172.21.15.189:0/1309731576 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:08:10.672 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f457b7fe700 1 --2- 172.21.15.189:0/1309731576 >> v2:172.21.15.47:3300/0 conn(0x7f457c067b50 0x7f457c136fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:10.672 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f457bfff700 1 -- 172.21.15.189:0/1309731576 >> v2:172.21.15.47:3300/0 conn(0x7f457c067b50 msgr2=0x7f457c136fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:10.672 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f457bfff700 1 --2- 172.21.15.189:0/1309731576 >> v2:172.21.15.47:3300/0 conn(0x7f457c067b50 0x7f457c136fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:10.672 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f457bfff700 1 -- 172.21.15.189:0/1309731576 >> v1:172.21.15.189:6789/0 conn(0x7f457c1374a0 legacy=0x7f457c133a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:10.673 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f457bfff700 1 -- 172.21.15.189:0/1309731576 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457c061690 con 0x7f457c0612b0 2020-11-20T12:08:10.673 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f457affd700 1 -- 172.21.15.189:0/1309731576 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2099500786 0 0) 0x7f457c068020 con 0x7f457c1374a0 2020-11-20T12:08:10.673 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f457bfff700 1 --2- 172.21.15.189:0/1309731576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f457c0612b0 0x7f457c0a0960 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f457c0ae330 tx=0x7f45740091d0).ready entity=mon.1 client_cookie=70a0c90f555ec8c1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:10.673 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f457affd700 1 -- 172.21.15.189:0/1309731576 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457400ba70 con 0x7f457c0612b0 2020-11-20T12:08:10.673 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f457affd700 1 -- 172.21.15.189:0/1309731576 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457400bbd0 con 0x7f457c0612b0 2020-11-20T12:08:10.674 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f457affd700 1 -- 172.21.15.189:0/1309731576 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4574011400 con 0x7f457c0612b0 2020-11-20T12:08:10.676 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f4581263700 1 -- 172.21.15.189:0/1309731576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f457c0612b0 msgr2=0x7f457c0a0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:10.676 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f4581263700 1 --2- 172.21.15.189:0/1309731576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f457c0612b0 0x7f457c0a0960 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f457c0ae330 tx=0x7f45740091d0).stop 2020-11-20T12:08:10.676 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f4581263700 1 -- 172.21.15.189:0/1309731576 shutdown_connections 2020-11-20T12:08:10.677 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f4581263700 1 --2- 172.21.15.189:0/1309731576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f457c0612b0 0x7f457c0a0960 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:10.677 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f4581263700 1 --2- 172.21.15.189:0/1309731576 >> v2:172.21.15.47:3300/0 conn(0x7f457c067b50 0x7f457c136fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:10.677 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f4581263700 1 -- 172.21.15.189:0/1309731576 >> 172.21.15.189:0/1309731576 conn(0x7f457c063210 msgr2=0x7f457c060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:10.682 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f4581263700 1 -- 172.21.15.189:0/1309731576 shutdown_connections 2020-11-20T12:08:10.682 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.666+0000 7f4581263700 1 -- 172.21.15.189:0/1309731576 wait complete. 2020-11-20T12:08:10.682 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.670+0000 7f4581263700 1 Processor -- start 2020-11-20T12:08:10.682 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.670+0000 7f4581263700 1 -- start start 2020-11-20T12:08:10.682 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.670+0000 7f4581263700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f457c067b50 0x7f457c132700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:10.683 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.670+0000 7f4581263700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f457c1374a0 0x7f457c12e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:10.683 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.670+0000 7f4581263700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f457c138490 0x7f457c12e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:10.683 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.670+0000 7f4581263700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457c12eb60 con 0x7f457c1374a0 2020-11-20T12:08:10.683 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.670+0000 7f4581263700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457c138a10 con 0x7f457c138490 2020-11-20T12:08:10.683 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.670+0000 7f4581263700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f457c138cb0 con 0x7f457c067b50 2020-11-20T12:08:10.684 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.670+0000 7f4580a62700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f457c138490 0x7f457c12e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:10.684 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.670+0000 7f4580a62700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f457c138490 0x7f457c12e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49596/0 (socket says 172.21.15.189:49596) 2020-11-20T12:08:10.684 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.670+0000 7f4580a62700 1 -- 172.21.15.189:0/1968030094 learned_addr learned my addr 172.21.15.189:0/1968030094 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:08:10.684 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.670+0000 7f457b7fe700 1 --2- 172.21.15.189:0/1968030094 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f457c1374a0 0x7f457c12e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:10.684 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.670+0000 7f457bfff700 1 --2- 172.21.15.189:0/1968030094 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f457c067b50 0x7f457c132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:10.685 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.670+0000 7f4580a62700 1 -- 172.21.15.189:0/1968030094 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f457c067b50 msgr2=0x7f457c132700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:10.685 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.670+0000 7f4580a62700 1 --2- 172.21.15.189:0/1968030094 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f457c067b50 0x7f457c132700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:10.685 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.670+0000 7f4580a62700 1 -- 172.21.15.189:0/1968030094 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f457c1374a0 msgr2=0x7f457c12e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:10.685 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.670+0000 7f4580a62700 1 --2- 172.21.15.189:0/1968030094 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f457c1374a0 0x7f457c12e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:10.686 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.670+0000 7f4580a62700 1 -- 172.21.15.189:0/1968030094 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457c12dce0 con 0x7f457c138490 2020-11-20T12:08:10.686 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.670+0000 7f4580a62700 1 --2- 172.21.15.189:0/1968030094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f457c138490 0x7f457c12e670 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f457000c790 tx=0x7f4570007c60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:10.686 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.670+0000 7f455bfff700 1 -- 172.21.15.189:0/1968030094 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45700072e0 con 0x7f457c138490 2020-11-20T12:08:10.686 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.670+0000 7f455bfff700 1 -- 172.21.15.189:0/1968030094 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4570007440 con 0x7f457c138490 2020-11-20T12:08:10.686 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.670+0000 7f4581263700 1 -- 172.21.15.189:0/1968030094 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f457c12f4f0 con 0x7f457c138490 2020-11-20T12:08:10.687 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.670+0000 7f455bfff700 1 -- 172.21.15.189:0/1968030094 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4570005820 con 0x7f457c138490 2020-11-20T12:08:10.687 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.674+0000 7f4581263700 1 -- 172.21.15.189:0/1968030094 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f457c132f90 con 0x7f457c138490 2020-11-20T12:08:10.688 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.674+0000 7f455bfff700 1 -- 172.21.15.189:0/1968030094 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f4570010070 con 0x7f457c138490 2020-11-20T12:08:10.688 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.674+0000 7f4581263700 1 -- 172.21.15.189:0/1968030094 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f457c12de70 con 0x7f457c138490 2020-11-20T12:08:10.688 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.674+0000 7f455bfff700 1 --2- 172.21.15.189:0/1968030094 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f45540557c0 0x7f4554057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:10.689 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.674+0000 7f455bfff700 1 -- 172.21.15.189:0/1968030094 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f457006dda0 con 0x7f457c138490 2020-11-20T12:08:10.689 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.674+0000 7f457bfff700 1 --2- 172.21.15.189:0/1968030094 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f45540557c0 0x7f4554057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:10.689 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.674+0000 7f457bfff700 1 --2- 172.21.15.189:0/1968030094 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f45540557c0 0x7f4554057c80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f457c0656c0 tx=0x7f4574016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:10.689 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.678+0000 7f455bfff700 1 -- 172.21.15.189:0/1968030094 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4570043ae0 con 0x7f457c138490 2020-11-20T12:08:10.884 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.878+0000 7f4581263700 1 -- 172.21.15.189:0/1968030094 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f457c066600 con 0x7f457c138490 2020-11-20T12:08:10.885 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.882+0000 7f455bfff700 1 -- 172.21.15.189:0/1968030094 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+2139 (secure 0 0 0) 0x7f4570047100 con 0x7f457c138490 2020-11-20T12:08:10.886 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:08:10.887 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4849,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1375880901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1375880901},{"type":"v1","addr":"172.21.15.189:6835","nonce":1375880901}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:06:55.583738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4248},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"b","rank":0,"incarnation":7,"state":"up:active","state_seq":34,"addr":"172.21.15.47:6833/2660417436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2660417436},{"type":"v1","addr":"172.21.15.47:6833","nonce":2660417436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:08:10.889 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.882+0000 7f4581263700 1 -- 172.21.15.189:0/1968030094 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f45540557c0 msgr2=0x7f4554057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:10.890 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.882+0000 7f4581263700 1 --2- 172.21.15.189:0/1968030094 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f45540557c0 0x7f4554057c80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f457c0656c0 tx=0x7f4574016000).stop 2020-11-20T12:08:10.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.882+0000 7f4581263700 1 -- 172.21.15.189:0/1968030094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f457c138490 msgr2=0x7f457c12e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:10.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.882+0000 7f4581263700 1 --2- 172.21.15.189:0/1968030094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f457c138490 0x7f457c12e670 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f457000c790 tx=0x7f4570007c60).stop 2020-11-20T12:08:10.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.882+0000 7f4581263700 1 -- 172.21.15.189:0/1968030094 shutdown_connections 2020-11-20T12:08:10.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.882+0000 7f4581263700 1 --2- 172.21.15.189:0/1968030094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f457c138490 0x7f457c12e670 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:10.893 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.882+0000 7f4581263700 1 --2- 172.21.15.189:0/1968030094 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f457c067b50 0x7f457c132700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:10.893 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.882+0000 7f4581263700 1 --2- 172.21.15.189:0/1968030094 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f45540557c0 0x7f4554057c80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:10.894 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.882+0000 7f4581263700 1 --2- 172.21.15.189:0/1968030094 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f457c1374a0 0x7f457c12e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:10.894 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.882+0000 7f4581263700 1 -- 172.21.15.189:0/1968030094 >> 172.21.15.189:0/1968030094 conn(0x7f457c063210 msgr2=0x7f457c133f30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:10.895 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.882+0000 7f4581263700 1 -- 172.21.15.189:0/1968030094 shutdown_connections 2020-11-20T12:08:10.896 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:10.882+0000 7f4581263700 1 -- 172.21.15.189:0/1968030094 wait complete. 2020-11-20T12:08:10.896 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 11 2020-11-20T12:08:10.904 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:08:12.904 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:08:13.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f0179a31700 1 Processor -- start 2020-11-20T12:08:13.042 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f0179a31700 1 -- start start 2020-11-20T12:08:13.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f0179a31700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f01740ad3e0 0x7f0174066d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:13.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f0179a31700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f017413a6e0 0x7f017413ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:13.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f0179a31700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0174067730 con 0x7f01740ad000 2020-11-20T12:08:13.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f0179a31700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0174066050 con 0x7f01740ad3e0 2020-11-20T12:08:13.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f0179a31700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0174138190 con 0x7f017413a6e0 2020-11-20T12:08:13.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f0178a2f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f01740ad000 0x7f017413e480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51226/0 (socket says 172.21.15.189:51226) 2020-11-20T12:08:13.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f0178a2f700 1 -- 172.21.15.189:0/3243410371 learned_addr learned my addr 172.21.15.189:0/3243410371 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:08:13.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f0179230700 1 --2- 172.21.15.189:0/3243410371 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f017413a6e0 0x7f017413ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:13.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f0173fff700 1 --2- 172.21.15.189:0/3243410371 >> v2:172.21.15.47:3300/0 conn(0x7f01740ad3e0 0x7f0174066d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:13.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f01737fe700 1 -- 172.21.15.189:0/3243410371 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649971475 0 0) 0x7f0174067730 con 0x7f01740ad000 2020-11-20T12:08:13.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f01737fe700 1 -- 172.21.15.189:0/3243410371 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0158003240 con 0x7f01740ad000 2020-11-20T12:08:13.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f01737fe700 1 -- 172.21.15.189:0/3243410371 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3334149206 0 0) 0x7f0158003240 con 0x7f01740ad000 2020-11-20T12:08:13.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f01737fe700 1 -- 172.21.15.189:0/3243410371 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f017413a6e0 msgr2=0x7f017413ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:13.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f01737fe700 1 --2- 172.21.15.189:0/3243410371 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f017413a6e0 0x7f017413ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:13.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f01737fe700 1 -- 172.21.15.189:0/3243410371 >> v2:172.21.15.47:3300/0 conn(0x7f01740ad3e0 msgr2=0x7f0174066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:13.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f01737fe700 1 --2- 172.21.15.189:0/3243410371 >> v2:172.21.15.47:3300/0 conn(0x7f01740ad3e0 0x7f0174066d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:13.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f01737fe700 1 -- 172.21.15.189:0/3243410371 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017413dd40 con 0x7f01740ad000 2020-11-20T12:08:13.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f01737fe700 1 -- 172.21.15.189:0/3243410371 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f01640014e0 con 0x7f01740ad000 2020-11-20T12:08:13.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f01737fe700 1 -- 172.21.15.189:0/3243410371 >> v1:172.21.15.189:6789/0 conn(0x7f01740ad000 legacy=0x7f017413e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:13.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f01737fe700 1 --2- 172.21.15.189:0/3243410371 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0158003760 0x7f0158003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:13.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f01737fe700 1 --2- 172.21.15.189:0/3243410371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0158004050 0x7f0158004520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:13.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f01737fe700 1 --2- 172.21.15.189:0/3243410371 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0158004b20 0x7f0158004fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:13.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f01737fe700 1 -- 172.21.15.189:0/3243410371 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0174138190 con 0x7f0158004b20 2020-11-20T12:08:13.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f01737fe700 1 -- 172.21.15.189:0/3243410371 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015800b830 con 0x7f0158004050 2020-11-20T12:08:13.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.038+0000 7f01737fe700 1 -- 172.21.15.189:0/3243410371 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015800b9f0 con 0x7f0158003760 2020-11-20T12:08:13.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.042+0000 7f0173fff700 1 --2- 172.21.15.189:0/3243410371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0158004050 0x7f0158004520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:13.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.042+0000 7f0178a2f700 1 --2- 172.21.15.189:0/3243410371 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0158003760 0x7f0158003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:13.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.042+0000 7f0179230700 1 --2- 172.21.15.189:0/3243410371 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0158004b20 0x7f0158004fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:13.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.042+0000 7f0178a2f700 1 -- 172.21.15.189:0/3243410371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0158004050 msgr2=0x7f0158004520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:13.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.042+0000 7f0178a2f700 1 --2- 172.21.15.189:0/3243410371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0158004050 0x7f0158004520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:13.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.042+0000 7f0178a2f700 1 -- 172.21.15.189:0/3243410371 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0158004b20 msgr2=0x7f0158004fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:13.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.042+0000 7f0178a2f700 1 --2- 172.21.15.189:0/3243410371 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0158004b20 0x7f0158004fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:13.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.042+0000 7f0178a2f700 1 -- 172.21.15.189:0/3243410371 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0158003240 con 0x7f0158003760 2020-11-20T12:08:13.052 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.042+0000 7f0178a2f700 1 --2- 172.21.15.189:0/3243410371 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0158003760 0x7f0158003b40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f016400c440 tx=0x7f017413ded0).ready entity=mon.2 client_cookie=cc4362b607010c9f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:13.052 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.042+0000 7f01737fe700 1 -- 172.21.15.189:0/3243410371 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01640070d0 con 0x7f0158003760 2020-11-20T12:08:13.052 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.042+0000 7f01737fe700 1 -- 172.21.15.189:0/3243410371 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0164007230 con 0x7f0158003760 2020-11-20T12:08:13.052 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.042+0000 7f01737fe700 1 -- 172.21.15.189:0/3243410371 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01640055f0 con 0x7f0158003760 2020-11-20T12:08:13.055 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.042+0000 7f0179a31700 1 -- 172.21.15.189:0/3243410371 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0158003760 msgr2=0x7f0158003b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:13.055 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.042+0000 7f0179a31700 1 --2- 172.21.15.189:0/3243410371 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0158003760 0x7f0158003b40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f016400c440 tx=0x7f017413ded0).stop 2020-11-20T12:08:13.056 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.042+0000 7f0179230700 1 -- 172.21.15.189:0/3243410371 reap_dead start 2020-11-20T12:08:13.056 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.042+0000 7f0179a31700 1 -- 172.21.15.189:0/3243410371 shutdown_connections 2020-11-20T12:08:13.056 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.042+0000 7f0179a31700 1 -- 172.21.15.189:0/3243410371 >> 172.21.15.189:0/3243410371 conn(0x7f0174062080 msgr2=0x7f0174062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:13.056 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.042+0000 7f0179a31700 1 -- 172.21.15.189:0/3243410371 shutdown_connections 2020-11-20T12:08:13.056 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.042+0000 7f0179a31700 1 -- 172.21.15.189:0/3243410371 wait complete. 2020-11-20T12:08:13.057 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.042+0000 7f0179a31700 1 Processor -- start 2020-11-20T12:08:13.057 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.046+0000 7f0179a31700 1 -- start start 2020-11-20T12:08:13.057 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.046+0000 7f0179a31700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0158003760 0x7f01740a2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:13.057 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.046+0000 7f0179a31700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01740a10b0 0x7f017413af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:13.057 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.046+0000 7f0179a31700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f017414cd30 0x7f017414d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:13.058 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.046+0000 7f0179a31700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01740a3760 con 0x7f0158003760 2020-11-20T12:08:13.058 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.046+0000 7f0179a31700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01740a34c0 con 0x7f01740a10b0 2020-11-20T12:08:13.058 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.046+0000 7f0179a31700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01740a0c10 con 0x7f017414cd30 2020-11-20T12:08:13.058 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.046+0000 7f0173fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01740a10b0 0x7f017413af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:13.058 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.046+0000 7f0173fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01740a10b0 0x7f017413af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49616/0 (socket says 172.21.15.189:49616) 2020-11-20T12:08:13.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.046+0000 7f0173fff700 1 -- 172.21.15.189:0/2364229178 learned_addr learned my addr 172.21.15.189:0/2364229178 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:08:13.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.046+0000 7f0178a2f700 1 --2- 172.21.15.189:0/2364229178 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0158003760 0x7f01740a2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:13.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.046+0000 7f0179230700 1 --2- 172.21.15.189:0/2364229178 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f017414cd30 0x7f017414d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:13.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.046+0000 7f0173fff700 1 -- 172.21.15.189:0/2364229178 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f017414cd30 msgr2=0x7f017414d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:13.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.046+0000 7f0173fff700 1 --2- 172.21.15.189:0/2364229178 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f017414cd30 0x7f017414d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:13.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.046+0000 7f0173fff700 1 -- 172.21.15.189:0/2364229178 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0158003760 msgr2=0x7f01740a2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:13.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.046+0000 7f0173fff700 1 --2- 172.21.15.189:0/2364229178 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0158003760 0x7f01740a2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:13.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.046+0000 7f0173fff700 1 -- 172.21.15.189:0/2364229178 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0174139bf0 con 0x7f01740a10b0 2020-11-20T12:08:13.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.046+0000 7f0179230700 1 --2- 172.21.15.189:0/2364229178 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f017414cd30 0x7f017414d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:08:13.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.046+0000 7f0173fff700 1 --2- 172.21.15.189:0/2364229178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01740a10b0 0x7f017413af00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f015c009f00 tx=0x7f015c0091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:13.061 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.046+0000 7f0170ff9700 1 -- 172.21.15.189:0/2364229178 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015c00fe30 con 0x7f01740a10b0 2020-11-20T12:08:13.061 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.046+0000 7f0170ff9700 1 -- 172.21.15.189:0/2364229178 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f015c014440 con 0x7f01740a10b0 2020-11-20T12:08:13.061 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.046+0000 7f0170ff9700 1 -- 172.21.15.189:0/2364229178 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015c0147c0 con 0x7f01740a10b0 2020-11-20T12:08:13.061 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.046+0000 7f0179a31700 1 -- 172.21.15.189:0/2364229178 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f017413b3f0 con 0x7f01740a10b0 2020-11-20T12:08:13.062 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.046+0000 7f0179a31700 1 -- 172.21.15.189:0/2364229178 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f017413a0f0 con 0x7f01740a10b0 2020-11-20T12:08:13.062 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.046+0000 7f0179a31700 1 -- 172.21.15.189:0/2364229178 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01740b1370 con 0x7f01740a10b0 2020-11-20T12:08:13.062 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.050+0000 7f0170ff9700 1 -- 172.21.15.189:0/2364229178 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f015c024070 con 0x7f01740a10b0 2020-11-20T12:08:13.063 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.050+0000 7f0170ff9700 1 --2- 172.21.15.189:0/2364229178 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f01440557c0 0x7f0144057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:13.063 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.050+0000 7f0170ff9700 1 -- 172.21.15.189:0/2364229178 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f015c06f2d0 con 0x7f01740a10b0 2020-11-20T12:08:13.063 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.050+0000 7f0178a2f700 1 --2- 172.21.15.189:0/2364229178 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f01440557c0 0x7f0144057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:13.064 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.050+0000 7f0178a2f700 1 --2- 172.21.15.189:0/2364229178 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f01440557c0 0x7f0144057c80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f01640078a0 tx=0x7f0164015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:13.064 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.054+0000 7f0170ff9700 1 -- 172.21.15.189:0/2364229178 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f015c0450c0 con 0x7f01740a10b0 2020-11-20T12:08:13.255 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.250+0000 7f0179a31700 1 -- 172.21.15.189:0/2364229178 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01740b18e0 con 0x7f01740a10b0 2020-11-20T12:08:13.256 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.254+0000 7f0170ff9700 1 -- 172.21.15.189:0/2364229178 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+2139 (secure 0 0 0) 0x7f015c022070 con 0x7f01740a10b0 2020-11-20T12:08:13.256 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:08:13.257 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4849,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1375880901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1375880901},{"type":"v1","addr":"172.21.15.189:6835","nonce":1375880901}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:06:55.583738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4248},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"b","rank":0,"incarnation":7,"state":"up:active","state_seq":34,"addr":"172.21.15.47:6833/2660417436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2660417436},{"type":"v1","addr":"172.21.15.47:6833","nonce":2660417436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:08:13.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.254+0000 7f0179a31700 1 -- 172.21.15.189:0/2364229178 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f01440557c0 msgr2=0x7f0144057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:13.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.254+0000 7f0179a31700 1 --2- 172.21.15.189:0/2364229178 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f01440557c0 0x7f0144057c80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f01640078a0 tx=0x7f0164015000).stop 2020-11-20T12:08:13.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.254+0000 7f0179a31700 1 -- 172.21.15.189:0/2364229178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01740a10b0 msgr2=0x7f017413af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:13.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.254+0000 7f0179a31700 1 --2- 172.21.15.189:0/2364229178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01740a10b0 0x7f017413af00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f015c009f00 tx=0x7f015c0091d0).stop 2020-11-20T12:08:13.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.254+0000 7f0179a31700 1 -- 172.21.15.189:0/2364229178 shutdown_connections 2020-11-20T12:08:13.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.254+0000 7f0179a31700 1 --2- 172.21.15.189:0/2364229178 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f01440557c0 0x7f0144057c80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:13.260 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.254+0000 7f0179a31700 1 --2- 172.21.15.189:0/2364229178 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0158003760 0x7f01740a2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:13.260 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.254+0000 7f0179a31700 1 --2- 172.21.15.189:0/2364229178 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f017414cd30 0x7f017414d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:13.260 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.254+0000 7f0179a31700 1 --2- 172.21.15.189:0/2364229178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01740a10b0 0x7f017413af00 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:13.260 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.254+0000 7f0179a31700 1 -- 172.21.15.189:0/2364229178 >> 172.21.15.189:0/2364229178 conn(0x7f0174062080 msgr2=0x7f0174137650 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:13.261 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.254+0000 7f0179a31700 1 -- 172.21.15.189:0/2364229178 shutdown_connections 2020-11-20T12:08:13.261 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:13.254+0000 7f0179a31700 1 -- 172.21.15.189:0/2364229178 wait complete. 2020-11-20T12:08:13.261 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 11 2020-11-20T12:08:13.271 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:08:15.272 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:08:15.405 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.402+0000 7f04cb302700 1 Processor -- start 2020-11-20T12:08:15.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.402+0000 7f04cb302700 1 -- start start 2020-11-20T12:08:15.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.402+0000 7f04cb302700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f04c40612b0 0x7f04c40a0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:15.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.402+0000 7f04cb302700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f04c40a85a0 0x7f04c40a8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:15.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.402+0000 7f04cb302700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c40a0f60 con 0x7f04c41339a0 2020-11-20T12:08:15.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.402+0000 7f04cb302700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04c4134540 con 0x7f04c40a85a0 2020-11-20T12:08:15.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.402+0000 7f04cb302700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04c4068440 con 0x7f04c40612b0 2020-11-20T12:08:15.409 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04c989f700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f04c41339a0 0x7f04c4133e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51248/0 (socket says 172.21.15.189:51248) 2020-11-20T12:08:15.409 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04c909e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f04c40612b0 0x7f04c40a0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:15.409 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04c989f700 1 -- 172.21.15.189:0/1718449227 learned_addr learned my addr 172.21.15.189:0/1718449227 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:08:15.409 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04c889d700 1 --2- 172.21.15.189:0/1718449227 >> v2:172.21.15.47:3300/0 conn(0x7f04c40a85a0 0x7f04c40a8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:15.410 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04b3fff700 1 -- 172.21.15.189:0/1718449227 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844617930 0 0) 0x7f04c40a0f60 con 0x7f04c41339a0 2020-11-20T12:08:15.410 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04b3fff700 1 -- 172.21.15.189:0/1718449227 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04ac003240 con 0x7f04c41339a0 2020-11-20T12:08:15.413 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04b3fff700 1 -- 172.21.15.189:0/1718449227 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 878438282 0 0) 0x7f04ac003240 con 0x7f04c41339a0 2020-11-20T12:08:15.413 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04b3fff700 1 -- 172.21.15.189:0/1718449227 >> v2:172.21.15.189:3300/0 conn(0x7f04c40612b0 msgr2=0x7f04c40a0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:15.413 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04b3fff700 1 --2- 172.21.15.189:0/1718449227 >> v2:172.21.15.189:3300/0 conn(0x7f04c40612b0 0x7f04c40a0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:15.413 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04b3fff700 1 -- 172.21.15.189:0/1718449227 >> v2:172.21.15.47:3300/0 conn(0x7f04c40a85a0 msgr2=0x7f04c40a8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:15.413 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04b3fff700 1 --2- 172.21.15.189:0/1718449227 >> v2:172.21.15.47:3300/0 conn(0x7f04c40a85a0 0x7f04c40a8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:15.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04b3fff700 1 -- 172.21.15.189:0/1718449227 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c4067d00 con 0x7f04c41339a0 2020-11-20T12:08:15.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04b3fff700 1 -- 172.21.15.189:0/1718449227 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f04bc001860 con 0x7f04c41339a0 2020-11-20T12:08:15.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04b3fff700 1 -- 172.21.15.189:0/1718449227 >> v1:172.21.15.189:6789/0 conn(0x7f04c41339a0 legacy=0x7f04c4133e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:15.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04b3fff700 1 --2- 172.21.15.189:0/1718449227 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04ac003730 0x7f04ac003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:15.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04b3fff700 1 --2- 172.21.15.189:0/1718449227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04ac0044e0 0x7f04ac004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:15.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04b3fff700 1 --2- 172.21.15.189:0/1718449227 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04ac004e50 0x7f04ac00c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:15.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04b3fff700 1 -- 172.21.15.189:0/1718449227 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c4134540 con 0x7f04ac004e50 2020-11-20T12:08:15.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04b3fff700 1 -- 172.21.15.189:0/1718449227 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c4068440 con 0x7f04ac0044e0 2020-11-20T12:08:15.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04b3fff700 1 -- 172.21.15.189:0/1718449227 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac004260 con 0x7f04ac003730 2020-11-20T12:08:15.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04c909e700 1 --2- 172.21.15.189:0/1718449227 >> v2:172.21.15.189:3300/0 conn(0x7f04c40612b0 0x7f04c40a0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:08:15.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04c889d700 1 --2- 172.21.15.189:0/1718449227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04ac0044e0 0x7f04ac004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:15.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04c989f700 1 --2- 172.21.15.189:0/1718449227 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04ac004e50 0x7f04ac00c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:15.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04c909e700 1 --2- 172.21.15.189:0/1718449227 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04ac003730 0x7f04ac003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:15.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04c889d700 1 -- 172.21.15.189:0/1718449227 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04ac003730 msgr2=0x7f04ac003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:15.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04c889d700 1 --2- 172.21.15.189:0/1718449227 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04ac003730 0x7f04ac003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:15.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04c889d700 1 -- 172.21.15.189:0/1718449227 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04ac004e50 msgr2=0x7f04ac00c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:15.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04c889d700 1 --2- 172.21.15.189:0/1718449227 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04ac004e50 0x7f04ac00c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:15.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04c889d700 1 -- 172.21.15.189:0/1718449227 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ac003240 con 0x7f04ac0044e0 2020-11-20T12:08:15.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04c989f700 1 -- 172.21.15.189:0/1718449227 reap_dead start 2020-11-20T12:08:15.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04c889d700 1 --2- 172.21.15.189:0/1718449227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04ac0044e0 0x7f04ac004960 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f04b400bc00 tx=0x7f04b4006df0).ready entity=mon.1 client_cookie=e6168b9f0bfa771c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:15.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04b3fff700 1 -- 172.21.15.189:0/1718449227 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b400b310 con 0x7f04ac0044e0 2020-11-20T12:08:15.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04b3fff700 1 -- 172.21.15.189:0/1718449227 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b400eb40 con 0x7f04ac0044e0 2020-11-20T12:08:15.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04b3fff700 1 -- 172.21.15.189:0/1718449227 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b400eec0 con 0x7f04ac0044e0 2020-11-20T12:08:15.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04cb302700 1 -- 172.21.15.189:0/1718449227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04ac0044e0 msgr2=0x7f04ac004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:15.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04cb302700 1 --2- 172.21.15.189:0/1718449227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04ac0044e0 0x7f04ac004960 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f04b400bc00 tx=0x7f04b4006df0).stop 2020-11-20T12:08:15.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04cb302700 1 -- 172.21.15.189:0/1718449227 shutdown_connections 2020-11-20T12:08:15.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04cb302700 1 --2- 172.21.15.189:0/1718449227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04ac0044e0 0x7f04ac004960 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:15.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04cb302700 1 -- 172.21.15.189:0/1718449227 >> 172.21.15.189:0/1718449227 conn(0x7f04c4065910 msgr2=0x7f04c4060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:15.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04cb302700 1 -- 172.21.15.189:0/1718449227 shutdown_connections 2020-11-20T12:08:15.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.406+0000 7f04cb302700 1 -- 172.21.15.189:0/1718449227 wait complete. 2020-11-20T12:08:15.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.410+0000 7f04cb302700 1 Processor -- start 2020-11-20T12:08:15.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.410+0000 7f04cb302700 1 -- start start 2020-11-20T12:08:15.423 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.410+0000 7f04cb302700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04ac0044e0 0x7f04c40a3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:15.423 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.410+0000 7f04cb302700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04c40a1460 0x7f04c40a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:15.423 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.410+0000 7f04cb302700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04c409f6b0 0x7f04c409faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:15.423 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.410+0000 7f04cb302700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c4137cf0 con 0x7f04c40a1460 2020-11-20T12:08:15.423 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.410+0000 7f04cb302700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c40a1dd0 con 0x7f04ac0044e0 2020-11-20T12:08:15.424 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.410+0000 7f04cb302700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c4145b90 con 0x7f04c409f6b0 2020-11-20T12:08:15.424 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.410+0000 7f04c909e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04ac0044e0 0x7f04c40a3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:15.424 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.410+0000 7f04c909e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04ac0044e0 0x7f04c40a3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49634/0 (socket says 172.21.15.189:49634) 2020-11-20T12:08:15.424 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.410+0000 7f04c909e700 1 -- 172.21.15.189:0/3240683748 learned_addr learned my addr 172.21.15.189:0/3240683748 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:08:15.425 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.410+0000 7f04c889d700 1 --2- 172.21.15.189:0/3240683748 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04c40a1460 0x7f04c40a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:15.425 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.410+0000 7f04c989f700 1 --2- 172.21.15.189:0/3240683748 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04c409f6b0 0x7f04c409faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:15.425 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.410+0000 7f04c909e700 1 -- 172.21.15.189:0/3240683748 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04c409f6b0 msgr2=0x7f04c409faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:15.425 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.410+0000 7f04c909e700 1 --2- 172.21.15.189:0/3240683748 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04c409f6b0 0x7f04c409faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:15.425 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.410+0000 7f04c909e700 1 -- 172.21.15.189:0/3240683748 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04c40a1460 msgr2=0x7f04c40a18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:15.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.410+0000 7f04c909e700 1 --2- 172.21.15.189:0/3240683748 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04c40a1460 0x7f04c40a18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:15.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.410+0000 7f04c909e700 1 -- 172.21.15.189:0/3240683748 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c4145e30 con 0x7f04ac0044e0 2020-11-20T12:08:15.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.410+0000 7f04c909e700 1 --2- 172.21.15.189:0/3240683748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04ac0044e0 0x7f04c40a3c90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f04c0010d20 tx=0x7f04c0008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:15.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.410+0000 7f04b17fa700 1 -- 172.21.15.189:0/3240683748 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c000ecc0 con 0x7f04ac0044e0 2020-11-20T12:08:15.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.410+0000 7f04b17fa700 1 -- 172.21.15.189:0/3240683748 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c000ee20 con 0x7f04ac0044e0 2020-11-20T12:08:15.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.410+0000 7f04cb302700 1 -- 172.21.15.189:0/3240683748 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c41460d0 con 0x7f04ac0044e0 2020-11-20T12:08:15.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.410+0000 7f04b17fa700 1 -- 172.21.15.189:0/3240683748 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c0014680 con 0x7f04ac0044e0 2020-11-20T12:08:15.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.410+0000 7f04cb302700 1 -- 172.21.15.189:0/3240683748 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c40a6650 con 0x7f04ac0044e0 2020-11-20T12:08:15.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.414+0000 7f04cb302700 1 -- 172.21.15.189:0/3240683748 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c41459d0 con 0x7f04ac0044e0 2020-11-20T12:08:15.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.414+0000 7f04b17fa700 1 -- 172.21.15.189:0/3240683748 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f04c0024070 con 0x7f04ac0044e0 2020-11-20T12:08:15.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.414+0000 7f04b17fa700 1 --2- 172.21.15.189:0/3240683748 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f049c0557c0 0x7f049c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:15.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.414+0000 7f04b17fa700 1 -- 172.21.15.189:0/3240683748 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f04c006f430 con 0x7f04ac0044e0 2020-11-20T12:08:15.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.414+0000 7f04c889d700 1 --2- 172.21.15.189:0/3240683748 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f049c0557c0 0x7f049c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:15.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.414+0000 7f04c889d700 1 --2- 172.21.15.189:0/3240683748 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f049c0557c0 0x7f049c057c80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f04c4066890 tx=0x7f04b4019000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:15.430 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.422+0000 7f04b17fa700 1 -- 172.21.15.189:0/3240683748 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f04c0045300 con 0x7f04ac0044e0 2020-11-20T12:08:15.626 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.622+0000 7f04cb302700 1 -- 172.21.15.189:0/3240683748 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04c40446d0 con 0x7f04ac0044e0 2020-11-20T12:08:15.626 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.622+0000 7f04b17fa700 1 -- 172.21.15.189:0/3240683748 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+2139 (secure 0 0 0) 0x7f04c0018020 con 0x7f04ac0044e0 2020-11-20T12:08:15.627 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:08:15.627 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4849,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1375880901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1375880901},{"type":"v1","addr":"172.21.15.189:6835","nonce":1375880901}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:06:55.583738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4248},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"b","rank":0,"incarnation":7,"state":"up:active","state_seq":34,"addr":"172.21.15.47:6833/2660417436","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":2660417436},{"type":"v1","addr":"172.21.15.47:6833","nonce":2660417436}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:08:15.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.626+0000 7f04cb302700 1 -- 172.21.15.189:0/3240683748 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f049c0557c0 msgr2=0x7f049c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:15.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.626+0000 7f04cb302700 1 --2- 172.21.15.189:0/3240683748 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f049c0557c0 0x7f049c057c80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f04c4066890 tx=0x7f04b4019000).stop 2020-11-20T12:08:15.630 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.626+0000 7f04cb302700 1 -- 172.21.15.189:0/3240683748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04ac0044e0 msgr2=0x7f04c40a3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:15.630 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.626+0000 7f04cb302700 1 --2- 172.21.15.189:0/3240683748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04ac0044e0 0x7f04c40a3c90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f04c0010d20 tx=0x7f04c0008040).stop 2020-11-20T12:08:15.630 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.626+0000 7f04cb302700 1 -- 172.21.15.189:0/3240683748 shutdown_connections 2020-11-20T12:08:15.630 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.626+0000 7f04cb302700 1 --2- 172.21.15.189:0/3240683748 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04c409f6b0 0x7f04c409faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:15.630 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.626+0000 7f04cb302700 1 --2- 172.21.15.189:0/3240683748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04ac0044e0 0x7f04c40a3c90 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:15.631 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.626+0000 7f04cb302700 1 --2- 172.21.15.189:0/3240683748 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f049c0557c0 0x7f049c057c80 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:15.631 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.626+0000 7f04cb302700 1 --2- 172.21.15.189:0/3240683748 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04c40a1460 0x7f04c40a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:15.631 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.626+0000 7f04cb302700 1 -- 172.21.15.189:0/3240683748 >> 172.21.15.189:0/3240683748 conn(0x7f04c4065910 msgr2=0x7f04c40a9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:15.631 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.626+0000 7f04cb302700 1 -- 172.21.15.189:0/3240683748 shutdown_connections 2020-11-20T12:08:15.631 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:15.626+0000 7f04cb302700 1 -- 172.21.15.189:0/3240683748 wait complete. 2020-11-20T12:08:15.632 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 11 2020-11-20T12:08:15.642 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:08:17.643 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:08:17.772 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662f4d5700 1 Processor -- start 2020-11-20T12:08:17.772 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662f4d5700 1 -- start start 2020-11-20T12:08:17.774 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662f4d5700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f66280ad000 0x7f662813e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:17.774 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662f4d5700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f66280ad3e0 0x7f6628066d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:17.774 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662f4d5700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6628067730 con 0x7f662813a6e0 2020-11-20T12:08:17.774 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662f4d5700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6628138190 con 0x7f66280ad000 2020-11-20T12:08:17.774 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662f4d5700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f662813dd40 con 0x7f66280ad3e0 2020-11-20T12:08:17.775 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662ecd4700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f662813a6e0 0x7f662813ad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51274/0 (socket says 172.21.15.189:51274) 2020-11-20T12:08:17.775 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662ecd4700 1 -- 172.21.15.189:0/2028504885 learned_addr learned my addr 172.21.15.189:0/2028504885 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:08:17.775 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662e4d3700 1 --2- 172.21.15.189:0/2028504885 >> v2:172.21.15.47:3300/0 conn(0x7f66280ad000 0x7f662813e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:17.775 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662dcd2700 1 --2- 172.21.15.189:0/2028504885 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f66280ad3e0 0x7f6628066d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:17.775 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662d4d1700 1 -- 172.21.15.189:0/2028504885 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2004563020 0 0) 0x7f6628067730 con 0x7f662813a6e0 2020-11-20T12:08:17.776 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662d4d1700 1 -- 172.21.15.189:0/2028504885 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6610003240 con 0x7f662813a6e0 2020-11-20T12:08:17.785 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662dcd2700 1 -- 172.21.15.189:0/2028504885 >> v2:172.21.15.47:3300/0 conn(0x7f66280ad000 msgr2=0x7f662813e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:17.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662dcd2700 1 --2- 172.21.15.189:0/2028504885 >> v2:172.21.15.47:3300/0 conn(0x7f66280ad000 0x7f662813e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:17.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662dcd2700 1 -- 172.21.15.189:0/2028504885 >> v1:172.21.15.189:6789/0 conn(0x7f662813a6e0 legacy=0x7f662813ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:17.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662dcd2700 1 -- 172.21.15.189:0/2028504885 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662813dfe0 con 0x7f66280ad3e0 2020-11-20T12:08:17.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662dcd2700 1 --2- 172.21.15.189:0/2028504885 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f66280ad3e0 0x7f6628066d40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f661800b9a0 tx=0x7f6618006df0).ready entity=mon.2 client_cookie=ed2eab25f115b069 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:17.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662d4d1700 1 -- 172.21.15.189:0/2028504885 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6618009680 con 0x7f66280ad3e0 2020-11-20T12:08:17.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662d4d1700 1 -- 172.21.15.189:0/2028504885 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66180097e0 con 0x7f66280ad3e0 2020-11-20T12:08:17.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662d4d1700 1 -- 172.21.15.189:0/2028504885 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6618009b60 con 0x7f66280ad3e0 2020-11-20T12:08:17.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662f4d5700 1 -- 172.21.15.189:0/2028504885 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f66280ad3e0 msgr2=0x7f6628066d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:17.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662f4d5700 1 --2- 172.21.15.189:0/2028504885 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f66280ad3e0 0x7f6628066d40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f661800b9a0 tx=0x7f6618006df0).stop 2020-11-20T12:08:17.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662f4d5700 1 -- 172.21.15.189:0/2028504885 shutdown_connections 2020-11-20T12:08:17.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662f4d5700 1 --2- 172.21.15.189:0/2028504885 >> v2:172.21.15.47:3300/0 conn(0x7f66280ad000 0x7f662813e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:17.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662f4d5700 1 --2- 172.21.15.189:0/2028504885 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f66280ad3e0 0x7f6628066d40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:17.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662f4d5700 1 -- 172.21.15.189:0/2028504885 >> 172.21.15.189:0/2028504885 conn(0x7f6628062080 msgr2=0x7f6628062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:17.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662f4d5700 1 -- 172.21.15.189:0/2028504885 shutdown_connections 2020-11-20T12:08:17.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.770+0000 7f662f4d5700 1 -- 172.21.15.189:0/2028504885 wait complete. 2020-11-20T12:08:17.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.774+0000 7f662f4d5700 1 Processor -- start 2020-11-20T12:08:17.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.774+0000 7f662f4d5700 1 -- start start 2020-11-20T12:08:17.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.774+0000 7f662f4d5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66280ad000 0x7f662813c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:17.790 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.774+0000 7f662f4d5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66280a62d0 0x7f662814ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:17.790 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.774+0000 7f662f4d5700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f66280aaa20 0x7f6628149b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:17.790 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.774+0000 7f662f4d5700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662814d700 con 0x7f66280a62d0 2020-11-20T12:08:17.790 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.774+0000 7f662f4d5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662814d9a0 con 0x7f66280ad000 2020-11-20T12:08:17.790 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.774+0000 7f662f4d5700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f662814dc40 con 0x7f66280aaa20 2020-11-20T12:08:17.791 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.774+0000 7f662e4d3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66280ad000 0x7f662813c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:17.791 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.774+0000 7f662e4d3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66280ad000 0x7f662813c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49660/0 (socket says 172.21.15.189:49660) 2020-11-20T12:08:17.791 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.774+0000 7f662e4d3700 1 -- 172.21.15.189:0/1917628351 learned_addr learned my addr 172.21.15.189:0/1917628351 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:08:17.791 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.774+0000 7f662dcd2700 1 --2- 172.21.15.189:0/1917628351 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66280a62d0 0x7f662814ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:17.791 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.774+0000 7f662ecd4700 1 --2- 172.21.15.189:0/1917628351 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f66280aaa20 0x7f6628149b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:17.792 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.774+0000 7f662e4d3700 1 -- 172.21.15.189:0/1917628351 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f66280aaa20 msgr2=0x7f6628149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:17.792 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.774+0000 7f662e4d3700 1 --2- 172.21.15.189:0/1917628351 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f66280aaa20 0x7f6628149b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:17.792 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.774+0000 7f662e4d3700 1 -- 172.21.15.189:0/1917628351 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66280a62d0 msgr2=0x7f662814ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:17.792 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.774+0000 7f662e4d3700 1 --2- 172.21.15.189:0/1917628351 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66280a62d0 0x7f662814ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:17.792 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.774+0000 7f662e4d3700 1 -- 172.21.15.189:0/1917628351 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662814d3c0 con 0x7f66280ad000 2020-11-20T12:08:17.793 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.774+0000 7f662e4d3700 1 --2- 172.21.15.189:0/1917628351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66280ad000 0x7f662813c690 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f662400ef50 tx=0x7f66240091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:17.793 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.774+0000 7f66167fc700 1 -- 172.21.15.189:0/1917628351 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662400cc10 con 0x7f66280ad000 2020-11-20T12:08:17.793 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.774+0000 7f66167fc700 1 -- 172.21.15.189:0/1917628351 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f662400cd70 con 0x7f66280ad000 2020-11-20T12:08:17.793 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.774+0000 7f662f4d5700 1 -- 172.21.15.189:0/1917628351 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66280aaea0 con 0x7f66280ad000 2020-11-20T12:08:17.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.774+0000 7f66167fc700 1 -- 172.21.15.189:0/1917628351 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6624012600 con 0x7f66280ad000 2020-11-20T12:08:17.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.774+0000 7f662f4d5700 1 -- 172.21.15.189:0/1917628351 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f662814a210 con 0x7f66280ad000 2020-11-20T12:08:17.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.778+0000 7f662f4d5700 1 -- 172.21.15.189:0/1917628351 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f662813cb80 con 0x7f66280ad000 2020-11-20T12:08:17.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.778+0000 7f66167fc700 1 -- 172.21.15.189:0/1917628351 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f6624022070 con 0x7f66280ad000 2020-11-20T12:08:17.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.778+0000 7f66167fc700 1 --2- 172.21.15.189:0/1917628351 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f66000557c0 0x7f6600057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:17.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.778+0000 7f66167fc700 1 -- 172.21.15.189:0/1917628351 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f662406ee90 con 0x7f66280ad000 2020-11-20T12:08:17.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.778+0000 7f662dcd2700 1 --2- 172.21.15.189:0/1917628351 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f66000557c0 0x7f6600057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:17.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.778+0000 7f662dcd2700 1 --2- 172.21.15.189:0/1917628351 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f66000557c0 0x7f6600057c80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f66280ab420 tx=0x7f6618015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:17.796 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.786+0000 7f66167fc700 1 -- 172.21.15.189:0/1917628351 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6624044c80 con 0x7f66280ad000 2020-11-20T12:08:17.991 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.986+0000 7f662f4d5700 1 -- 172.21.15.189:0/1917628351 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f662814d550 con 0x7f66280ad000 2020-11-20T12:08:17.992 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.990+0000 7f66167fc700 1 -- 172.21.15.189:0/1917628351 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+1787 (secure 0 0 0) 0x7f6624016070 con 0x7f66280ad000 2020-11-20T12:08:17.992 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:08:17.992 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:08:17.533277+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"a","rank":0,"incarnation":12,"state":"up:replay","state_seq":1,"addr":"172.21.15.189:6835/1375880901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1375880901},{"type":"v1","addr":"172.21.15.189:6835","nonce":1375880901}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:08:17.994 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.990+0000 7f662f4d5700 1 -- 172.21.15.189:0/1917628351 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f66000557c0 msgr2=0x7f6600057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:17.994 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.990+0000 7f662f4d5700 1 --2- 172.21.15.189:0/1917628351 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f66000557c0 0x7f6600057c80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f66280ab420 tx=0x7f6618015000).stop 2020-11-20T12:08:17.995 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.990+0000 7f662f4d5700 1 -- 172.21.15.189:0/1917628351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66280ad000 msgr2=0x7f662813c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:17.995 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.990+0000 7f662f4d5700 1 --2- 172.21.15.189:0/1917628351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66280ad000 0x7f662813c690 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f662400ef50 tx=0x7f66240091d0).stop 2020-11-20T12:08:17.995 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.990+0000 7f662f4d5700 1 -- 172.21.15.189:0/1917628351 shutdown_connections 2020-11-20T12:08:17.995 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.990+0000 7f662f4d5700 1 --2- 172.21.15.189:0/1917628351 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f66280aaa20 0x7f6628149b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:17.995 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.990+0000 7f662f4d5700 1 --2- 172.21.15.189:0/1917628351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66280ad000 0x7f662813c690 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:17.996 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.990+0000 7f662f4d5700 1 --2- 172.21.15.189:0/1917628351 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f66000557c0 0x7f6600057c80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:17.996 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.990+0000 7f662f4d5700 1 --2- 172.21.15.189:0/1917628351 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66280a62d0 0x7f662814ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:17.996 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.990+0000 7f662f4d5700 1 -- 172.21.15.189:0/1917628351 >> 172.21.15.189:0/1917628351 conn(0x7f6628062080 msgr2=0x7f66280ad7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:17.996 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.990+0000 7f662f4d5700 1 -- 172.21.15.189:0/1917628351 shutdown_connections 2020-11-20T12:08:17.996 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:17.990+0000 7f662f4d5700 1 -- 172.21.15.189:0/1917628351 wait complete. 2020-11-20T12:08:17.997 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 12 2020-11-20T12:08:18.007 INFO:tasks.mds_thrash.fs.[None]:mds.b down, removed from mdsmap 2020-11-20T12:08:18.007 INFO:tasks.mds_thrash.fs.[None]:waiting for mds cluster to stabilize... 2020-11-20T12:08:18.008 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:08:18.146 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.134+0000 7f16c272d700 1 Processor -- start 2020-11-20T12:08:18.146 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.134+0000 7f16c272d700 1 -- start start 2020-11-20T12:08:18.147 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.134+0000 7f16c272d700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f16bc0a9ed0 0x7f16bc133b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:18.147 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.134+0000 7f16c272d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f16bc134020 0x7f16bc1344d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:18.147 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.134+0000 7f16c272d700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16bc13ad10 con 0x7f16bc0ae190 2020-11-20T12:08:18.147 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.134+0000 7f16c272d700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16bc1349c0 con 0x7f16bc0a9ed0 2020-11-20T12:08:18.147 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.134+0000 7f16c272d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16bc067920 con 0x7f16bc134020 2020-11-20T12:08:18.148 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.134+0000 7f16c1f2c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f16bc134020 0x7f16bc1344d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:18.148 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.134+0000 7f16c0f2a700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f16bc0a9ed0 0x7f16bc133b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:18.148 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.134+0000 7f16c1f2c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f16bc134020 0x7f16bc1344d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49668/0 (socket says 172.21.15.189:49668) 2020-11-20T12:08:18.148 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.134+0000 7f16c1f2c700 1 -- 172.21.15.189:0/1361746853 learned_addr learned my addr 172.21.15.189:0/1361746853 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:08:18.148 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.138+0000 7f16c1f2c700 1 -- 172.21.15.189:0/1361746853 >> v2:172.21.15.47:3300/0 conn(0x7f16bc0a9ed0 msgr2=0x7f16bc133b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:18.149 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.138+0000 7f16c1f2c700 1 --2- 172.21.15.189:0/1361746853 >> v2:172.21.15.47:3300/0 conn(0x7f16bc0a9ed0 0x7f16bc133b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:18.149 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.138+0000 7f16c1f2c700 1 -- 172.21.15.189:0/1361746853 >> v1:172.21.15.47:6789/0 conn(0x7f16bc0ae190 legacy=0x7f16bc13a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:18.149 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.138+0000 7f16c1f2c700 1 -- 172.21.15.189:0/1361746853 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16bc067bc0 con 0x7f16bc134020 2020-11-20T12:08:18.149 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.138+0000 7f16c1f2c700 1 --2- 172.21.15.189:0/1361746853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16bc134020 0x7f16bc1344d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f16bc134c10 tx=0x7f16b4009290).ready entity=mon.1 client_cookie=975757e7713d4633 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:18.150 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.138+0000 7f16b3fff700 1 -- 172.21.15.189:0/1361746853 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b400bc50 con 0x7f16bc134020 2020-11-20T12:08:18.150 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.138+0000 7f16b3fff700 1 -- 172.21.15.189:0/1361746853 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b400bdb0 con 0x7f16bc134020 2020-11-20T12:08:18.150 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.138+0000 7f16b3fff700 1 -- 172.21.15.189:0/1361746853 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b4011610 con 0x7f16bc134020 2020-11-20T12:08:18.150 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.138+0000 7f16c272d700 1 -- 172.21.15.189:0/1361746853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16bc134020 msgr2=0x7f16bc1344d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:18.150 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.138+0000 7f16c272d700 1 --2- 172.21.15.189:0/1361746853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16bc134020 0x7f16bc1344d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f16bc134c10 tx=0x7f16b4009290).stop 2020-11-20T12:08:18.151 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.138+0000 7f16c272d700 1 -- 172.21.15.189:0/1361746853 shutdown_connections 2020-11-20T12:08:18.151 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.138+0000 7f16c272d700 1 --2- 172.21.15.189:0/1361746853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16bc134020 0x7f16bc1344d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:18.151 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.138+0000 7f16c272d700 1 --2- 172.21.15.189:0/1361746853 >> v2:172.21.15.47:3300/0 conn(0x7f16bc0a9ed0 0x7f16bc133b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:18.152 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.138+0000 7f16c272d700 1 -- 172.21.15.189:0/1361746853 >> 172.21.15.189:0/1361746853 conn(0x7f16bc065fe0 msgr2=0x7f16bc0ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:18.152 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.138+0000 7f16c272d700 1 -- 172.21.15.189:0/1361746853 shutdown_connections 2020-11-20T12:08:18.152 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.138+0000 7f16c272d700 1 -- 172.21.15.189:0/1361746853 wait complete. 2020-11-20T12:08:18.152 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.142+0000 7f16c272d700 1 Processor -- start 2020-11-20T12:08:18.152 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.142+0000 7f16c272d700 1 -- start start 2020-11-20T12:08:18.153 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.142+0000 7f16c272d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16bc0a9ed0 0x7f16bc135c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:18.153 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.142+0000 7f16c272d700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f16bc0ae190 0x7f16bc149660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:18.153 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.142+0000 7f16c272d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16bc134020 0x7f16bc149b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:18.153 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.142+0000 7f16c272d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc14db20 con 0x7f16bc0a9ed0 2020-11-20T12:08:18.154 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.142+0000 7f16c272d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc14d140 con 0x7f16bc134020 2020-11-20T12:08:18.154 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.142+0000 7f16c272d700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc14d3e0 con 0x7f16bc0ae190 2020-11-20T12:08:18.154 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.142+0000 7f16c1f2c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16bc134020 0x7f16bc149b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:18.154 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.142+0000 7f16c1f2c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16bc134020 0x7f16bc149b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49672/0 (socket says 172.21.15.189:49672) 2020-11-20T12:08:18.154 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.142+0000 7f16c1f2c700 1 -- 172.21.15.189:0/2786314952 learned_addr learned my addr 172.21.15.189:0/2786314952 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:08:18.154 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.142+0000 7f16c172b700 1 --2- 172.21.15.189:0/2786314952 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16bc0a9ed0 0x7f16bc135c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:18.155 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.142+0000 7f16c0f2a700 1 --2- 172.21.15.189:0/2786314952 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f16bc0ae190 0x7f16bc149660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:18.157 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.142+0000 7f16c1f2c700 1 -- 172.21.15.189:0/2786314952 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f16bc0ae190 msgr2=0x7f16bc149660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:18.157 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.142+0000 7f16c1f2c700 1 --2- 172.21.15.189:0/2786314952 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f16bc0ae190 0x7f16bc149660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:18.157 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.142+0000 7f16c1f2c700 1 -- 172.21.15.189:0/2786314952 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16bc0a9ed0 msgr2=0x7f16bc135c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:18.157 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.142+0000 7f16c1f2c700 1 --2- 172.21.15.189:0/2786314952 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16bc0a9ed0 0x7f16bc135c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:18.158 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.142+0000 7f16c1f2c700 1 -- 172.21.15.189:0/2786314952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16bc14d680 con 0x7f16bc134020 2020-11-20T12:08:18.158 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.142+0000 7f16c1f2c700 1 --2- 172.21.15.189:0/2786314952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16bc134020 0x7f16bc149b50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f16b4007fd0 tx=0x7f16b4016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:18.158 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.142+0000 7f16b17fa700 1 -- 172.21.15.189:0/2786314952 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b400bc50 con 0x7f16bc134020 2020-11-20T12:08:18.158 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.142+0000 7f16b17fa700 1 -- 172.21.15.189:0/2786314952 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b4010aa0 con 0x7f16bc134020 2020-11-20T12:08:18.158 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.142+0000 7f16c272d700 1 -- 172.21.15.189:0/2786314952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16bc1363c0 con 0x7f16bc134020 2020-11-20T12:08:18.159 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.142+0000 7f16b17fa700 1 -- 172.21.15.189:0/2786314952 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b4010e20 con 0x7f16bc134020 2020-11-20T12:08:18.159 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.142+0000 7f16c272d700 1 -- 172.21.15.189:0/2786314952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16bc14ce50 con 0x7f16bc134020 2020-11-20T12:08:18.159 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.146+0000 7f16c272d700 1 -- 172.21.15.189:0/2786314952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16bc14d810 con 0x7f16bc134020 2020-11-20T12:08:18.159 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.146+0000 7f16b17fa700 1 -- 172.21.15.189:0/2786314952 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f16b4017070 con 0x7f16bc134020 2020-11-20T12:08:18.159 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.146+0000 7f16b17fa700 1 --2- 172.21.15.189:0/2786314952 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f16940557f0 0x7f1694057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:18.160 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.146+0000 7f16b17fa700 1 -- 172.21.15.189:0/2786314952 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f16b4070800 con 0x7f16bc134020 2020-11-20T12:08:18.160 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.146+0000 7f16c172b700 1 --2- 172.21.15.189:0/2786314952 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f16940557f0 0x7f1694057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:18.160 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.146+0000 7f16c172b700 1 --2- 172.21.15.189:0/2786314952 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f16940557f0 0x7f1694057cb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f16b80035f0 tx=0x7f16b8007b90).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:18.160 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.154+0000 7f16b17fa700 1 -- 172.21.15.189:0/2786314952 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f16b40464e0 con 0x7f16bc134020 2020-11-20T12:08:18.356 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.354+0000 7f16c272d700 1 -- 172.21.15.189:0/2786314952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16bc13aaf0 con 0x7f16bc134020 2020-11-20T12:08:18.356 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.354+0000 7f16b17fa700 1 -- 172.21.15.189:0/2786314952 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+1787 (secure 0 0 0) 0x7f16b4015090 con 0x7f16bc134020 2020-11-20T12:08:18.357 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:08:18.357 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:08:17.533277+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"a","rank":0,"incarnation":12,"state":"up:replay","state_seq":1,"addr":"172.21.15.189:6835/1375880901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1375880901},{"type":"v1","addr":"172.21.15.189:6835","nonce":1375880901}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:08:18.359 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.354+0000 7f16c272d700 1 -- 172.21.15.189:0/2786314952 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f16940557f0 msgr2=0x7f1694057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:18.360 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.354+0000 7f16c272d700 1 --2- 172.21.15.189:0/2786314952 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f16940557f0 0x7f1694057cb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f16b80035f0 tx=0x7f16b8007b90).stop 2020-11-20T12:08:18.360 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.354+0000 7f16c272d700 1 -- 172.21.15.189:0/2786314952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16bc134020 msgr2=0x7f16bc149b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:18.360 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.354+0000 7f16c272d700 1 --2- 172.21.15.189:0/2786314952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16bc134020 0x7f16bc149b50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f16b4007fd0 tx=0x7f16b4016000).stop 2020-11-20T12:08:18.360 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.358+0000 7f16c272d700 1 -- 172.21.15.189:0/2786314952 shutdown_connections 2020-11-20T12:08:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.358+0000 7f16c272d700 1 --2- 172.21.15.189:0/2786314952 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f16940557f0 0x7f1694057cb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.358+0000 7f16c272d700 1 --2- 172.21.15.189:0/2786314952 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f16bc0a9ed0 0x7f16bc135c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.358+0000 7f16c272d700 1 --2- 172.21.15.189:0/2786314952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16bc134020 0x7f16bc149b50 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:18.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.358+0000 7f16c272d700 1 --2- 172.21.15.189:0/2786314952 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f16bc0ae190 0x7f16bc149660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:18.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.358+0000 7f16c272d700 1 -- 172.21.15.189:0/2786314952 >> 172.21.15.189:0/2786314952 conn(0x7f16bc065fe0 msgr2=0x7f16bc0668d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:18.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.358+0000 7f16c272d700 1 -- 172.21.15.189:0/2786314952 shutdown_connections 2020-11-20T12:08:18.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:18.358+0000 7f16c272d700 1 -- 172.21.15.189:0/2786314952 wait complete. 2020-11-20T12:08:18.363 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 12 2020-11-20T12:08:18.379 INFO:tasks.mds_thrash.fs.[None]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 12, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2020-11-20T12:04:54.012110+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 12, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4849": { "addr": "172.21.15.189:6835/1375880901", "addrs": { "addrvec": [ { "addr": "172.21.15.189:6834", "nonce": 1375880901, "type": "v2" }, { "addr": "172.21.15.189:6835", "nonce": 1375880901, "type": "v1" } ] }, "export_targets": [], "features": 4540138297136906239, "flags": 0, "gid": 4849, "incarnation": 12, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 24, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2020-11-20T12:08:17.533277+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4849 } } } ], "standbys": [] } 2020-11-20T12:08:20.380 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:08:20.509 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.506+0000 7fb37aa4a700 1 Processor -- start 2020-11-20T12:08:20.510 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.506+0000 7fb37aa4a700 1 -- start start 2020-11-20T12:08:20.511 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.506+0000 7fb37aa4a700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb3740a8320 0x7fb374066a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:20.512 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.506+0000 7fb37aa4a700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb3740a8700 0x7fb37413a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:20.512 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.506+0000 7fb37aa4a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb374068520 0x7fb3740a6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:20.512 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.506+0000 7fb37aa4a700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3740664c0 con 0x7fb3740a8700 2020-11-20T12:08:20.512 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.506+0000 7fb37aa4a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb374067200 con 0x7fb374068520 2020-11-20T12:08:20.512 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.506+0000 7fb37aa4a700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb374066110 con 0x7fb3740a8320 2020-11-20T12:08:20.513 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.506+0000 7fb37a249700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb374068520 0x7fb3740a6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:20.513 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.506+0000 7fb37a249700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb374068520 0x7fb3740a6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49684/0 (socket says 172.21.15.189:49684) 2020-11-20T12:08:20.513 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.506+0000 7fb379a48700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb3740a8320 0x7fb374066a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:20.513 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.506+0000 7fb37a249700 1 -- 172.21.15.189:0/2155872626 learned_addr learned my addr 172.21.15.189:0/2155872626 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:08:20.513 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.506+0000 7fb379247700 1 --2- 172.21.15.189:0/2155872626 >> v2:172.21.15.47:3300/0 conn(0x7fb3740a8700 0x7fb37413a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:20.514 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.510+0000 7fb37a249700 1 -- 172.21.15.189:0/2155872626 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb3740a8320 msgr2=0x7fb374066a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:20.514 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.510+0000 7fb37a249700 1 --2- 172.21.15.189:0/2155872626 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb3740a8320 0x7fb374066a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:20.514 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.510+0000 7fb37a249700 1 -- 172.21.15.189:0/2155872626 >> v2:172.21.15.47:3300/0 conn(0x7fb3740a8700 msgr2=0x7fb37413a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:20.514 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.510+0000 7fb37a249700 1 --2- 172.21.15.189:0/2155872626 >> v2:172.21.15.47:3300/0 conn(0x7fb3740a8700 0x7fb37413a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:20.514 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.510+0000 7fb37a249700 1 -- 172.21.15.189:0/2155872626 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb374066f70 con 0x7fb374068520 2020-11-20T12:08:20.515 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.510+0000 7fb37a249700 1 --2- 172.21.15.189:0/2155872626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb374068520 0x7fb3740a6500 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fb3740a8e80 tx=0x7fb36c009290).ready entity=mon.1 client_cookie=f26cadc0b05785f2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:20.517 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.510+0000 7fb378a46700 1 -- 172.21.15.189:0/2155872626 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36c00bb10 con 0x7fb374068520 2020-11-20T12:08:20.518 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.510+0000 7fb378a46700 1 -- 172.21.15.189:0/2155872626 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb36c00bc70 con 0x7fb374068520 2020-11-20T12:08:20.518 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.510+0000 7fb378a46700 1 -- 172.21.15.189:0/2155872626 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36c011400 con 0x7fb374068520 2020-11-20T12:08:20.518 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.510+0000 7fb37aa4a700 1 -- 172.21.15.189:0/2155872626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb374068520 msgr2=0x7fb3740a6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:20.518 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.510+0000 7fb37aa4a700 1 --2- 172.21.15.189:0/2155872626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb374068520 0x7fb3740a6500 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fb3740a8e80 tx=0x7fb36c009290).stop 2020-11-20T12:08:20.519 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.510+0000 7fb37aa4a700 1 -- 172.21.15.189:0/2155872626 shutdown_connections 2020-11-20T12:08:20.519 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.510+0000 7fb37aa4a700 1 --2- 172.21.15.189:0/2155872626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb374068520 0x7fb3740a6500 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:20.519 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.510+0000 7fb37aa4a700 1 --2- 172.21.15.189:0/2155872626 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb3740a8320 0x7fb374066a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:20.519 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.510+0000 7fb37aa4a700 1 --2- 172.21.15.189:0/2155872626 >> v2:172.21.15.47:3300/0 conn(0x7fb3740a8700 0x7fb37413a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:20.520 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.510+0000 7fb37aa4a700 1 -- 172.21.15.189:0/2155872626 >> 172.21.15.189:0/2155872626 conn(0x7fb374060170 msgr2=0x7fb374060550 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:20.520 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.510+0000 7fb37aa4a700 1 -- 172.21.15.189:0/2155872626 shutdown_connections 2020-11-20T12:08:20.520 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.510+0000 7fb37aa4a700 1 -- 172.21.15.189:0/2155872626 wait complete. 2020-11-20T12:08:20.520 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.510+0000 7fb37aa4a700 1 Processor -- start 2020-11-20T12:08:20.521 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.510+0000 7fb37aa4a700 1 -- start start 2020-11-20T12:08:20.521 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.514+0000 7fb37aa4a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb374068520 0x7fb37413c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:20.521 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.514+0000 7fb37aa4a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3740a8320 0x7fb3740ab300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:20.521 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.514+0000 7fb37aa4a700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb3740a8700 0x7fb374133100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:20.521 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.514+0000 7fb37aa4a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37413c8e0 con 0x7fb374068520 2020-11-20T12:08:20.522 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.514+0000 7fb37aa4a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37413c640 con 0x7fb3740a8320 2020-11-20T12:08:20.522 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.514+0000 7fb37aa4a700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3740a17a0 con 0x7fb3740a8700 2020-11-20T12:08:20.522 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.514+0000 7fb379247700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3740a8320 0x7fb3740ab300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:20.522 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.514+0000 7fb379247700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3740a8320 0x7fb3740ab300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49688/0 (socket says 172.21.15.189:49688) 2020-11-20T12:08:20.522 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.514+0000 7fb379247700 1 -- 172.21.15.189:0/4108923073 learned_addr learned my addr 172.21.15.189:0/4108923073 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:08:20.523 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.514+0000 7fb379a48700 1 --2- 172.21.15.189:0/4108923073 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb374068520 0x7fb37413c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:20.523 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.514+0000 7fb37a249700 1 --2- 172.21.15.189:0/4108923073 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb3740a8700 0x7fb374133100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:20.523 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.514+0000 7fb379247700 1 -- 172.21.15.189:0/4108923073 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb3740a8700 msgr2=0x7fb374133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:20.523 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.514+0000 7fb379247700 1 --2- 172.21.15.189:0/4108923073 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb3740a8700 0x7fb374133100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:20.523 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.514+0000 7fb379247700 1 -- 172.21.15.189:0/4108923073 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb374068520 msgr2=0x7fb37413c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:20.524 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.514+0000 7fb379247700 1 --2- 172.21.15.189:0/4108923073 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb374068520 0x7fb37413c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:20.524 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.514+0000 7fb379247700 1 -- 172.21.15.189:0/4108923073 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb374134e60 con 0x7fb3740a8320 2020-11-20T12:08:20.524 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.514+0000 7fb379a48700 1 --2- 172.21.15.189:0/4108923073 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb374068520 0x7fb37413c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:08:20.524 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.514+0000 7fb379247700 1 --2- 172.21.15.189:0/4108923073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3740a8320 0x7fb3740ab300 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fb36400ebd0 tx=0x7fb364006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:20.524 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.514+0000 7fb361ffb700 1 -- 172.21.15.189:0/4108923073 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36400c970 con 0x7fb3740a8320 2020-11-20T12:08:20.525 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.514+0000 7fb361ffb700 1 -- 172.21.15.189:0/4108923073 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb36400cad0 con 0x7fb3740a8320 2020-11-20T12:08:20.525 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.514+0000 7fb37aa4a700 1 -- 172.21.15.189:0/4108923073 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3740a4ec0 con 0x7fb3740a8320 2020-11-20T12:08:20.525 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.514+0000 7fb361ffb700 1 -- 172.21.15.189:0/4108923073 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36400ce50 con 0x7fb3740a8320 2020-11-20T12:08:20.525 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.514+0000 7fb37aa4a700 1 -- 172.21.15.189:0/4108923073 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb374133790 con 0x7fb3740a8320 2020-11-20T12:08:20.526 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.514+0000 7fb37aa4a700 1 -- 172.21.15.189:0/4108923073 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3740a1960 con 0x7fb3740a8320 2020-11-20T12:08:20.526 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.514+0000 7fb361ffb700 1 -- 172.21.15.189:0/4108923073 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fb364012600 con 0x7fb3740a8320 2020-11-20T12:08:20.527 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.518+0000 7fb361ffb700 1 --2- 172.21.15.189:0/4108923073 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb34c0557c0 0x7fb34c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:20.527 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.518+0000 7fb361ffb700 1 -- 172.21.15.189:0/4108923073 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fb36406eb30 con 0x7fb3740a8320 2020-11-20T12:08:20.527 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.518+0000 7fb379a48700 1 --2- 172.21.15.189:0/4108923073 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb34c0557c0 0x7fb34c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:20.527 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.518+0000 7fb379a48700 1 --2- 172.21.15.189:0/4108923073 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb34c0557c0 0x7fb34c057c80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fb370006fd0 tx=0x7fb370006df0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:20.528 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.526+0000 7fb361ffb700 1 -- 172.21.15.189:0/4108923073 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb364044920 con 0x7fb3740a8320 2020-11-20T12:08:20.755 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.754+0000 7fb37aa4a700 1 -- 172.21.15.189:0/4108923073 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3740446d0 con 0x7fb3740a8320 2020-11-20T12:08:20.756 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.754+0000 7fb361ffb700 1 -- 172.21.15.189:0/4108923073 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+1788 (secure 0 0 0) 0x7fb364047f40 con 0x7fb3740a8320 2020-11-20T12:08:20.756 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:08:20.757 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:08:20.595791+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/1375880901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1375880901},{"type":"v1","addr":"172.21.15.189:6835","nonce":1375880901}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:08:20.758 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.754+0000 7fb37aa4a700 1 -- 172.21.15.189:0/4108923073 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb34c0557c0 msgr2=0x7fb34c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:20.759 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.754+0000 7fb37aa4a700 1 --2- 172.21.15.189:0/4108923073 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb34c0557c0 0x7fb34c057c80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fb370006fd0 tx=0x7fb370006df0).stop 2020-11-20T12:08:20.759 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.754+0000 7fb37aa4a700 1 -- 172.21.15.189:0/4108923073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3740a8320 msgr2=0x7fb3740ab300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:20.759 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.754+0000 7fb37aa4a700 1 --2- 172.21.15.189:0/4108923073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3740a8320 0x7fb3740ab300 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fb36400ebd0 tx=0x7fb364006df0).stop 2020-11-20T12:08:20.760 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.754+0000 7fb37aa4a700 1 -- 172.21.15.189:0/4108923073 shutdown_connections 2020-11-20T12:08:20.760 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.754+0000 7fb37aa4a700 1 --2- 172.21.15.189:0/4108923073 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb34c0557c0 0x7fb34c057c80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:20.760 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.754+0000 7fb37aa4a700 1 --2- 172.21.15.189:0/4108923073 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb374068520 0x7fb37413c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:20.760 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.754+0000 7fb37aa4a700 1 --2- 172.21.15.189:0/4108923073 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb3740a8700 0x7fb374133100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:20.761 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.754+0000 7fb37aa4a700 1 --2- 172.21.15.189:0/4108923073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3740a8320 0x7fb3740ab300 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:20.761 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.754+0000 7fb37aa4a700 1 -- 172.21.15.189:0/4108923073 >> 172.21.15.189:0/4108923073 conn(0x7fb374060170 msgr2=0x7fb3740a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:20.761 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.754+0000 7fb37aa4a700 1 -- 172.21.15.189:0/4108923073 shutdown_connections 2020-11-20T12:08:20.761 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.754+0000 7fb37aa4a700 1 -- 172.21.15.189:0/4108923073 wait complete. 2020-11-20T12:08:20.762 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 15 2020-11-20T12:08:20.772 INFO:tasks.mds_thrash.fs.[None]:mds.a has gained rank=0, replacing gid=4248 2020-11-20T12:08:20.772 INFO:tasks.mds_thrash.fs.[None]:waiting for 0 secs before reviving mds.b 2020-11-20T12:08:20.772 INFO:tasks.mds_thrash.fs.[None]:reviving mds.b 2020-11-20T12:08:20.773 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:08:20.773 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:08:20.775 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:08:20.776 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:08:20.853 INFO:tasks.ceph.mds.b.smithi047.stdout:starting mds.b at 2020-11-20T12:08:20.909 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.906+0000 7f428616a700 1 Processor -- start 2020-11-20T12:08:20.909 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.906+0000 7f428616a700 1 -- start start 2020-11-20T12:08:20.911 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.906+0000 7f428616a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4280060310 0x7f42800a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:20.911 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.906+0000 7f428616a700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f428013fc70 0x7f4280142060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:20.912 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.906+0000 7f428616a700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4280063b50 con 0x7f4280134460 2020-11-20T12:08:20.912 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.906+0000 7f428616a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4280067a00 con 0x7f4280060310 2020-11-20T12:08:20.912 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.906+0000 7f428616a700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4280066500 con 0x7f428013fc70 2020-11-20T12:08:20.912 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.906+0000 7f4285168700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4280060310 0x7f42800a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:20.912 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.906+0000 7f4285168700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4280060310 0x7f42800a11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49696/0 (socket says 172.21.15.189:49696) 2020-11-20T12:08:20.913 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.906+0000 7f4285168700 1 -- 172.21.15.189:0/1012162712 learned_addr learned my addr 172.21.15.189:0/1012162712 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:08:20.913 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.906+0000 7f4285969700 1 --2- 172.21.15.189:0/1012162712 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f428013fc70 0x7f4280142060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:20.913 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.906+0000 7f4285168700 1 -- 172.21.15.189:0/1012162712 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f428013fc70 msgr2=0x7f4280142060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:20.913 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.906+0000 7f4285168700 1 --2- 172.21.15.189:0/1012162712 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f428013fc70 0x7f4280142060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:20.913 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.906+0000 7f4285168700 1 -- 172.21.15.189:0/1012162712 >> v1:172.21.15.47:6789/0 conn(0x7f4280134460 legacy=0x7f42801348e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:20.914 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.906+0000 7f4285168700 1 -- 172.21.15.189:0/1012162712 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42800a09a0 con 0x7f4280060310 2020-11-20T12:08:20.914 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.906+0000 7f4285168700 1 --2- 172.21.15.189:0/1012162712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4280060310 0x7f42800a11c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f427c009d50 tx=0x7f427c009290).ready entity=mon.1 client_cookie=baf02d8170b938e0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:20.914 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.906+0000 7f426ffff700 1 -- 172.21.15.189:0/1012162712 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c00bc80 con 0x7f4280060310 2020-11-20T12:08:20.914 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.906+0000 7f426ffff700 1 -- 172.21.15.189:0/1012162712 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f427c00bde0 con 0x7f4280060310 2020-11-20T12:08:20.914 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.906+0000 7f426ffff700 1 -- 172.21.15.189:0/1012162712 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c011610 con 0x7f4280060310 2020-11-20T12:08:20.917 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.910+0000 7f428616a700 1 -- 172.21.15.189:0/1012162712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4280060310 msgr2=0x7f42800a11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:20.917 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.910+0000 7f428616a700 1 --2- 172.21.15.189:0/1012162712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4280060310 0x7f42800a11c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f427c009d50 tx=0x7f427c009290).stop 2020-11-20T12:08:20.917 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.910+0000 7f428616a700 1 -- 172.21.15.189:0/1012162712 shutdown_connections 2020-11-20T12:08:20.917 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.910+0000 7f428616a700 1 --2- 172.21.15.189:0/1012162712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4280060310 0x7f42800a11c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:20.917 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.910+0000 7f428616a700 1 --2- 172.21.15.189:0/1012162712 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f428013fc70 0x7f4280142060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:20.918 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.910+0000 7f428616a700 1 -- 172.21.15.189:0/1012162712 >> 172.21.15.189:0/1012162712 conn(0x7f4280063210 msgr2=0x7f4280060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:20.918 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.910+0000 7f428616a700 1 -- 172.21.15.189:0/1012162712 shutdown_connections 2020-11-20T12:08:20.918 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.910+0000 7f428616a700 1 -- 172.21.15.189:0/1012162712 wait complete. 2020-11-20T12:08:20.918 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.910+0000 7f428616a700 1 Processor -- start 2020-11-20T12:08:20.918 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.910+0000 7f428616a700 1 -- start start 2020-11-20T12:08:20.919 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.910+0000 7f428616a700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4280060310 0x7f4280135e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:20.919 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.910+0000 7f428616a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4280134460 0x7f4280149ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:20.919 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.910+0000 7f428616a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f428013fc70 0x7f42801369f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:20.919 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.910+0000 7f428616a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4280149fd0 con 0x7f428013fc70 2020-11-20T12:08:20.919 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.910+0000 7f428616a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4280135470 con 0x7f4280134460 2020-11-20T12:08:20.920 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.910+0000 7f428616a700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4280135710 con 0x7f4280060310 2020-11-20T12:08:20.920 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.910+0000 7f4284967700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4280134460 0x7f4280149ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:20.920 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.910+0000 7f4284967700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4280134460 0x7f4280149ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49704/0 (socket says 172.21.15.189:49704) 2020-11-20T12:08:20.920 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.910+0000 7f4284967700 1 -- 172.21.15.189:0/3346864875 learned_addr learned my addr 172.21.15.189:0/3346864875 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:08:20.921 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.910+0000 7f4285168700 1 --2- 172.21.15.189:0/3346864875 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4280060310 0x7f4280135e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:20.921 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.910+0000 7f4285969700 1 --2- 172.21.15.189:0/3346864875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f428013fc70 0x7f42801369f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:20.921 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.910+0000 7f4284967700 1 -- 172.21.15.189:0/3346864875 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4280060310 msgr2=0x7f4280135e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:20.921 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.910+0000 7f4284967700 1 --2- 172.21.15.189:0/3346864875 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4280060310 0x7f4280135e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:20.922 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.910+0000 7f4284967700 1 -- 172.21.15.189:0/3346864875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f428013fc70 msgr2=0x7f42801369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:20.922 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.910+0000 7f4284967700 1 --2- 172.21.15.189:0/3346864875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f428013fc70 0x7f42801369f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:20.922 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.910+0000 7f4284967700 1 -- 172.21.15.189:0/3346864875 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42801359b0 con 0x7f4280134460 2020-11-20T12:08:20.922 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.914+0000 7f4284967700 1 --2- 172.21.15.189:0/3346864875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4280134460 0x7f4280149ae0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f4270001e70 tx=0x7f4270007c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:20.922 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.914+0000 7f426d7fa700 1 -- 172.21.15.189:0/3346864875 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4270007020 con 0x7f4280134460 2020-11-20T12:08:20.923 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.914+0000 7f426d7fa700 1 -- 172.21.15.189:0/3346864875 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4270007180 con 0x7f4280134460 2020-11-20T12:08:20.923 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.914+0000 7f426d7fa700 1 -- 172.21.15.189:0/3346864875 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427000e560 con 0x7f4280134460 2020-11-20T12:08:20.923 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.914+0000 7f428616a700 1 -- 172.21.15.189:0/3346864875 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4280136560 con 0x7f4280134460 2020-11-20T12:08:20.923 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.914+0000 7f428616a700 1 -- 172.21.15.189:0/3346864875 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f428013f6a0 con 0x7f4280134460 2020-11-20T12:08:20.924 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.914+0000 7f426d7fa700 1 -- 172.21.15.189:0/3346864875 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f4270010070 con 0x7f4280134460 2020-11-20T12:08:20.925 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.914+0000 7f428616a700 1 -- 172.21.15.189:0/3346864875 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4280135b40 con 0x7f4280134460 2020-11-20T12:08:20.925 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.914+0000 7f426d7fa700 1 --2- 172.21.15.189:0/3346864875 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f42580557c0 0x7f4258057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:20.925 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.914+0000 7f426d7fa700 1 -- 172.21.15.189:0/3346864875 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f427006db00 con 0x7f4280134460 2020-11-20T12:08:20.925 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.914+0000 7f4285168700 1 --2- 172.21.15.189:0/3346864875 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f42580557c0 0x7f4258057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:20.925 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.914+0000 7f4285168700 1 --2- 172.21.15.189:0/3346864875 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f42580557c0 0x7f4258057c80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f427c009fd0 tx=0x7f427c016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:20.926 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:20.918+0000 7f426d7fa700 1 -- 172.21.15.189:0/3346864875 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4270042fd0 con 0x7f4280134460 2020-11-20T12:08:21.139 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:21.134+0000 7f428616a700 1 -- 172.21.15.189:0/3346864875 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42800446d0 con 0x7f4280134460 2020-11-20T12:08:21.140 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:21.138+0000 7f426d7fa700 1 -- 172.21.15.189:0/3346864875 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+1788 (secure 0 0 0) 0x7f42700438f0 con 0x7f4280134460 2020-11-20T12:08:21.140 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:08:21.141 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:08:20.595791+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/1375880901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1375880901},{"type":"v1","addr":"172.21.15.189:6835","nonce":1375880901}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:08:21.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:21.138+0000 7f428616a700 1 -- 172.21.15.189:0/3346864875 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f42580557c0 msgr2=0x7f4258057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:21.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:21.138+0000 7f428616a700 1 --2- 172.21.15.189:0/3346864875 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f42580557c0 0x7f4258057c80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f427c009fd0 tx=0x7f427c016000).stop 2020-11-20T12:08:21.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:21.138+0000 7f428616a700 1 -- 172.21.15.189:0/3346864875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4280134460 msgr2=0x7f4280149ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:21.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:21.138+0000 7f428616a700 1 --2- 172.21.15.189:0/3346864875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4280134460 0x7f4280149ae0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f4270001e70 tx=0x7f4270007c50).stop 2020-11-20T12:08:21.143 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:21.138+0000 7f428616a700 1 -- 172.21.15.189:0/3346864875 shutdown_connections 2020-11-20T12:08:21.143 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:21.138+0000 7f428616a700 1 --2- 172.21.15.189:0/3346864875 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f42580557c0 0x7f4258057c80 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:21.143 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:21.138+0000 7f428616a700 1 --2- 172.21.15.189:0/3346864875 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f428013fc70 0x7f42801369f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:21.143 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:21.138+0000 7f428616a700 1 --2- 172.21.15.189:0/3346864875 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4280060310 0x7f4280135e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:21.143 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:21.138+0000 7f428616a700 1 --2- 172.21.15.189:0/3346864875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4280134460 0x7f4280149ae0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:21.144 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:21.138+0000 7f428616a700 1 -- 172.21.15.189:0/3346864875 >> 172.21.15.189:0/3346864875 conn(0x7f4280063210 msgr2=0x7f42800a9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:21.144 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:21.138+0000 7f428616a700 1 -- 172.21.15.189:0/3346864875 shutdown_connections 2020-11-20T12:08:21.144 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:21.138+0000 7f428616a700 1 -- 172.21.15.189:0/3346864875 wait complete. 2020-11-20T12:08:21.145 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 15 2020-11-20T12:08:21.158 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2020-11-20T12:08:22.587 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:08:22.592 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:08:23.159 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:08:23.290 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.286+0000 7fa82af4c700 1 Processor -- start 2020-11-20T12:08:23.342 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.286+0000 7fa82af4c700 1 -- start start 2020-11-20T12:08:23.347 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.286+0000 7fa82af4c700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa8240a9a40 0x7fa8240a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:23.348 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.286+0000 7fa82af4c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa82413c080 0x7fa824142cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:23.348 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.286+0000 7fa82af4c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa824139270 con 0x7fa8240a9660 2020-11-20T12:08:23.348 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.286+0000 7fa82af4c700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8240a54c0 con 0x7fa8240a9a40 2020-11-20T12:08:23.348 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.286+0000 7fa82af4c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8240a9fb0 con 0x7fa82413c080 2020-11-20T12:08:23.349 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.286+0000 7fa8294e9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa82413c080 0x7fa824142cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:23.349 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.286+0000 7fa8294e9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa82413c080 0x7fa824142cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49714/0 (socket says 172.21.15.189:49714) 2020-11-20T12:08:23.350 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.286+0000 7fa8294e9700 1 -- 172.21.15.189:0/4176375301 learned_addr learned my addr 172.21.15.189:0/4176375301 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:08:23.350 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.286+0000 7fa823fff700 1 --2- 172.21.15.189:0/4176375301 >> v2:172.21.15.47:3300/0 conn(0x7fa8240a9a40 0x7fa8240a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:23.351 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.286+0000 7fa8294e9700 1 -- 172.21.15.189:0/4176375301 >> v2:172.21.15.47:3300/0 conn(0x7fa8240a9a40 msgr2=0x7fa8240a4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:23.354 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.286+0000 7fa8294e9700 1 --2- 172.21.15.189:0/4176375301 >> v2:172.21.15.47:3300/0 conn(0x7fa8240a9a40 0x7fa8240a4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:23.354 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.286+0000 7fa8294e9700 1 -- 172.21.15.189:0/4176375301 >> v1:172.21.15.189:6789/0 conn(0x7fa8240a9660 legacy=0x7fa824138b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:23.354 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.286+0000 7fa8294e9700 1 -- 172.21.15.189:0/4176375301 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8240a5750 con 0x7fa82413c080 2020-11-20T12:08:23.354 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.286+0000 7fa823fff700 1 --2- 172.21.15.189:0/4176375301 >> v2:172.21.15.47:3300/0 conn(0x7fa8240a9a40 0x7fa8240a4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:08:23.355 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.286+0000 7fa8294e9700 1 --2- 172.21.15.189:0/4176375301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa82413c080 0x7fa824142cc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fa81c009d50 tx=0x7fa81c009290).ready entity=mon.1 client_cookie=19e66f5e6799e909 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:23.355 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.286+0000 7fa8237fe700 1 -- 172.21.15.189:0/4176375301 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81c00bc40 con 0x7fa82413c080 2020-11-20T12:08:23.355 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa8237fe700 1 -- 172.21.15.189:0/4176375301 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa81c00bda0 con 0x7fa82413c080 2020-11-20T12:08:23.356 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa8237fe700 1 -- 172.21.15.189:0/4176375301 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81c011610 con 0x7fa82413c080 2020-11-20T12:08:23.356 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa82af4c700 1 -- 172.21.15.189:0/4176375301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa82413c080 msgr2=0x7fa824142cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:23.356 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa82af4c700 1 --2- 172.21.15.189:0/4176375301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa82413c080 0x7fa824142cc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fa81c009d50 tx=0x7fa81c009290).stop 2020-11-20T12:08:23.357 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa82af4c700 1 -- 172.21.15.189:0/4176375301 shutdown_connections 2020-11-20T12:08:23.357 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa82af4c700 1 --2- 172.21.15.189:0/4176375301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa82413c080 0x7fa824142cc0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:23.357 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa82af4c700 1 --2- 172.21.15.189:0/4176375301 >> v2:172.21.15.47:3300/0 conn(0x7fa8240a9a40 0x7fa8240a4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:23.358 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa82af4c700 1 -- 172.21.15.189:0/4176375301 >> 172.21.15.189:0/4176375301 conn(0x7fa824061f60 msgr2=0x7fa824062370 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:23.358 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa82af4c700 1 -- 172.21.15.189:0/4176375301 shutdown_connections 2020-11-20T12:08:23.358 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa82af4c700 1 -- 172.21.15.189:0/4176375301 wait complete. 2020-11-20T12:08:23.358 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa82af4c700 1 Processor -- start 2020-11-20T12:08:23.359 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa82af4c700 1 -- start start 2020-11-20T12:08:23.359 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa82af4c700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa8240a9660 0x7fa8240acae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:23.359 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa82af4c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8240a9a40 0x7fa824148b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:23.359 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa82af4c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa82413c080 0x7fa824149030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:23.360 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa82af4c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8240a3a90 con 0x7fa8240a9a40 2020-11-20T12:08:23.360 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa82af4c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa824149520 con 0x7fa82413c080 2020-11-20T12:08:23.360 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa82af4c700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8240a0920 con 0x7fa8240a9660 2020-11-20T12:08:23.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa8294e9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa82413c080 0x7fa824149030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:23.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa8294e9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa82413c080 0x7fa824149030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49720/0 (socket says 172.21.15.189:49720) 2020-11-20T12:08:23.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa8294e9700 1 -- 172.21.15.189:0/1830128539 learned_addr learned my addr 172.21.15.189:0/1830128539 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:08:23.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa823fff700 1 --2- 172.21.15.189:0/1830128539 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8240a9a40 0x7fa824148b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:23.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa828ce8700 1 --2- 172.21.15.189:0/1830128539 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa8240a9660 0x7fa8240acae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:23.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa8294e9700 1 -- 172.21.15.189:0/1830128539 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa8240a9660 msgr2=0x7fa8240acae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:23.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa8294e9700 1 --2- 172.21.15.189:0/1830128539 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa8240a9660 0x7fa8240acae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:23.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa8294e9700 1 -- 172.21.15.189:0/1830128539 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8240a9a40 msgr2=0x7fa824148b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:23.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa8294e9700 1 --2- 172.21.15.189:0/1830128539 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8240a9a40 0x7fa824148b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:23.366 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa8294e9700 1 -- 172.21.15.189:0/1830128539 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8240a0bc0 con 0x7fa82413c080 2020-11-20T12:08:23.366 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa8294e9700 1 --2- 172.21.15.189:0/1830128539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa82413c080 0x7fa824149030 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fa81c0109a0 tx=0x7fa81c016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:23.367 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa820ff9700 1 -- 172.21.15.189:0/1830128539 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81c00bc40 con 0x7fa82413c080 2020-11-20T12:08:23.367 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa820ff9700 1 -- 172.21.15.189:0/1830128539 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa81c011e60 con 0x7fa82413c080 2020-11-20T12:08:23.367 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa82af4c700 1 -- 172.21.15.189:0/1830128539 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8240a05a0 con 0x7fa82413c080 2020-11-20T12:08:23.368 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa820ff9700 1 -- 172.21.15.189:0/1830128539 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81c01c470 con 0x7fa82413c080 2020-11-20T12:08:23.368 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.290+0000 7fa82af4c700 1 -- 172.21.15.189:0/1830128539 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa824066e70 con 0x7fa82413c080 2020-11-20T12:08:23.368 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.294+0000 7fa82af4c700 1 -- 172.21.15.189:0/1830128539 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8241488a0 con 0x7fa82413c080 2020-11-20T12:08:23.369 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.294+0000 7fa820ff9700 1 -- 172.21.15.189:0/1830128539 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fa81c017070 con 0x7fa82413c080 2020-11-20T12:08:23.369 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.294+0000 7fa820ff9700 1 --2- 172.21.15.189:0/1830128539 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa7fc0557c0 0x7fa7fc057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:23.370 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.294+0000 7fa820ff9700 1 -- 172.21.15.189:0/1830128539 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fa81c070970 con 0x7fa82413c080 2020-11-20T12:08:23.370 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.294+0000 7fa828ce8700 1 --2- 172.21.15.189:0/1830128539 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa7fc0557c0 0x7fa7fc057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:23.371 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.294+0000 7fa828ce8700 1 --2- 172.21.15.189:0/1830128539 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa7fc0557c0 0x7fa7fc057c80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fa82413b940 tx=0x7fa818009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:23.371 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.302+0000 7fa820ff9700 1 -- 172.21.15.189:0/1830128539 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa8241488a0 con 0x7fa82413c080 2020-11-20T12:08:23.542 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.538+0000 7fa82af4c700 1 -- 172.21.15.189:0/1830128539 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8241488a0 con 0x7fa82413c080 2020-11-20T12:08:23.542 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.538+0000 7fa820ff9700 1 -- 172.21.15.189:0/1830128539 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+1788 (secure 0 0 0) 0x7fa8241488a0 con 0x7fa82413c080 2020-11-20T12:08:23.543 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:08:23.543 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:08:20.595791+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/1375880901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1375880901},{"type":"v1","addr":"172.21.15.189:6835","nonce":1375880901}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:08:23.546 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.542+0000 7fa82af4c700 1 -- 172.21.15.189:0/1830128539 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa7fc0557c0 msgr2=0x7fa7fc057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:23.546 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.542+0000 7fa82af4c700 1 --2- 172.21.15.189:0/1830128539 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa7fc0557c0 0x7fa7fc057c80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fa82413b940 tx=0x7fa818009040).stop 2020-11-20T12:08:23.546 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.542+0000 7fa82af4c700 1 -- 172.21.15.189:0/1830128539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa82413c080 msgr2=0x7fa824149030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:23.547 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.542+0000 7fa82af4c700 1 --2- 172.21.15.189:0/1830128539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa82413c080 0x7fa824149030 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fa81c0109a0 tx=0x7fa81c016000).stop 2020-11-20T12:08:23.547 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.542+0000 7fa82af4c700 1 -- 172.21.15.189:0/1830128539 shutdown_connections 2020-11-20T12:08:23.547 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.542+0000 7fa82af4c700 1 --2- 172.21.15.189:0/1830128539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa82413c080 0x7fa824149030 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:23.547 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.542+0000 7fa82af4c700 1 --2- 172.21.15.189:0/1830128539 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa8240a9660 0x7fa8240acae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:23.547 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.542+0000 7fa82af4c700 1 --2- 172.21.15.189:0/1830128539 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa7fc0557c0 0x7fa7fc057c80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:23.548 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.542+0000 7fa82af4c700 1 --2- 172.21.15.189:0/1830128539 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8240a9a40 0x7fa824148b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:23.548 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.542+0000 7fa82af4c700 1 -- 172.21.15.189:0/1830128539 >> 172.21.15.189:0/1830128539 conn(0x7fa824061f60 msgr2=0x7fa8240a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:23.548 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.542+0000 7fa82af4c700 1 -- 172.21.15.189:0/1830128539 shutdown_connections 2020-11-20T12:08:23.548 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:23.542+0000 7fa82af4c700 1 -- 172.21.15.189:0/1830128539 wait complete. 2020-11-20T12:08:23.549 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 15 2020-11-20T12:08:23.562 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2020-11-20T12:08:24.526 INFO:tasks.ceph.mds.a.smithi189.stderr:2020-11-20T12:08:24.522+0000 7f9cbb928700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:08:25.564 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:08:25.692 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.690+0000 7f8ae1405700 1 Processor -- start 2020-11-20T12:08:25.692 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.690+0000 7f8ae1405700 1 -- start start 2020-11-20T12:08:25.694 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.690+0000 7f8ae1405700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8adc060310 0x7f8adc0a1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:25.695 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.690+0000 7f8ae1405700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8adc0a2440 0x7f8adc0a7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:25.695 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.690+0000 7f8ae1405700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8adc0a8fa0 con 0x7f8adc0a8230 2020-11-20T12:08:25.695 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.690+0000 7f8ae1405700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8adc0618b0 con 0x7f8adc060310 2020-11-20T12:08:25.695 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.690+0000 7f8ae1405700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc0a9240 con 0x7f8adc0a2440 2020-11-20T12:08:25.695 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.690+0000 7f8ae0c04700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8adc0a8230 0x7f8adc135110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51350/0 (socket says 172.21.15.189:51350) 2020-11-20T12:08:25.696 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.690+0000 7f8ae0c04700 1 -- 172.21.15.189:0/963840818 learned_addr learned my addr 172.21.15.189:0/963840818 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:08:25.696 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.690+0000 7f8adbfff700 1 --2- 172.21.15.189:0/963840818 >> v2:172.21.15.47:3300/0 conn(0x7f8adc060310 0x7f8adc0a1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:25.696 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.690+0000 7f8adb7fe700 1 --2- 172.21.15.189:0/963840818 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8adc0a2440 0x7f8adc0a7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:25.696 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.690+0000 7f8adaffd700 1 -- 172.21.15.189:0/963840818 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1814029880 0 0) 0x7f8adc0a8fa0 con 0x7f8adc0a8230 2020-11-20T12:08:25.697 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.690+0000 7f8adaffd700 1 -- 172.21.15.189:0/963840818 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ac8003240 con 0x7f8adc0a8230 2020-11-20T12:08:25.697 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.690+0000 7f8adbfff700 1 -- 172.21.15.189:0/963840818 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8adc0a2440 msgr2=0x7f8adc0a7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:08:25.697 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.690+0000 7f8adbfff700 1 --2- 172.21.15.189:0/963840818 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8adc0a2440 0x7f8adc0a7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:25.697 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.690+0000 7f8adbfff700 1 -- 172.21.15.189:0/963840818 >> v1:172.21.15.189:6789/0 conn(0x7f8adc0a8230 legacy=0x7f8adc135110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:25.698 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.690+0000 7f8adbfff700 1 -- 172.21.15.189:0/963840818 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8adc0a94e0 con 0x7f8adc060310 2020-11-20T12:08:25.698 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.690+0000 7f8adbfff700 1 --2- 172.21.15.189:0/963840818 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8adc060310 0x7f8adc0a1f50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f8acc00b9a0 tx=0x7f8acc006df0).ready entity=mon.0 client_cookie=438d9b87fd86d934 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:25.698 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.690+0000 7f8adaffd700 1 -- 172.21.15.189:0/963840818 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8acc009680 con 0x7f8adc060310 2020-11-20T12:08:25.698 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.690+0000 7f8adaffd700 1 -- 172.21.15.189:0/963840818 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8acc0097e0 con 0x7f8adc060310 2020-11-20T12:08:25.699 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.690+0000 7f8adaffd700 1 -- 172.21.15.189:0/963840818 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8acc009b60 con 0x7f8adc060310 2020-11-20T12:08:25.702 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8ae1405700 1 -- 172.21.15.189:0/963840818 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8adc060310 msgr2=0x7f8adc0a1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:25.702 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8ae1405700 1 --2- 172.21.15.189:0/963840818 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8adc060310 0x7f8adc0a1f50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f8acc00b9a0 tx=0x7f8acc006df0).stop 2020-11-20T12:08:25.702 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8ae1405700 1 -- 172.21.15.189:0/963840818 shutdown_connections 2020-11-20T12:08:25.703 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8ae1405700 1 --2- 172.21.15.189:0/963840818 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8adc060310 0x7f8adc0a1f50 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:25.703 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8ae1405700 1 --2- 172.21.15.189:0/963840818 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8adc0a2440 0x7f8adc0a7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:25.703 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8ae1405700 1 -- 172.21.15.189:0/963840818 >> 172.21.15.189:0/963840818 conn(0x7f8adc063210 msgr2=0x7f8adc060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:25.703 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8ae1405700 1 -- 172.21.15.189:0/963840818 shutdown_connections 2020-11-20T12:08:25.703 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8ae1405700 1 -- 172.21.15.189:0/963840818 wait complete. 2020-11-20T12:08:25.704 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8ae1405700 1 Processor -- start 2020-11-20T12:08:25.704 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8ae1405700 1 -- start start 2020-11-20T12:08:25.704 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8ae1405700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8adc060310 0x7f8adc0a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:25.704 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8ae1405700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8adc0a2440 0x7f8adc13a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:25.704 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8ae1405700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8adc0a8230 0x7f8adc09fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:25.705 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8ae1405700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc139af0 con 0x7f8adc0a2440 2020-11-20T12:08:25.705 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8ae1405700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc139d50 con 0x7f8adc0a8230 2020-11-20T12:08:25.705 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8ae1405700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc13aee0 con 0x7f8adc060310 2020-11-20T12:08:25.705 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8ae0c04700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8adc0a8230 0x7f8adc09fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:25.705 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8ae0c04700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8adc0a8230 0x7f8adc09fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49734/0 (socket says 172.21.15.189:49734) 2020-11-20T12:08:25.706 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8ae0c04700 1 -- 172.21.15.189:0/3356084051 learned_addr learned my addr 172.21.15.189:0/3356084051 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:08:25.706 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8adb7fe700 1 --2- 172.21.15.189:0/3356084051 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8adc0a2440 0x7f8adc13a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:25.706 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8adbfff700 1 --2- 172.21.15.189:0/3356084051 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8adc060310 0x7f8adc0a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:25.706 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8ae0c04700 1 -- 172.21.15.189:0/3356084051 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8adc060310 msgr2=0x7f8adc0a30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:25.706 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8ae0c04700 1 --2- 172.21.15.189:0/3356084051 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8adc060310 0x7f8adc0a30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:25.707 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8ae0c04700 1 -- 172.21.15.189:0/3356084051 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8adc0a2440 msgr2=0x7f8adc13a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:25.707 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8ae0c04700 1 --2- 172.21.15.189:0/3356084051 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8adc0a2440 0x7f8adc13a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:25.707 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8ae0c04700 1 -- 172.21.15.189:0/3356084051 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8adc13b250 con 0x7f8adc0a8230 2020-11-20T12:08:25.707 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8ae0c04700 1 --2- 172.21.15.189:0/3356084051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8adc0a8230 0x7f8adc09fd30 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f8ad4001420 tx=0x7f8ad4007e20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:25.707 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.694+0000 7f8abbfff700 1 -- 172.21.15.189:0/3356084051 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad400ce10 con 0x7f8adc0a8230 2020-11-20T12:08:25.708 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.698+0000 7f8abbfff700 1 -- 172.21.15.189:0/3356084051 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad4005b80 con 0x7f8adc0a8230 2020-11-20T12:08:25.708 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.698+0000 7f8abbfff700 1 -- 172.21.15.189:0/3356084051 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad4007110 con 0x7f8adc0a8230 2020-11-20T12:08:25.708 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.698+0000 7f8ae1405700 1 -- 172.21.15.189:0/3356084051 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8adc13a6e0 con 0x7f8adc0a8230 2020-11-20T12:08:25.708 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.698+0000 7f8ae1405700 1 -- 172.21.15.189:0/3356084051 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8adc14ada0 con 0x7f8adc0a8230 2020-11-20T12:08:25.709 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.698+0000 7f8ae1405700 1 -- 172.21.15.189:0/3356084051 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8adc13b050 con 0x7f8adc0a8230 2020-11-20T12:08:25.709 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.698+0000 7f8abbfff700 1 -- 172.21.15.189:0/3356084051 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f8ad4010030 con 0x7f8adc0a8230 2020-11-20T12:08:25.710 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.698+0000 7f8abbfff700 1 --2- 172.21.15.189:0/3356084051 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f8ab40557c0 0x7f8ab4057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:08:25.710 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.698+0000 7f8abbfff700 1 -- 172.21.15.189:0/3356084051 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f8ad406e310 con 0x7f8adc0a8230 2020-11-20T12:08:25.711 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.698+0000 7f8adbfff700 1 --2- 172.21.15.189:0/3356084051 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f8ab40557c0 0x7f8ab4057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:08:25.711 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.698+0000 7f8adbfff700 1 --2- 172.21.15.189:0/3356084051 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f8ab40557c0 0x7f8ab4057c80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f8acc00b150 tx=0x7f8acc015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:08:25.711 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.706+0000 7f8abbfff700 1 -- 172.21.15.189:0/3356084051 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8ad4044100 con 0x7f8adc0a8230 2020-11-20T12:08:25.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.942+0000 7f8ae1405700 1 -- 172.21.15.189:0/3356084051 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8adc14ada0 con 0x7f8adc0a8230 2020-11-20T12:08:25.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.942+0000 7f8abbfff700 1 -- 172.21.15.189:0/3356084051 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f8adc14ada0 con 0x7f8adc0a8230 2020-11-20T12:08:25.947 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:08:25.947 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4676,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.47:6833/4000022090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4000022090},{"type":"v1","addr":"172.21.15.47:6833","nonce":4000022090}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:08:20.595791+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/1375880901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1375880901},{"type":"v1","addr":"172.21.15.189:6835","nonce":1375880901}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:08:25.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.946+0000 7f8ae1405700 1 -- 172.21.15.189:0/3356084051 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f8ab40557c0 msgr2=0x7f8ab4057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:25.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.946+0000 7f8ae1405700 1 --2- 172.21.15.189:0/3356084051 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f8ab40557c0 0x7f8ab4057c80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f8acc00b150 tx=0x7f8acc015000).stop 2020-11-20T12:08:25.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.946+0000 7f8ae1405700 1 -- 172.21.15.189:0/3356084051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8adc0a8230 msgr2=0x7f8adc09fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:08:25.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.946+0000 7f8ae1405700 1 --2- 172.21.15.189:0/3356084051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8adc0a8230 0x7f8adc09fd30 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f8ad4001420 tx=0x7f8ad4007e20).stop 2020-11-20T12:08:25.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.946+0000 7f8ae1405700 1 -- 172.21.15.189:0/3356084051 shutdown_connections 2020-11-20T12:08:25.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.946+0000 7f8ae1405700 1 --2- 172.21.15.189:0/3356084051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8adc0a8230 0x7f8adc09fd30 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:25.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.946+0000 7f8ae1405700 1 --2- 172.21.15.189:0/3356084051 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8adc060310 0x7f8adc0a30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:25.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.946+0000 7f8ae1405700 1 --2- 172.21.15.189:0/3356084051 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f8ab40557c0 0x7f8ab4057c80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:25.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.946+0000 7f8ae1405700 1 --2- 172.21.15.189:0/3356084051 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8adc0a2440 0x7f8adc13a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:08:25.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.946+0000 7f8ae1405700 1 -- 172.21.15.189:0/3356084051 >> 172.21.15.189:0/3356084051 conn(0x7f8adc063210 msgr2=0x7f8adc0ad290 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:08:25.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.946+0000 7f8ae1405700 1 -- 172.21.15.189:0/3356084051 shutdown_connections 2020-11-20T12:08:25.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:08:25.946+0000 7f8ae1405700 1 -- 172.21.15.189:0/3356084051 wait complete. 2020-11-20T12:08:25.952 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 16 2020-11-20T12:08:25.961 INFO:tasks.mds_thrash.fs.[None]:mds.b reported in up:standby state 2020-11-20T12:08:25.962 INFO:tasks.mds_thrash.fs.[None]:waiting for 70 secs before thrashing 2020-11-20T12:08:52.631 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:08:52.634 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:09:22.792 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:09:22.831 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:09:35.963 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:36.084 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.082+0000 7fae6de7e700 1 Processor -- start 2020-11-20T12:09:36.085 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.082+0000 7fae6de7e700 1 -- start start 2020-11-20T12:09:36.086 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.082+0000 7fae6de7e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fae68140c80 0x7fae68143070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:36.087 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.082+0000 7fae6de7e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae68063b50 con 0x7fae68060310 2020-11-20T12:09:36.087 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.082+0000 7fae6de7e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae680a09a0 con 0x7fae68134460 2020-11-20T12:09:36.087 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.082+0000 7fae6de7e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae68067a00 con 0x7fae68140c80 2020-11-20T12:09:36.087 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.082+0000 7fae67fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fae68134460 0x7fae681348e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51362/0 (socket says 172.21.15.189:51362) 2020-11-20T12:09:36.088 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.082+0000 7fae67fff700 1 -- 172.21.15.189:0/2625149241 learned_addr learned my addr 172.21.15.189:0/2625149241 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:09:36.088 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.082+0000 7fae6d67d700 1 --2- 172.21.15.189:0/2625149241 >> v2:172.21.15.189:3300/0 conn(0x7fae68140c80 0x7fae68143070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:36.088 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.082+0000 7fae677fe700 1 -- 172.21.15.189:0/2625149241 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4139926539 0 0) 0x7fae680a09a0 con 0x7fae68134460 2020-11-20T12:09:36.088 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.086+0000 7fae677fe700 1 -- 172.21.15.189:0/2625149241 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae4c003240 con 0x7fae68134460 2020-11-20T12:09:36.088 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.086+0000 7fae6d67d700 1 -- 172.21.15.189:0/2625149241 >> v1:172.21.15.189:6789/0 conn(0x7fae68134460 legacy=0x7fae681348e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:36.089 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.086+0000 7fae6d67d700 1 -- 172.21.15.189:0/2625149241 >> v1:172.21.15.47:6789/0 conn(0x7fae68060310 legacy=0x7fae680a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:36.089 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.086+0000 7fae6d67d700 1 -- 172.21.15.189:0/2625149241 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae68066500 con 0x7fae68140c80 2020-11-20T12:09:36.089 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.086+0000 7fae6d67d700 1 --2- 172.21.15.189:0/2625149241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae68140c80 0x7fae68143070 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fae60009cc0 tx=0x7fae600091d0).ready entity=mon.1 client_cookie=ac378976b3681d98 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:36.089 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.086+0000 7fae677fe700 1 -- 172.21.15.189:0/2625149241 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6000ba90 con 0x7fae68140c80 2020-11-20T12:09:36.090 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.086+0000 7fae677fe700 1 -- 172.21.15.189:0/2625149241 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae6000bbf0 con 0x7fae68140c80 2020-11-20T12:09:36.090 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.086+0000 7fae677fe700 1 -- 172.21.15.189:0/2625149241 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae60011400 con 0x7fae68140c80 2020-11-20T12:09:36.092 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.086+0000 7fae6de7e700 1 -- 172.21.15.189:0/2625149241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae68140c80 msgr2=0x7fae68143070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:36.093 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.086+0000 7fae6de7e700 1 --2- 172.21.15.189:0/2625149241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae68140c80 0x7fae68143070 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fae60009cc0 tx=0x7fae600091d0).stop 2020-11-20T12:09:36.093 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.086+0000 7fae6de7e700 1 -- 172.21.15.189:0/2625149241 shutdown_connections 2020-11-20T12:09:36.093 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.086+0000 7fae6de7e700 1 --2- 172.21.15.189:0/2625149241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae68140c80 0x7fae68143070 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:36.093 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.086+0000 7fae6de7e700 1 -- 172.21.15.189:0/2625149241 >> 172.21.15.189:0/2625149241 conn(0x7fae68063210 msgr2=0x7fae68060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:36.093 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.086+0000 7fae6de7e700 1 -- 172.21.15.189:0/2625149241 shutdown_connections 2020-11-20T12:09:36.094 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.086+0000 7fae6de7e700 1 -- 172.21.15.189:0/2625149241 wait complete. 2020-11-20T12:09:36.094 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.086+0000 7fae6de7e700 1 Processor -- start 2020-11-20T12:09:36.094 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.086+0000 7fae6de7e700 1 -- start start 2020-11-20T12:09:36.094 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.086+0000 7fae6de7e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fae68060310 0x7fae680a6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:36.095 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.086+0000 7fae6de7e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae68134460 0x7fae6814b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:36.095 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.086+0000 7fae6de7e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae68140c80 0x7fae6814b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:36.095 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.086+0000 7fae6de7e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6809ccf0 con 0x7fae68140c80 2020-11-20T12:09:36.095 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.086+0000 7fae6de7e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6814ae40 con 0x7fae68134460 2020-11-20T12:09:36.095 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.086+0000 7fae6de7e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae6809c4e0 con 0x7fae68060310 2020-11-20T12:09:36.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.090+0000 7fae67fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae68134460 0x7fae6814b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:36.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.090+0000 7fae67fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae68134460 0x7fae6814b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49746/0 (socket says 172.21.15.189:49746) 2020-11-20T12:09:36.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.090+0000 7fae67fff700 1 -- 172.21.15.189:0/132678461 learned_addr learned my addr 172.21.15.189:0/132678461 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:09:36.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.090+0000 7fae6d67d700 1 --2- 172.21.15.189:0/132678461 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae68140c80 0x7fae6814b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:36.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.090+0000 7fae6ce7c700 1 --2- 172.21.15.189:0/132678461 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fae68060310 0x7fae680a6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:36.097 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.090+0000 7fae67fff700 1 -- 172.21.15.189:0/132678461 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fae68060310 msgr2=0x7fae680a6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:36.097 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.090+0000 7fae67fff700 1 --2- 172.21.15.189:0/132678461 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fae68060310 0x7fae680a6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:36.097 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.090+0000 7fae67fff700 1 -- 172.21.15.189:0/132678461 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae68140c80 msgr2=0x7fae6814b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:36.097 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.090+0000 7fae67fff700 1 --2- 172.21.15.189:0/132678461 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae68140c80 0x7fae6814b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:36.098 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.090+0000 7fae67fff700 1 -- 172.21.15.189:0/132678461 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae6809cf90 con 0x7fae68134460 2020-11-20T12:09:36.098 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.090+0000 7fae6ce7c700 1 --2- 172.21.15.189:0/132678461 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fae68060310 0x7fae680a6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:09:36.098 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.090+0000 7fae67fff700 1 --2- 172.21.15.189:0/132678461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae68134460 0x7fae6814b2e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fae5400c6f0 tx=0x7fae54007ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:36.098 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.090+0000 7fae64ff9700 1 -- 172.21.15.189:0/132678461 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae540072e0 con 0x7fae68134460 2020-11-20T12:09:36.098 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.090+0000 7fae64ff9700 1 -- 172.21.15.189:0/132678461 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae54007440 con 0x7fae68134460 2020-11-20T12:09:36.099 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.090+0000 7fae6de7e700 1 -- 172.21.15.189:0/132678461 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae6809c7e0 con 0x7fae68134460 2020-11-20T12:09:36.099 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.090+0000 7fae64ff9700 1 -- 172.21.15.189:0/132678461 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae54005800 con 0x7fae68134460 2020-11-20T12:09:36.099 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.090+0000 7fae6de7e700 1 -- 172.21.15.189:0/132678461 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae6809eaf0 con 0x7fae68134460 2020-11-20T12:09:36.099 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.090+0000 7fae6de7e700 1 -- 172.21.15.189:0/132678461 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae6814b000 con 0x7fae68134460 2020-11-20T12:09:36.100 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.090+0000 7fae64ff9700 1 -- 172.21.15.189:0/132678461 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fae5401a060 con 0x7fae68134460 2020-11-20T12:09:36.101 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.090+0000 7fae64ff9700 1 --2- 172.21.15.189:0/132678461 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fae400557c0 0x7fae40057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:36.101 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.090+0000 7fae64ff9700 1 -- 172.21.15.189:0/132678461 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fae5406e640 con 0x7fae68134460 2020-11-20T12:09:36.101 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.090+0000 7fae6ce7c700 1 --2- 172.21.15.189:0/132678461 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fae400557c0 0x7fae40057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:36.101 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.094+0000 7fae6ce7c700 1 --2- 172.21.15.189:0/132678461 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fae400557c0 0x7fae40057c80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fae5c0014b0 tx=0x7fae5c009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:36.102 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.098+0000 7fae64ff9700 1 -- 172.21.15.189:0/132678461 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fae540444b0 con 0x7fae68134460 2020-11-20T12:09:36.328 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.326+0000 7fae6de7e700 1 -- 172.21.15.189:0/132678461 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae6809cae0 con 0x7fae68134460 2020-11-20T12:09:36.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.326+0000 7fae64ff9700 1 -- 172.21.15.189:0/132678461 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fae54018070 con 0x7fae68134460 2020-11-20T12:09:36.329 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:09:36.330 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4676,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.47:6833/4000022090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4000022090},{"type":"v1","addr":"172.21.15.47:6833","nonce":4000022090}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:08:20.595791+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/1375880901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1375880901},{"type":"v1","addr":"172.21.15.189:6835","nonce":1375880901}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:09:36.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.330+0000 7fae6de7e700 1 -- 172.21.15.189:0/132678461 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fae400557c0 msgr2=0x7fae40057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:36.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.330+0000 7fae6de7e700 1 --2- 172.21.15.189:0/132678461 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fae400557c0 0x7fae40057c80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fae5c0014b0 tx=0x7fae5c009040).stop 2020-11-20T12:09:36.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.330+0000 7fae6de7e700 1 -- 172.21.15.189:0/132678461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae68134460 msgr2=0x7fae6814b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:36.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.330+0000 7fae6de7e700 1 --2- 172.21.15.189:0/132678461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae68134460 0x7fae6814b2e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fae5400c6f0 tx=0x7fae54007ad0).stop 2020-11-20T12:09:36.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.330+0000 7fae6de7e700 1 -- 172.21.15.189:0/132678461 shutdown_connections 2020-11-20T12:09:36.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.330+0000 7fae6de7e700 1 --2- 172.21.15.189:0/132678461 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fae400557c0 0x7fae40057c80 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:36.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.330+0000 7fae6de7e700 1 --2- 172.21.15.189:0/132678461 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fae68140c80 0x7fae6814b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:36.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.330+0000 7fae6de7e700 1 --2- 172.21.15.189:0/132678461 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fae68060310 0x7fae680a6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:36.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.330+0000 7fae6de7e700 1 --2- 172.21.15.189:0/132678461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae68134460 0x7fae6814b2e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:36.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.330+0000 7fae6de7e700 1 -- 172.21.15.189:0/132678461 >> 172.21.15.189:0/132678461 conn(0x7fae68063210 msgr2=0x7fae680ada10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:36.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.330+0000 7fae6de7e700 1 -- 172.21.15.189:0/132678461 shutdown_connections 2020-11-20T12:09:36.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.330+0000 7fae6de7e700 1 -- 172.21.15.189:0/132678461 wait complete. 2020-11-20T12:09:36.335 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 16 2020-11-20T12:09:36.344 INFO:tasks.mds_thrash.fs.[None]:kill mds.a (rank=0) 2020-11-20T12:09:36.345 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T12:09:36.345 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:09:36.370 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T12:09:36.370 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:36.543 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.542+0000 7f19f105a700 1 Processor -- start 2020-11-20T12:09:36.543 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.542+0000 7f19f105a700 1 -- start start 2020-11-20T12:09:36.545 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.542+0000 7f19f105a700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f19e40075c0 0x7f19e40caff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:36.546 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.542+0000 7f19f105a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f19e4034d10 0x7f19e40cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:36.546 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.542+0000 7f19f105a700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f19e40ce1d0 0x7f19e40ce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:36.546 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.542+0000 7f19f105a700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19e40355a0 con 0x7f19e40075c0 2020-11-20T12:09:36.546 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.542+0000 7f19f105a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19e40cac00 con 0x7f19e4034d10 2020-11-20T12:09:36.546 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.542+0000 7f19f105a700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e40cb9d0 con 0x7f19e40ce1d0 2020-11-20T12:09:36.547 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.542+0000 7f19eb7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f19e4034d10 0x7f19e40cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:36.547 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.542+0000 7f19eb7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f19e4034d10 0x7f19e40cb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49758/0 (socket says 172.21.15.189:49758) 2020-11-20T12:09:36.547 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.542+0000 7f19eb7fe700 1 -- 172.21.15.189:0/4260178133 learned_addr learned my addr 172.21.15.189:0/4260178133 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:09:36.547 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.542+0000 7f19ebfff700 1 --2- 172.21.15.189:0/4260178133 >> v2:172.21.15.47:3300/0 conn(0x7f19e40075c0 0x7f19e40caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:36.548 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.542+0000 7f19f0859700 1 --2- 172.21.15.189:0/4260178133 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f19e40ce1d0 0x7f19e40ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:36.548 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.542+0000 7f19eb7fe700 1 -- 172.21.15.189:0/4260178133 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f19e40ce1d0 msgr2=0x7f19e40ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:36.548 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.542+0000 7f19eb7fe700 1 --2- 172.21.15.189:0/4260178133 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f19e40ce1d0 0x7f19e40ce610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:36.548 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.542+0000 7f19eb7fe700 1 -- 172.21.15.189:0/4260178133 >> v2:172.21.15.47:3300/0 conn(0x7f19e40075c0 msgr2=0x7f19e40caff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:36.548 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.542+0000 7f19eb7fe700 1 --2- 172.21.15.189:0/4260178133 >> v2:172.21.15.47:3300/0 conn(0x7f19e40075c0 0x7f19e40caff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:36.549 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.542+0000 7f19eb7fe700 1 -- 172.21.15.189:0/4260178133 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e40397a0 con 0x7f19e4034d10 2020-11-20T12:09:36.549 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.542+0000 7f19ebfff700 1 --2- 172.21.15.189:0/4260178133 >> v2:172.21.15.47:3300/0 conn(0x7f19e40075c0 0x7f19e40caff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:09:36.549 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.542+0000 7f19f0859700 1 --2- 172.21.15.189:0/4260178133 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f19e40ce1d0 0x7f19e40ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:09:36.549 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.542+0000 7f19eb7fe700 1 --2- 172.21.15.189:0/4260178133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19e4034d10 0x7f19e40cb4e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f19e400be00 tx=0x7f19d8006eb0).ready entity=mon.1 client_cookie=29650d9a2d2c6a46 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:36.549 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.542+0000 7f19eaffd700 1 -- 172.21.15.189:0/4260178133 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d8009890 con 0x7f19e4034d10 2020-11-20T12:09:36.550 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.542+0000 7f19eaffd700 1 -- 172.21.15.189:0/4260178133 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d80099f0 con 0x7f19e4034d10 2020-11-20T12:09:36.550 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.542+0000 7f19eaffd700 1 -- 172.21.15.189:0/4260178133 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d8009d70 con 0x7f19e4034d10 2020-11-20T12:09:36.553 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19f105a700 1 -- 172.21.15.189:0/4260178133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19e4034d10 msgr2=0x7f19e40cb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:36.556 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19f105a700 1 --2- 172.21.15.189:0/4260178133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19e4034d10 0x7f19e40cb4e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f19e400be00 tx=0x7f19d8006eb0).stop 2020-11-20T12:09:36.556 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19f105a700 1 -- 172.21.15.189:0/4260178133 shutdown_connections 2020-11-20T12:09:36.557 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19f105a700 1 --2- 172.21.15.189:0/4260178133 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f19e40ce1d0 0x7f19e40ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:36.557 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19f105a700 1 --2- 172.21.15.189:0/4260178133 >> v2:172.21.15.47:3300/0 conn(0x7f19e40075c0 0x7f19e40caff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:36.557 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19f105a700 1 --2- 172.21.15.189:0/4260178133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19e4034d10 0x7f19e40cb4e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:36.557 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19f105a700 1 -- 172.21.15.189:0/4260178133 >> 172.21.15.189:0/4260178133 conn(0x7f19e403e3a0 msgr2=0x7f19e403a370 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:36.559 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19f105a700 1 -- 172.21.15.189:0/4260178133 shutdown_connections 2020-11-20T12:09:36.559 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19f105a700 1 -- 172.21.15.189:0/4260178133 wait complete. 2020-11-20T12:09:36.560 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19f105a700 1 Processor -- start 2020-11-20T12:09:36.560 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19f105a700 1 -- start start 2020-11-20T12:09:36.560 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19f105a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19e40075c0 0x7f19e403c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:36.560 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19f105a700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f19e4034d10 0x7f19e40da510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:36.561 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19f105a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f19e40ce1d0 0x7f19e40cd750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:36.561 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19f105a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e40d9be0 con 0x7f19e40ce1d0 2020-11-20T12:09:36.561 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19f105a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e40d9dd0 con 0x7f19e40075c0 2020-11-20T12:09:36.562 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19f105a700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e40da070 con 0x7f19e4034d10 2020-11-20T12:09:36.562 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19ebfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19e40075c0 0x7f19e403c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:36.562 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19ebfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19e40075c0 0x7f19e403c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49760/0 (socket says 172.21.15.189:49760) 2020-11-20T12:09:36.562 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19ebfff700 1 -- 172.21.15.189:0/3755642971 learned_addr learned my addr 172.21.15.189:0/3755642971 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:09:36.563 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19eb7fe700 1 --2- 172.21.15.189:0/3755642971 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f19e4034d10 0x7f19e40da510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:36.563 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19f0859700 1 --2- 172.21.15.189:0/3755642971 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f19e40ce1d0 0x7f19e40cd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:36.563 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19ebfff700 1 -- 172.21.15.189:0/3755642971 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f19e4034d10 msgr2=0x7f19e40da510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:36.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19ebfff700 1 --2- 172.21.15.189:0/3755642971 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f19e4034d10 0x7f19e40da510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:36.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19ebfff700 1 -- 172.21.15.189:0/3755642971 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f19e40ce1d0 msgr2=0x7f19e40cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:36.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19ebfff700 1 --2- 172.21.15.189:0/3755642971 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f19e40ce1d0 0x7f19e40cd750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:36.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.546+0000 7f19ebfff700 1 -- 172.21.15.189:0/3755642971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e4031440 con 0x7f19e40075c0 2020-11-20T12:09:36.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.550+0000 7f19ebfff700 1 --2- 172.21.15.189:0/3755642971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19e40075c0 0x7f19e403c850 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f19e000db90 tx=0x7f19e0008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:36.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.550+0000 7f19cffff700 1 -- 172.21.15.189:0/3755642971 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0009790 con 0x7f19e40075c0 2020-11-20T12:09:36.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.550+0000 7f19cffff700 1 -- 172.21.15.189:0/3755642971 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e00098f0 con 0x7f19e40075c0 2020-11-20T12:09:36.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.550+0000 7f19f105a700 1 -- 172.21.15.189:0/3755642971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e40daa00 con 0x7f19e40075c0 2020-11-20T12:09:36.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.550+0000 7f19cffff700 1 -- 172.21.15.189:0/3755642971 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0009cb0 con 0x7f19e40075c0 2020-11-20T12:09:36.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.550+0000 7f19f105a700 1 -- 172.21.15.189:0/3755642971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19e40cdc40 con 0x7f19e40075c0 2020-11-20T12:09:36.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.550+0000 7f19f105a700 1 -- 172.21.15.189:0/3755642971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19e4005330 con 0x7f19e40075c0 2020-11-20T12:09:36.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.550+0000 7f19cffff700 1 -- 172.21.15.189:0/3755642971 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f19e0013690 con 0x7f19e40075c0 2020-11-20T12:09:36.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.550+0000 7f19cffff700 1 --2- 172.21.15.189:0/3755642971 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f19c40557c0 0x7f19c4057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:36.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.550+0000 7f19cffff700 1 -- 172.21.15.189:0/3755642971 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f19e006e670 con 0x7f19e40075c0 2020-11-20T12:09:36.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.550+0000 7f19eb7fe700 1 --2- 172.21.15.189:0/3755642971 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f19c40557c0 0x7f19c4057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:36.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.550+0000 7f19eb7fe700 1 --2- 172.21.15.189:0/3755642971 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f19c40557c0 0x7f19c4057c80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f19e4039690 tx=0x7f19d8015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:36.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.558+0000 7f19cffff700 1 -- 172.21.15.189:0/3755642971 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f19e0044400 con 0x7f19e40075c0 2020-11-20T12:09:36.783 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.774+0000 7f19f105a700 1 -- 172.21.15.189:0/3755642971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19e40c85b0 con 0x7f19e40075c0 2020-11-20T12:09:36.783 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.774+0000 7f19cffff700 1 -- 172.21.15.189:0/3755642971 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f19e0047a20 con 0x7f19e40075c0 2020-11-20T12:09:36.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.778+0000 7f19f105a700 1 -- 172.21.15.189:0/3755642971 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f19c40557c0 msgr2=0x7f19c4057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:36.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.778+0000 7f19f105a700 1 --2- 172.21.15.189:0/3755642971 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f19c40557c0 0x7f19c4057c80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f19e4039690 tx=0x7f19d8015000).stop 2020-11-20T12:09:36.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.778+0000 7f19f105a700 1 -- 172.21.15.189:0/3755642971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19e40075c0 msgr2=0x7f19e403c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:36.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.778+0000 7f19f105a700 1 --2- 172.21.15.189:0/3755642971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19e40075c0 0x7f19e403c850 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f19e000db90 tx=0x7f19e0008040).stop 2020-11-20T12:09:36.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.778+0000 7f19f105a700 1 -- 172.21.15.189:0/3755642971 shutdown_connections 2020-11-20T12:09:36.785 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.778+0000 7f19f105a700 1 --2- 172.21.15.189:0/3755642971 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f19c40557c0 0x7f19c4057c80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:36.785 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.778+0000 7f19f105a700 1 --2- 172.21.15.189:0/3755642971 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f19e40ce1d0 0x7f19e40cd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:36.785 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.778+0000 7f19f105a700 1 --2- 172.21.15.189:0/3755642971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19e40075c0 0x7f19e403c850 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:36.785 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.778+0000 7f19f105a700 1 --2- 172.21.15.189:0/3755642971 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f19e4034d10 0x7f19e40da510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:36.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.778+0000 7f19f105a700 1 -- 172.21.15.189:0/3755642971 >> 172.21.15.189:0/3755642971 conn(0x7f19e403e3a0 msgr2=0x7f19e40c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:36.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.778+0000 7f19f105a700 1 -- 172.21.15.189:0/3755642971 shutdown_connections 2020-11-20T12:09:36.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:36.778+0000 7f19f105a700 1 -- 172.21.15.189:0/3755642971 wait complete. 2020-11-20T12:09:36.786 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 16 2020-11-20T12:09:36.786 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:09:36.787 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4676,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.47:6833/4000022090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4000022090},{"type":"v1","addr":"172.21.15.47:6833","nonce":4000022090}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:08:20.595791+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/1375880901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1375880901},{"type":"v1","addr":"172.21.15.189:6835","nonce":1375880901}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:09:36.803 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:09:38.805 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:38.931 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.930+0000 7f5ceb3c9700 1 Processor -- start 2020-11-20T12:09:38.932 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.930+0000 7f5ceb3c9700 1 -- start start 2020-11-20T12:09:38.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.930+0000 7f5ceb3c9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5ce4060310 0x7f5ce40a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:38.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.930+0000 7f5ceb3c9700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f5ce413fc70 0x7f5ce4142060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:38.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.930+0000 7f5ceb3c9700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce4063b50 con 0x7f5ce4134460 2020-11-20T12:09:38.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.930+0000 7f5ceb3c9700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce4067a00 con 0x7f5ce413fc70 2020-11-20T12:09:38.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.930+0000 7f5ceb3c9700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce4066500 con 0x7f5ce4060310 2020-11-20T12:09:38.935 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.930+0000 7f5cea3c7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5ce4060310 0x7f5ce40a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:38.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.930+0000 7f5cea3c7700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5ce4060310 0x7f5ce40a11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49774/0 (socket says 172.21.15.189:49774) 2020-11-20T12:09:38.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.930+0000 7f5cea3c7700 1 -- 172.21.15.189:0/282767415 learned_addr learned my addr 172.21.15.189:0/282767415 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:09:38.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.930+0000 7f5ceabc8700 1 --2- 172.21.15.189:0/282767415 >> v2:172.21.15.47:3300/0 conn(0x7f5ce413fc70 0x7f5ce4142060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:38.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.930+0000 7f5cea3c7700 1 -- 172.21.15.189:0/282767415 >> v2:172.21.15.47:3300/0 conn(0x7f5ce413fc70 msgr2=0x7f5ce4142060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:38.937 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.930+0000 7f5cea3c7700 1 --2- 172.21.15.189:0/282767415 >> v2:172.21.15.47:3300/0 conn(0x7f5ce413fc70 0x7f5ce4142060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:38.938 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.930+0000 7f5cea3c7700 1 -- 172.21.15.189:0/282767415 >> v1:172.21.15.47:6789/0 conn(0x7f5ce4134460 legacy=0x7f5ce41348e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:38.938 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.930+0000 7f5cea3c7700 1 -- 172.21.15.189:0/282767415 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce40a09a0 con 0x7f5ce4060310 2020-11-20T12:09:38.938 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.930+0000 7f5ceabc8700 1 --2- 172.21.15.189:0/282767415 >> v2:172.21.15.47:3300/0 conn(0x7f5ce413fc70 0x7f5ce4142060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:09:38.938 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.930+0000 7f5cea3c7700 1 --2- 172.21.15.189:0/282767415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ce4060310 0x7f5ce40a11c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f5ce0006fd0 tx=0x7f5ce0006eb0).ready entity=mon.1 client_cookie=2404ec76ee6da5ff server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:38.939 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.934+0000 7f5ce93c5700 1 -- 172.21.15.189:0/282767415 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce0009890 con 0x7f5ce4060310 2020-11-20T12:09:38.939 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.934+0000 7f5ce93c5700 1 -- 172.21.15.189:0/282767415 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce00099f0 con 0x7f5ce4060310 2020-11-20T12:09:38.939 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.934+0000 7f5ce93c5700 1 -- 172.21.15.189:0/282767415 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce0009d70 con 0x7f5ce4060310 2020-11-20T12:09:38.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.934+0000 7f5ceb3c9700 1 -- 172.21.15.189:0/282767415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ce4060310 msgr2=0x7f5ce40a11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:38.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.934+0000 7f5ceb3c9700 1 --2- 172.21.15.189:0/282767415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ce4060310 0x7f5ce40a11c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f5ce0006fd0 tx=0x7f5ce0006eb0).stop 2020-11-20T12:09:38.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.934+0000 7f5ceb3c9700 1 -- 172.21.15.189:0/282767415 shutdown_connections 2020-11-20T12:09:38.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.934+0000 7f5ceb3c9700 1 --2- 172.21.15.189:0/282767415 >> v2:172.21.15.47:3300/0 conn(0x7f5ce413fc70 0x7f5ce4142060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:38.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.934+0000 7f5ceb3c9700 1 --2- 172.21.15.189:0/282767415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ce4060310 0x7f5ce40a11c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:38.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.934+0000 7f5ceb3c9700 1 -- 172.21.15.189:0/282767415 >> 172.21.15.189:0/282767415 conn(0x7f5ce4063210 msgr2=0x7f5ce4060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:38.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.934+0000 7f5ceb3c9700 1 -- 172.21.15.189:0/282767415 shutdown_connections 2020-11-20T12:09:38.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.934+0000 7f5ceb3c9700 1 -- 172.21.15.189:0/282767415 wait complete. 2020-11-20T12:09:38.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.934+0000 7f5ceb3c9700 1 Processor -- start 2020-11-20T12:09:38.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.934+0000 7f5ceb3c9700 1 -- start start 2020-11-20T12:09:38.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.934+0000 7f5ceb3c9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ce4134460 0x7f5ce409fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:38.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.934+0000 7f5ceb3c9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ce413fc70 0x7f5ce4147ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:38.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.934+0000 7f5ceb3c9700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5ce409e070 0x7f5ce409e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:38.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.934+0000 7f5ceb3c9700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce409e9c0 con 0x7f5ce413fc70 2020-11-20T12:09:38.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.934+0000 7f5ceb3c9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce40a07b0 con 0x7f5ce4134460 2020-11-20T12:09:38.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.934+0000 7f5ceb3c9700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce4148a10 con 0x7f5ce409e070 2020-11-20T12:09:38.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.934+0000 7f5cea3c7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ce4134460 0x7f5ce409fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:38.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.934+0000 7f5cea3c7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ce4134460 0x7f5ce409fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49780/0 (socket says 172.21.15.189:49780) 2020-11-20T12:09:38.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.934+0000 7f5cea3c7700 1 -- 172.21.15.189:0/394492189 learned_addr learned my addr 172.21.15.189:0/394492189 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:09:38.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.934+0000 7f5ce9bc6700 1 --2- 172.21.15.189:0/394492189 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ce413fc70 0x7f5ce4147ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:38.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.934+0000 7f5ceabc8700 1 --2- 172.21.15.189:0/394492189 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5ce409e070 0x7f5ce409e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:38.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.938+0000 7f5cea3c7700 1 -- 172.21.15.189:0/394492189 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5ce409e070 msgr2=0x7f5ce409e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:38.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.938+0000 7f5cea3c7700 1 --2- 172.21.15.189:0/394492189 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5ce409e070 0x7f5ce409e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:38.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.938+0000 7f5cea3c7700 1 -- 172.21.15.189:0/394492189 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ce413fc70 msgr2=0x7f5ce4147ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:38.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.938+0000 7f5cea3c7700 1 --2- 172.21.15.189:0/394492189 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ce413fc70 0x7f5ce4147ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:38.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.938+0000 7f5cea3c7700 1 -- 172.21.15.189:0/394492189 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce41484e0 con 0x7f5ce4134460 2020-11-20T12:09:38.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.938+0000 7f5ce9bc6700 1 --2- 172.21.15.189:0/394492189 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ce413fc70 0x7f5ce4147ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:09:38.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.938+0000 7f5cea3c7700 1 --2- 172.21.15.189:0/394492189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ce4134460 0x7f5ce409fe80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f5ce4066640 tx=0x7f5ce0015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:38.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.938+0000 7f5cd27fc700 1 -- 172.21.15.189:0/394492189 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce0011d00 con 0x7f5ce4134460 2020-11-20T12:09:38.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.938+0000 7f5cd27fc700 1 -- 172.21.15.189:0/394492189 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce0011e60 con 0x7f5ce4134460 2020-11-20T12:09:38.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.938+0000 7f5cd27fc700 1 -- 172.21.15.189:0/394492189 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce001e670 con 0x7f5ce4134460 2020-11-20T12:09:38.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.938+0000 7f5ceb3c9700 1 -- 172.21.15.189:0/394492189 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce4148770 con 0x7f5ce4134460 2020-11-20T12:09:38.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.938+0000 7f5ceb3c9700 1 -- 172.21.15.189:0/394492189 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce40a7470 con 0x7f5ce4134460 2020-11-20T12:09:38.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.938+0000 7f5ceb3c9700 1 -- 172.21.15.189:0/394492189 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce413fa50 con 0x7f5ce4134460 2020-11-20T12:09:38.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.938+0000 7f5cd27fc700 1 -- 172.21.15.189:0/394492189 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f5ce0025030 con 0x7f5ce4134460 2020-11-20T12:09:38.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.938+0000 7f5cd27fc700 1 --2- 172.21.15.189:0/394492189 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5cbc0557c0 0x7f5cbc057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:38.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.938+0000 7f5cd27fc700 1 -- 172.21.15.189:0/394492189 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f5ce006fe40 con 0x7f5ce4134460 2020-11-20T12:09:38.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.938+0000 7f5ce9bc6700 1 --2- 172.21.15.189:0/394492189 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5cbc0557c0 0x7f5cbc057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:38.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.938+0000 7f5ce9bc6700 1 --2- 172.21.15.189:0/394492189 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5cbc0557c0 0x7f5cbc057c80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f5ce4145610 tx=0x7f5cd4009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:38.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:38.946+0000 7f5cd27fc700 1 -- 172.21.15.189:0/394492189 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5ce0045ce0 con 0x7f5ce4134460 2020-11-20T12:09:39.173 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:39.170+0000 7f5ceb3c9700 1 -- 172.21.15.189:0/394492189 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ce40446d0 con 0x7f5ce4134460 2020-11-20T12:09:39.174 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:39.174+0000 7f5cd27fc700 1 -- 172.21.15.189:0/394492189 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f5ce0023020 con 0x7f5ce4134460 2020-11-20T12:09:39.174 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:09:39.175 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4676,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.47:6833/4000022090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4000022090},{"type":"v1","addr":"172.21.15.47:6833","nonce":4000022090}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:08:20.595791+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/1375880901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1375880901},{"type":"v1","addr":"172.21.15.189:6835","nonce":1375880901}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:09:39.177 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:39.174+0000 7f5ceb3c9700 1 -- 172.21.15.189:0/394492189 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5cbc0557c0 msgr2=0x7f5cbc057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:39.177 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:39.174+0000 7f5ceb3c9700 1 --2- 172.21.15.189:0/394492189 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5cbc0557c0 0x7f5cbc057c80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f5ce4145610 tx=0x7f5cd4009040).stop 2020-11-20T12:09:39.177 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:39.174+0000 7f5ceb3c9700 1 -- 172.21.15.189:0/394492189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ce4134460 msgr2=0x7f5ce409fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:39.177 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:39.174+0000 7f5ceb3c9700 1 --2- 172.21.15.189:0/394492189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ce4134460 0x7f5ce409fe80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f5ce4066640 tx=0x7f5ce0015000).stop 2020-11-20T12:09:39.178 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:39.174+0000 7f5ceb3c9700 1 -- 172.21.15.189:0/394492189 shutdown_connections 2020-11-20T12:09:39.178 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:39.174+0000 7f5ceb3c9700 1 --2- 172.21.15.189:0/394492189 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5ce409e070 0x7f5ce409e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:39.178 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:39.174+0000 7f5ceb3c9700 1 --2- 172.21.15.189:0/394492189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ce4134460 0x7f5ce409fe80 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:39.178 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:39.174+0000 7f5ceb3c9700 1 --2- 172.21.15.189:0/394492189 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5cbc0557c0 0x7f5cbc057c80 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:39.179 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:39.174+0000 7f5ceb3c9700 1 --2- 172.21.15.189:0/394492189 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5ce413fc70 0x7f5ce4147ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:39.180 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:39.174+0000 7f5ceb3c9700 1 -- 172.21.15.189:0/394492189 >> 172.21.15.189:0/394492189 conn(0x7f5ce4063210 msgr2=0x7f5ce4060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:39.180 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:39.174+0000 7f5ceb3c9700 1 -- 172.21.15.189:0/394492189 shutdown_connections 2020-11-20T12:09:39.180 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:39.174+0000 7f5ceb3c9700 1 -- 172.21.15.189:0/394492189 wait complete. 2020-11-20T12:09:39.181 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 16 2020-11-20T12:09:39.192 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:09:41.194 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:41.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60cdca6700 1 Processor -- start 2020-11-20T12:09:41.336 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60cdca6700 1 -- start start 2020-11-20T12:09:41.338 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60cdca6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f60c80a8840 0x7f60c813dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:41.338 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60cdca6700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f60c813e100 0x7f60c81371b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:41.339 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60cdca6700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c8066800 con 0x7f60c81376a0 2020-11-20T12:09:41.345 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60cdca6700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60c8066aa0 con 0x7f60c813e100 2020-11-20T12:09:41.345 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60cdca6700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60c80672f0 con 0x7f60c80a8840 2020-11-20T12:09:41.345 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60ccca4700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f60c80a8840 0x7f60c813dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:41.346 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60ccca4700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f60c80a8840 0x7f60c813dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49794/0 (socket says 172.21.15.189:49794) 2020-11-20T12:09:41.346 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60ccca4700 1 -- 172.21.15.189:0/2014925878 learned_addr learned my addr 172.21.15.189:0/2014925878 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:09:41.346 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60c7fff700 1 --2- 172.21.15.189:0/2014925878 >> v2:172.21.15.47:3300/0 conn(0x7f60c813e100 0x7f60c81371b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:41.346 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60ccca4700 1 -- 172.21.15.189:0/2014925878 >> v2:172.21.15.47:3300/0 conn(0x7f60c813e100 msgr2=0x7f60c81371b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:41.346 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60ccca4700 1 --2- 172.21.15.189:0/2014925878 >> v2:172.21.15.47:3300/0 conn(0x7f60c813e100 0x7f60c81371b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:41.347 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60ccca4700 1 -- 172.21.15.189:0/2014925878 >> v1:172.21.15.47:6789/0 conn(0x7f60c81376a0 legacy=0x7f60c813a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:41.347 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60ccca4700 1 -- 172.21.15.189:0/2014925878 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c8067590 con 0x7f60c80a8840 2020-11-20T12:09:41.347 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60ccca4700 1 --2- 172.21.15.189:0/2014925878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c80a8840 0x7f60c813dc10 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f60c8068350 tx=0x7f60b0006eb0).ready entity=mon.1 client_cookie=c2242f12228f28d5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:41.358 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60c77fe700 1 -- 172.21.15.189:0/2014925878 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b00098d0 con 0x7f60c80a8840 2020-11-20T12:09:41.358 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60c77fe700 1 -- 172.21.15.189:0/2014925878 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b0009a30 con 0x7f60c80a8840 2020-11-20T12:09:41.359 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60c77fe700 1 -- 172.21.15.189:0/2014925878 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b0009db0 con 0x7f60c80a8840 2020-11-20T12:09:41.359 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60cdca6700 1 -- 172.21.15.189:0/2014925878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c80a8840 msgr2=0x7f60c813dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:41.359 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60cdca6700 1 --2- 172.21.15.189:0/2014925878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c80a8840 0x7f60c813dc10 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f60c8068350 tx=0x7f60b0006eb0).stop 2020-11-20T12:09:41.359 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60cdca6700 1 -- 172.21.15.189:0/2014925878 shutdown_connections 2020-11-20T12:09:41.359 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60cdca6700 1 --2- 172.21.15.189:0/2014925878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c80a8840 0x7f60c813dc10 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:41.360 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60cdca6700 1 --2- 172.21.15.189:0/2014925878 >> v2:172.21.15.47:3300/0 conn(0x7f60c813e100 0x7f60c81371b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:41.360 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60cdca6700 1 -- 172.21.15.189:0/2014925878 >> 172.21.15.189:0/2014925878 conn(0x7f60c8060170 msgr2=0x7f60c8060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:41.360 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60cdca6700 1 -- 172.21.15.189:0/2014925878 shutdown_connections 2020-11-20T12:09:41.360 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.334+0000 7f60cdca6700 1 -- 172.21.15.189:0/2014925878 wait complete. 2020-11-20T12:09:41.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.338+0000 7f60cdca6700 1 Processor -- start 2020-11-20T12:09:41.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.338+0000 7f60cdca6700 1 -- start start 2020-11-20T12:09:41.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.338+0000 7f60cdca6700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f60c80a8840 0x7f60c813d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:41.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.338+0000 7f60cdca6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60c81376a0 0x7f60c813a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:41.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.338+0000 7f60cdca6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c813e100 0x7f60c809e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:41.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.338+0000 7f60cdca6700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c809c4e0 con 0x7f60c81376a0 2020-11-20T12:09:41.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.338+0000 7f60cdca6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c809c710 con 0x7f60c813e100 2020-11-20T12:09:41.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.338+0000 7f60cdca6700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60c809c9b0 con 0x7f60c80a8840 2020-11-20T12:09:41.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.338+0000 7f60cd4a5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c813e100 0x7f60c809e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:41.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.338+0000 7f60cd4a5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c813e100 0x7f60c809e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49800/0 (socket says 172.21.15.189:49800) 2020-11-20T12:09:41.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.338+0000 7f60cd4a5700 1 -- 172.21.15.189:0/1003462428 learned_addr learned my addr 172.21.15.189:0/1003462428 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:09:41.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.338+0000 7f60c7fff700 1 --2- 172.21.15.189:0/1003462428 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60c81376a0 0x7f60c813a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:41.364 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.338+0000 7f60ccca4700 1 --2- 172.21.15.189:0/1003462428 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f60c80a8840 0x7f60c813d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:41.364 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.338+0000 7f60cd4a5700 1 -- 172.21.15.189:0/1003462428 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f60c80a8840 msgr2=0x7f60c813d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:41.364 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.338+0000 7f60cd4a5700 1 --2- 172.21.15.189:0/1003462428 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f60c80a8840 0x7f60c813d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:41.364 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.338+0000 7f60cd4a5700 1 -- 172.21.15.189:0/1003462428 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60c81376a0 msgr2=0x7f60c813a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:41.365 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.338+0000 7f60cd4a5700 1 --2- 172.21.15.189:0/1003462428 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60c81376a0 0x7f60c813a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:41.365 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.338+0000 7f60cd4a5700 1 -- 172.21.15.189:0/1003462428 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c809cc50 con 0x7f60c813e100 2020-11-20T12:09:41.367 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.338+0000 7f60cd4a5700 1 --2- 172.21.15.189:0/1003462428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c813e100 0x7f60c809e110 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f60c000c710 tx=0x7f60c0007bc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:41.368 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.338+0000 7f60c4ff9700 1 -- 172.21.15.189:0/1003462428 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c000e490 con 0x7f60c813e100 2020-11-20T12:09:41.368 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.338+0000 7f60cdca6700 1 -- 172.21.15.189:0/1003462428 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c8149c90 con 0x7f60c813e100 2020-11-20T12:09:41.369 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.338+0000 7f60c4ff9700 1 -- 172.21.15.189:0/1003462428 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c000e5f0 con 0x7f60c813e100 2020-11-20T12:09:41.369 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.338+0000 7f60c4ff9700 1 -- 172.21.15.189:0/1003462428 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c000e970 con 0x7f60c813e100 2020-11-20T12:09:41.370 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.338+0000 7f60cdca6700 1 -- 172.21.15.189:0/1003462428 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60c813ab80 con 0x7f60c813e100 2020-11-20T12:09:41.370 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.342+0000 7f60c4ff9700 1 -- 172.21.15.189:0/1003462428 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f60c0010030 con 0x7f60c813e100 2020-11-20T12:09:41.371 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.342+0000 7f60cdca6700 1 -- 172.21.15.189:0/1003462428 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c80a9230 con 0x7f60c813e100 2020-11-20T12:09:41.371 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.342+0000 7f60c4ff9700 1 --2- 172.21.15.189:0/1003462428 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f60a40557f0 0x7f60a4057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:41.372 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.342+0000 7f60c4ff9700 1 -- 172.21.15.189:0/1003462428 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f60c006e510 con 0x7f60c813e100 2020-11-20T12:09:41.373 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.342+0000 7f60ccca4700 1 --2- 172.21.15.189:0/1003462428 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f60a40557f0 0x7f60a4057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:41.374 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.342+0000 7f60ccca4700 1 --2- 172.21.15.189:0/1003462428 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f60a40557f0 0x7f60a4057cb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f60b0006ef0 tx=0x7f60b0015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:41.377 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.346+0000 7f60c4ff9700 1 -- 172.21.15.189:0/1003462428 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f60c0044300 con 0x7f60c813e100 2020-11-20T12:09:41.552 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.550+0000 7f60cdca6700 1 -- 172.21.15.189:0/1003462428 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60c80446d0 con 0x7f60c813e100 2020-11-20T12:09:41.553 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.550+0000 7f60c4ff9700 1 -- 172.21.15.189:0/1003462428 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f60c0047920 con 0x7f60c813e100 2020-11-20T12:09:41.553 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:09:41.553 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4676,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.47:6833/4000022090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4000022090},{"type":"v1","addr":"172.21.15.47:6833","nonce":4000022090}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:08:20.595791+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/1375880901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1375880901},{"type":"v1","addr":"172.21.15.189:6835","nonce":1375880901}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:09:41.555 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.554+0000 7f60cdca6700 1 -- 172.21.15.189:0/1003462428 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f60a40557f0 msgr2=0x7f60a4057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:41.555 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.554+0000 7f60cdca6700 1 --2- 172.21.15.189:0/1003462428 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f60a40557f0 0x7f60a4057cb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f60b0006ef0 tx=0x7f60b0015000).stop 2020-11-20T12:09:41.556 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.554+0000 7f60cdca6700 1 -- 172.21.15.189:0/1003462428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c813e100 msgr2=0x7f60c809e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:41.556 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.554+0000 7f60cdca6700 1 --2- 172.21.15.189:0/1003462428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c813e100 0x7f60c809e110 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f60c000c710 tx=0x7f60c0007bc0).stop 2020-11-20T12:09:41.556 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.554+0000 7f60cdca6700 1 -- 172.21.15.189:0/1003462428 shutdown_connections 2020-11-20T12:09:41.556 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.554+0000 7f60cdca6700 1 --2- 172.21.15.189:0/1003462428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c813e100 0x7f60c809e110 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:41.556 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.554+0000 7f60cdca6700 1 --2- 172.21.15.189:0/1003462428 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f60c80a8840 0x7f60c813d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:41.557 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.554+0000 7f60cdca6700 1 --2- 172.21.15.189:0/1003462428 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f60a40557f0 0x7f60a4057cb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:41.557 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.554+0000 7f60cdca6700 1 --2- 172.21.15.189:0/1003462428 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f60c81376a0 0x7f60c813a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:41.557 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.554+0000 7f60cdca6700 1 -- 172.21.15.189:0/1003462428 >> 172.21.15.189:0/1003462428 conn(0x7f60c8060170 msgr2=0x7f60c80a9d10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:41.557 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.554+0000 7f60cdca6700 1 -- 172.21.15.189:0/1003462428 shutdown_connections 2020-11-20T12:09:41.557 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:41.554+0000 7f60cdca6700 1 -- 172.21.15.189:0/1003462428 wait complete. 2020-11-20T12:09:41.558 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 16 2020-11-20T12:09:41.568 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:09:43.571 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:43.710 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.706+0000 7fafee357700 1 Processor -- start 2020-11-20T12:09:43.719 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.706+0000 7fafee357700 1 -- start start 2020-11-20T12:09:43.720 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.710+0000 7fafee357700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fafe80aeb80 0x7fafe813a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:43.721 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.710+0000 7fafee357700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fafe80aef60 0x7fafe813e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:43.721 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.710+0000 7fafee357700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe813dcb0 con 0x7fafe813ade0 2020-11-20T12:09:43.722 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.710+0000 7fafee357700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafe813eb40 con 0x7fafe80aef60 2020-11-20T12:09:43.722 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.710+0000 7fafee357700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe813df10 con 0x7fafe80aeb80 2020-11-20T12:09:43.723 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.710+0000 7fafed355700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fafe80aeb80 0x7fafe813a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:43.723 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.710+0000 7fafed355700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fafe80aeb80 0x7fafe813a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.189:42840/0 (socket says 172.21.15.189:42840) 2020-11-20T12:09:43.728 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.710+0000 7fafed355700 1 -- 172.21.15.189:0/798934886 learned_addr learned my addr 172.21.15.189:0/798934886 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:09:43.729 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.710+0000 7fafecb54700 1 --2- 172.21.15.189:0/798934886 >> v2:172.21.15.47:3300/0 conn(0x7fafe80aef60 0x7fafe813e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:43.730 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.710+0000 7fafed355700 1 -- 172.21.15.189:0/798934886 >> v2:172.21.15.47:3300/0 conn(0x7fafe80aef60 msgr2=0x7fafe813e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:43.746 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.710+0000 7fafed355700 1 --2- 172.21.15.189:0/798934886 >> v2:172.21.15.47:3300/0 conn(0x7fafe80aef60 0x7fafe813e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:43.747 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.710+0000 7fafed355700 1 -- 172.21.15.189:0/798934886 >> v1:172.21.15.47:6789/0 conn(0x7fafe813ade0 legacy=0x7fafe80a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:43.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.710+0000 7fafed355700 1 -- 172.21.15.189:0/798934886 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe8137fa0 con 0x7fafe80aeb80 2020-11-20T12:09:43.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.710+0000 7fafecb54700 1 --2- 172.21.15.189:0/798934886 >> v2:172.21.15.47:3300/0 conn(0x7fafe80aef60 0x7fafe813e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:09:43.749 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.710+0000 7fafed355700 1 --2- 172.21.15.189:0/798934886 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fafe80aeb80 0x7fafe813a640 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fafdc006fd0 tx=0x7fafdc006eb0).ready entity=mon.2 client_cookie=581352dd58fead78 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:43.751 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.710+0000 7fafd3fff700 1 -- 172.21.15.189:0/798934886 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafdc009810 con 0x7fafe80aeb80 2020-11-20T12:09:43.751 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.710+0000 7fafd3fff700 1 -- 172.21.15.189:0/798934886 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafdc009970 con 0x7fafe80aeb80 2020-11-20T12:09:43.751 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.710+0000 7fafd3fff700 1 -- 172.21.15.189:0/798934886 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafdc009cf0 con 0x7fafe80aeb80 2020-11-20T12:09:43.752 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.710+0000 7fafee357700 1 -- 172.21.15.189:0/798934886 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fafe80aeb80 msgr2=0x7fafe813a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:43.753 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.710+0000 7fafee357700 1 --2- 172.21.15.189:0/798934886 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fafe80aeb80 0x7fafe813a640 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fafdc006fd0 tx=0x7fafdc006eb0).stop 2020-11-20T12:09:43.753 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.710+0000 7fafee357700 1 -- 172.21.15.189:0/798934886 shutdown_connections 2020-11-20T12:09:43.753 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.710+0000 7fafee357700 1 --2- 172.21.15.189:0/798934886 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fafe80aeb80 0x7fafe813a640 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:43.753 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.710+0000 7fafee357700 1 --2- 172.21.15.189:0/798934886 >> v2:172.21.15.47:3300/0 conn(0x7fafe80aef60 0x7fafe813e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:43.754 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.710+0000 7fafee357700 1 -- 172.21.15.189:0/798934886 >> 172.21.15.189:0/798934886 conn(0x7fafe8062080 msgr2=0x7fafe8062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:43.754 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.710+0000 7fafee357700 1 -- 172.21.15.189:0/798934886 shutdown_connections 2020-11-20T12:09:43.754 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.710+0000 7fafee357700 1 -- 172.21.15.189:0/798934886 wait complete. 2020-11-20T12:09:43.754 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.714+0000 7fafee357700 1 Processor -- start 2020-11-20T12:09:43.754 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.714+0000 7fafee357700 1 -- start start 2020-11-20T12:09:43.755 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.714+0000 7fafee357700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fafe80aeb80 0x7fafe8064620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:43.755 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.714+0000 7fafee357700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fafe80aef60 0x7fafe80a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:43.755 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.714+0000 7fafee357700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafe813ade0 0x7fafe80a80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:43.755 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.714+0000 7fafee357700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe80a64a0 con 0x7fafe80aef60 2020-11-20T12:09:43.755 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.714+0000 7fafee357700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe8139b90 con 0x7fafe813ade0 2020-11-20T12:09:43.756 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.714+0000 7fafee357700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe8139e30 con 0x7fafe80aeb80 2020-11-20T12:09:43.756 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.714+0000 7fafedb56700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafe813ade0 0x7fafe80a80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:43.756 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.714+0000 7fafedb56700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafe813ade0 0x7fafe80a80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49816/0 (socket says 172.21.15.189:49816) 2020-11-20T12:09:43.756 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.714+0000 7fafedb56700 1 -- 172.21.15.189:0/270610756 learned_addr learned my addr 172.21.15.189:0/270610756 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:09:43.757 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.714+0000 7fafecb54700 1 --2- 172.21.15.189:0/270610756 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fafe80aef60 0x7fafe80a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:43.757 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.718+0000 7fafed355700 1 --2- 172.21.15.189:0/270610756 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fafe80aeb80 0x7fafe8064620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:43.757 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.718+0000 7fafedb56700 1 -- 172.21.15.189:0/270610756 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fafe80aeb80 msgr2=0x7fafe8064620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:43.758 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.718+0000 7fafedb56700 1 --2- 172.21.15.189:0/270610756 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fafe80aeb80 0x7fafe8064620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:43.758 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.718+0000 7fafedb56700 1 -- 172.21.15.189:0/270610756 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fafe80aef60 msgr2=0x7fafe80a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:43.758 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.718+0000 7fafedb56700 1 --2- 172.21.15.189:0/270610756 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fafe80aef60 0x7fafe80a5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:43.758 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.718+0000 7fafedb56700 1 -- 172.21.15.189:0/270610756 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe813a0d0 con 0x7fafe813ade0 2020-11-20T12:09:43.758 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.718+0000 7fafed355700 1 --2- 172.21.15.189:0/270610756 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fafe80aeb80 0x7fafe8064620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:09:43.759 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.718+0000 7fafedb56700 1 --2- 172.21.15.189:0/270610756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafe813ade0 0x7fafe80a80c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fafe000ca90 tx=0x7fafe0007f70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:43.759 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.718+0000 7fafd9ffb700 1 -- 172.21.15.189:0/270610756 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe000a730 con 0x7fafe813ade0 2020-11-20T12:09:43.759 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.718+0000 7fafd9ffb700 1 -- 172.21.15.189:0/270610756 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe000a890 con 0x7fafe813ade0 2020-11-20T12:09:43.759 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.718+0000 7fafee357700 1 -- 172.21.15.189:0/270610756 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe8065200 con 0x7fafe813ade0 2020-11-20T12:09:43.759 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.718+0000 7fafd9ffb700 1 -- 172.21.15.189:0/270610756 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe000ac30 con 0x7fafe813ade0 2020-11-20T12:09:43.760 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.718+0000 7fafee357700 1 -- 172.21.15.189:0/270610756 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe80a8750 con 0x7fafe813ade0 2020-11-20T12:09:43.760 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.718+0000 7fafd9ffb700 1 -- 172.21.15.189:0/270610756 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fafe0019030 con 0x7fafe813ade0 2020-11-20T12:09:43.760 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.718+0000 7fafee357700 1 -- 172.21.15.189:0/270610756 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafe813a260 con 0x7fafe813ade0 2020-11-20T12:09:43.760 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.718+0000 7fafd9ffb700 1 --2- 172.21.15.189:0/270610756 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fafbc0557c0 0x7fafbc057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:43.760 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.718+0000 7fafed355700 1 --2- 172.21.15.189:0/270610756 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fafbc0557c0 0x7fafbc057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:43.761 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.718+0000 7fafd9ffb700 1 -- 172.21.15.189:0/270610756 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fafe006def0 con 0x7fafe813ade0 2020-11-20T12:09:43.761 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.718+0000 7fafed355700 1 --2- 172.21.15.189:0/270610756 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fafbc0557c0 0x7fafbc057c80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fafdc00beb0 tx=0x7fafdc015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:43.761 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.730+0000 7fafd9ffb700 1 -- 172.21.15.189:0/270610756 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fafe0043ce0 con 0x7fafe813ade0 2020-11-20T12:09:43.929 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.926+0000 7fafee357700 1 -- 172.21.15.189:0/270610756 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafe80a5d80 con 0x7fafe813ade0 2020-11-20T12:09:43.930 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.930+0000 7fafd9ffb700 1 -- 172.21.15.189:0/270610756 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fafe0047300 con 0x7fafe813ade0 2020-11-20T12:09:43.931 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:09:43.931 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4676,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.47:6833/4000022090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4000022090},{"type":"v1","addr":"172.21.15.47:6833","nonce":4000022090}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:08:20.595791+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/1375880901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1375880901},{"type":"v1","addr":"172.21.15.189:6835","nonce":1375880901}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:09:43.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.930+0000 7fafee357700 1 -- 172.21.15.189:0/270610756 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fafbc0557c0 msgr2=0x7fafbc057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:43.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.930+0000 7fafee357700 1 --2- 172.21.15.189:0/270610756 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fafbc0557c0 0x7fafbc057c80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fafdc00beb0 tx=0x7fafdc015000).stop 2020-11-20T12:09:43.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.930+0000 7fafee357700 1 -- 172.21.15.189:0/270610756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafe813ade0 msgr2=0x7fafe80a80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:43.935 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.930+0000 7fafee357700 1 --2- 172.21.15.189:0/270610756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafe813ade0 0x7fafe80a80c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fafe000ca90 tx=0x7fafe0007f70).stop 2020-11-20T12:09:43.935 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.930+0000 7fafee357700 1 -- 172.21.15.189:0/270610756 shutdown_connections 2020-11-20T12:09:43.935 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.930+0000 7fafee357700 1 --2- 172.21.15.189:0/270610756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafe813ade0 0x7fafe80a80c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:43.935 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.930+0000 7fafee357700 1 --2- 172.21.15.189:0/270610756 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fafe80aeb80 0x7fafe8064620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:43.935 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.930+0000 7fafee357700 1 --2- 172.21.15.189:0/270610756 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fafbc0557c0 0x7fafbc057c80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:43.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.930+0000 7fafee357700 1 --2- 172.21.15.189:0/270610756 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fafe80aef60 0x7fafe80a5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:43.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.930+0000 7fafee357700 1 -- 172.21.15.189:0/270610756 >> 172.21.15.189:0/270610756 conn(0x7fafe8062080 msgr2=0x7fafe80a1530 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:43.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.930+0000 7fafee357700 1 -- 172.21.15.189:0/270610756 shutdown_connections 2020-11-20T12:09:43.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:43.930+0000 7fafee357700 1 -- 172.21.15.189:0/270610756 wait complete. 2020-11-20T12:09:43.936 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 16 2020-11-20T12:09:43.946 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:09:45.948 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:46.078 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.074+0000 7f80b359e700 1 Processor -- start 2020-11-20T12:09:46.078 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.078+0000 7f80b359e700 1 -- start start 2020-11-20T12:09:46.082 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.078+0000 7f80b359e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f80b413a6e0 0x7f80b413ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:46.083 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.078+0000 7f80b359e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b4067730 con 0x7f80b40ad3e0 2020-11-20T12:09:46.083 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.078+0000 7f80b359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b413dd40 con 0x7f80b40ad000 2020-11-20T12:09:46.083 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.078+0000 7f80b359e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80b4066050 con 0x7f80b413a6e0 2020-11-20T12:09:46.084 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.078+0000 7f80b2d9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f80b413a6e0 0x7f80b413ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:46.084 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.078+0000 7f80b259c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f80b40ad000 0x7f80b413e480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51442/0 (socket says 172.21.15.189:51442) 2020-11-20T12:09:46.084 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.078+0000 7f80b259c700 1 -- 172.21.15.189:0/673091758 learned_addr learned my addr 172.21.15.189:0/673091758 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:09:46.085 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.078+0000 7f80b2d9d700 1 -- 172.21.15.189:0/673091758 >> v1:172.21.15.189:6789/0 conn(0x7f80b40ad000 legacy=0x7f80b413e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:46.085 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.078+0000 7f80b2d9d700 1 -- 172.21.15.189:0/673091758 >> v1:172.21.15.47:6789/0 conn(0x7f80b40ad3e0 legacy=0x7f80b4066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:46.085 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.078+0000 7f80b2d9d700 1 -- 172.21.15.189:0/673091758 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b413dfe0 con 0x7f80b413a6e0 2020-11-20T12:09:46.086 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.078+0000 7f80b159a700 1 -- 172.21.15.189:0/673091758 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075503082 0 0) 0x7f80b413dd40 con 0x7f80b40ad000 2020-11-20T12:09:46.086 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.078+0000 7f80b2d9d700 1 --2- 172.21.15.189:0/673091758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80b413a6e0 0x7f80b413ad00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f80ac009c90 tx=0x7f80ac0091d0).ready entity=mon.1 client_cookie=2b79737437306b6c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:46.086 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.078+0000 7f80b159a700 1 -- 172.21.15.189:0/673091758 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac00bbc0 con 0x7f80b413a6e0 2020-11-20T12:09:46.086 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.078+0000 7f80b159a700 1 -- 172.21.15.189:0/673091758 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80ac00bd20 con 0x7f80b413a6e0 2020-11-20T12:09:46.086 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.078+0000 7f80b159a700 1 -- 172.21.15.189:0/673091758 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac011610 con 0x7f80b413a6e0 2020-11-20T12:09:46.087 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.078+0000 7f80b359e700 1 -- 172.21.15.189:0/673091758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80b413a6e0 msgr2=0x7f80b413ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:46.087 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.078+0000 7f80b359e700 1 --2- 172.21.15.189:0/673091758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80b413a6e0 0x7f80b413ad00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f80ac009c90 tx=0x7f80ac0091d0).stop 2020-11-20T12:09:46.087 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.078+0000 7f80b359e700 1 -- 172.21.15.189:0/673091758 shutdown_connections 2020-11-20T12:09:46.087 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.078+0000 7f80b359e700 1 --2- 172.21.15.189:0/673091758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80b413a6e0 0x7f80b413ad00 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:46.088 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.078+0000 7f80b359e700 1 -- 172.21.15.189:0/673091758 >> 172.21.15.189:0/673091758 conn(0x7f80b4062080 msgr2=0x7f80b4062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:46.088 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.078+0000 7f80b359e700 1 -- 172.21.15.189:0/673091758 shutdown_connections 2020-11-20T12:09:46.088 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.078+0000 7f80b359e700 1 -- 172.21.15.189:0/673091758 wait complete. 2020-11-20T12:09:46.088 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.082+0000 7f80b359e700 1 Processor -- start 2020-11-20T12:09:46.089 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.082+0000 7f80b359e700 1 -- start start 2020-11-20T12:09:46.091 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.082+0000 7f80b359e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f80b40ad000 0x7f80b4141820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:46.091 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.082+0000 7f80b359e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80b40ad3e0 0x7f80b40a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:46.091 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.082+0000 7f80b359e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80b40ab030 0x7f80b40a1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:46.092 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.082+0000 7f80b359e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b40a28a0 con 0x7f80b40ab030 2020-11-20T12:09:46.092 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.082+0000 7f80b359e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b40a2ad0 con 0x7f80b40ad3e0 2020-11-20T12:09:46.092 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.082+0000 7f80b359e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80b40a2d70 con 0x7f80b40ad000 2020-11-20T12:09:46.092 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.082+0000 7f80b1d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80b40ad3e0 0x7f80b40a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:46.092 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.082+0000 7f80b1d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80b40ad3e0 0x7f80b40a1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49828/0 (socket says 172.21.15.189:49828) 2020-11-20T12:09:46.093 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.082+0000 7f80b1d9b700 1 -- 172.21.15.189:0/293731535 learned_addr learned my addr 172.21.15.189:0/293731535 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:09:46.093 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.082+0000 7f80b259c700 1 --2- 172.21.15.189:0/293731535 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f80b40ad000 0x7f80b4141820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:46.093 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.082+0000 7f80b2d9d700 1 --2- 172.21.15.189:0/293731535 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80b40ab030 0x7f80b40a1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:46.093 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.082+0000 7f80b1d9b700 1 -- 172.21.15.189:0/293731535 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f80b40ad000 msgr2=0x7f80b4141820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:46.094 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.082+0000 7f80b1d9b700 1 --2- 172.21.15.189:0/293731535 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f80b40ad000 0x7f80b4141820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:46.094 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.082+0000 7f80b1d9b700 1 -- 172.21.15.189:0/293731535 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80b40ab030 msgr2=0x7f80b40a1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:46.094 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.082+0000 7f80b1d9b700 1 --2- 172.21.15.189:0/293731535 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80b40ab030 0x7f80b40a1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:46.094 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.082+0000 7f80b1d9b700 1 -- 172.21.15.189:0/293731535 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b40a3010 con 0x7f80b40ad3e0 2020-11-20T12:09:46.094 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.082+0000 7f80b1d9b700 1 --2- 172.21.15.189:0/293731535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80b40ad3e0 0x7f80b40a1780 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f809c00c6e0 tx=0x7f809c007c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:46.095 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.082+0000 7f80aa7fc700 1 -- 172.21.15.189:0/293731535 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809c007020 con 0x7f80b40ad3e0 2020-11-20T12:09:46.095 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.082+0000 7f80aa7fc700 1 -- 172.21.15.189:0/293731535 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809c007180 con 0x7f80b40ad3e0 2020-11-20T12:09:46.095 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.082+0000 7f80b359e700 1 -- 172.21.15.189:0/293731535 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80b40a0ce0 con 0x7f80b40ad3e0 2020-11-20T12:09:46.095 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.082+0000 7f80aa7fc700 1 -- 172.21.15.189:0/293731535 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809c00e560 con 0x7f80b40ad3e0 2020-11-20T12:09:46.095 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.082+0000 7f80b359e700 1 -- 172.21.15.189:0/293731535 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80b40a5390 con 0x7f80b40ad3e0 2020-11-20T12:09:46.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.086+0000 7f80b359e700 1 -- 172.21.15.189:0/293731535 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80b40a31a0 con 0x7f80b40ad3e0 2020-11-20T12:09:46.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.086+0000 7f80aa7fc700 1 -- 172.21.15.189:0/293731535 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f809c018020 con 0x7f80b40ad3e0 2020-11-20T12:09:46.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.086+0000 7f80aa7fc700 1 --2- 172.21.15.189:0/293731535 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f80900557c0 0x7f8090057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:46.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.086+0000 7f80b259c700 1 --2- 172.21.15.189:0/293731535 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f80900557c0 0x7f8090057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:46.097 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.086+0000 7f80aa7fc700 1 -- 172.21.15.189:0/293731535 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f809c06e0a0 con 0x7f80b40ad3e0 2020-11-20T12:09:46.097 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.086+0000 7f80b259c700 1 --2- 172.21.15.189:0/293731535 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f80900557c0 0x7f8090057c80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f80a40011a0 tx=0x7f80a4009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:46.097 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.090+0000 7f80aa7fc700 1 -- 172.21.15.189:0/293731535 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f809c043e90 con 0x7f80b40ad3e0 2020-11-20T12:09:46.296 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.294+0000 7f80b359e700 1 -- 172.21.15.189:0/293731535 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80b40446d0 con 0x7f80b40ad3e0 2020-11-20T12:09:46.297 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.294+0000 7f80aa7fc700 1 -- 172.21.15.189:0/293731535 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f809c0474b0 con 0x7f80b40ad3e0 2020-11-20T12:09:46.298 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:09:46.298 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4676,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.47:6833/4000022090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4000022090},{"type":"v1","addr":"172.21.15.47:6833","nonce":4000022090}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:08:20.595791+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/1375880901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1375880901},{"type":"v1","addr":"172.21.15.189:6835","nonce":1375880901}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:09:46.300 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.298+0000 7f80b359e700 1 -- 172.21.15.189:0/293731535 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f80900557c0 msgr2=0x7f8090057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:46.300 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.298+0000 7f80b359e700 1 --2- 172.21.15.189:0/293731535 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f80900557c0 0x7f8090057c80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f80a40011a0 tx=0x7f80a4009040).stop 2020-11-20T12:09:46.300 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.298+0000 7f80b359e700 1 -- 172.21.15.189:0/293731535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80b40ad3e0 msgr2=0x7f80b40a1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:46.301 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.298+0000 7f80b359e700 1 --2- 172.21.15.189:0/293731535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80b40ad3e0 0x7f80b40a1780 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f809c00c6e0 tx=0x7f809c007c50).stop 2020-11-20T12:09:46.301 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.298+0000 7f80b359e700 1 -- 172.21.15.189:0/293731535 shutdown_connections 2020-11-20T12:09:46.301 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.298+0000 7f80b359e700 1 --2- 172.21.15.189:0/293731535 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f80900557c0 0x7f8090057c80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:46.302 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.298+0000 7f80b359e700 1 --2- 172.21.15.189:0/293731535 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f80b40ab030 0x7f80b40a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:46.302 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.298+0000 7f80b359e700 1 --2- 172.21.15.189:0/293731535 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f80b40ad000 0x7f80b4141820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:46.302 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.298+0000 7f80b359e700 1 --2- 172.21.15.189:0/293731535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80b40ad3e0 0x7f80b40a1780 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:46.302 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.298+0000 7f80b359e700 1 -- 172.21.15.189:0/293731535 >> 172.21.15.189:0/293731535 conn(0x7f80b4062080 msgr2=0x7f80b40ac380 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:46.303 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.298+0000 7f80b359e700 1 -- 172.21.15.189:0/293731535 shutdown_connections 2020-11-20T12:09:46.303 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:46.298+0000 7f80b359e700 1 -- 172.21.15.189:0/293731535 wait complete. 2020-11-20T12:09:46.304 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 16 2020-11-20T12:09:46.312 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:09:48.313 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:48.443 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.442+0000 7f485cf36700 1 Processor -- start 2020-11-20T12:09:48.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.442+0000 7f485cf36700 1 -- start start 2020-11-20T12:09:48.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.442+0000 7f485cf36700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f48580a9800 0x7f485813ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:48.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.442+0000 7f485cf36700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f485813f260 0x7f485813b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:48.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.442+0000 7f485cf36700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48580676a0 con 0x7f485813be50 2020-11-20T12:09:48.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.442+0000 7f485cf36700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4858065ec0 con 0x7f485813f260 2020-11-20T12:09:48.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.442+0000 7f485cf36700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4858067c50 con 0x7f48580a9800 2020-11-20T12:09:48.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.442+0000 7f4855d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f485813f260 0x7f485813b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:48.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.442+0000 7f4855d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f485813f260 0x7f485813b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49834/0 (socket says 172.21.15.189:49834) 2020-11-20T12:09:48.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.442+0000 7f4855d9b700 1 -- 172.21.15.189:0/2139988589 learned_addr learned my addr 172.21.15.189:0/2139988589 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:09:48.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.442+0000 7f485659c700 1 --2- 172.21.15.189:0/2139988589 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f48580a9800 0x7f485813ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:48.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.442+0000 7f4855d9b700 1 -- 172.21.15.189:0/2139988589 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f48580a9800 msgr2=0x7f485813ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:48.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.442+0000 7f4855d9b700 1 --2- 172.21.15.189:0/2139988589 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f48580a9800 0x7f485813ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:48.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.442+0000 7f4855d9b700 1 -- 172.21.15.189:0/2139988589 >> v1:172.21.15.47:6789/0 conn(0x7f485813be50 legacy=0x7f4858060170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:48.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.442+0000 7f4855d9b700 1 -- 172.21.15.189:0/2139988589 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4858067160 con 0x7f485813f260 2020-11-20T12:09:48.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.442+0000 7f4855d9b700 1 --2- 172.21.15.189:0/2139988589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f485813f260 0x7f485813b960 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f4848009d50 tx=0x7f4848009290).ready entity=mon.1 client_cookie=1c56bb2112d49ae8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:48.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.442+0000 7f485559a700 1 -- 172.21.15.189:0/2139988589 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484800bd10 con 0x7f485813f260 2020-11-20T12:09:48.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f485559a700 1 -- 172.21.15.189:0/2139988589 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484800be70 con 0x7f485813f260 2020-11-20T12:09:48.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f485559a700 1 -- 172.21.15.189:0/2139988589 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4848011610 con 0x7f485813f260 2020-11-20T12:09:48.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f485cf36700 1 -- 172.21.15.189:0/2139988589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f485813f260 msgr2=0x7f485813b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:48.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f485cf36700 1 --2- 172.21.15.189:0/2139988589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f485813f260 0x7f485813b960 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f4848009d50 tx=0x7f4848009290).stop 2020-11-20T12:09:48.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f485cf36700 1 -- 172.21.15.189:0/2139988589 shutdown_connections 2020-11-20T12:09:48.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f485cf36700 1 --2- 172.21.15.189:0/2139988589 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f48580a9800 0x7f485813ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:48.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f485cf36700 1 --2- 172.21.15.189:0/2139988589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f485813f260 0x7f485813b960 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:48.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f485cf36700 1 -- 172.21.15.189:0/2139988589 >> 172.21.15.189:0/2139988589 conn(0x7f4858061f60 msgr2=0x7f4858062370 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:48.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f485cf36700 1 -- 172.21.15.189:0/2139988589 shutdown_connections 2020-11-20T12:09:48.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f485cf36700 1 -- 172.21.15.189:0/2139988589 wait complete. 2020-11-20T12:09:48.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f485cf36700 1 Processor -- start 2020-11-20T12:09:48.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f485cf36700 1 -- start start 2020-11-20T12:09:48.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f485cf36700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48580a9800 0x7f48580abf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:48.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f485cf36700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f485813be50 0x7f4858151a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:48.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f485cf36700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f485813f260 0x7f48580a66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:48.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f485cf36700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48581512e0 con 0x7f48580a9800 2020-11-20T12:09:48.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f485cf36700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4858151580 con 0x7f485813be50 2020-11-20T12:09:48.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f485cf36700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4858151f10 con 0x7f485813f260 2020-11-20T12:09:48.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f4855d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f485813be50 0x7f4858151a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:48.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f4855d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f485813be50 0x7f4858151a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49842/0 (socket says 172.21.15.189:49842) 2020-11-20T12:09:48.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f4855d9b700 1 -- 172.21.15.189:0/3664820494 learned_addr learned my addr 172.21.15.189:0/3664820494 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:09:48.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f4856d9d700 1 --2- 172.21.15.189:0/3664820494 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f485813f260 0x7f48580a66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:48.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f485659c700 1 --2- 172.21.15.189:0/3664820494 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48580a9800 0x7f48580abf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:48.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f4855d9b700 1 -- 172.21.15.189:0/3664820494 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f485813f260 msgr2=0x7f48580a66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:48.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f4855d9b700 1 --2- 172.21.15.189:0/3664820494 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f485813f260 0x7f48580a66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:48.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f4855d9b700 1 -- 172.21.15.189:0/3664820494 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48580a9800 msgr2=0x7f48580abf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:48.459 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.446+0000 7f4855d9b700 1 --2- 172.21.15.189:0/3664820494 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48580a9800 0x7f48580abf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:48.459 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.450+0000 7f4855d9b700 1 -- 172.21.15.189:0/3664820494 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485809dc60 con 0x7f485813be50 2020-11-20T12:09:48.459 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.450+0000 7f4856d9d700 1 --2- 172.21.15.189:0/3664820494 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f485813f260 0x7f48580a66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:09:48.459 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.450+0000 7f4855d9b700 1 --2- 172.21.15.189:0/3664820494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f485813be50 0x7f4858151a20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f4848011000 tx=0x7f4848016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:48.459 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.450+0000 7f48467fc700 1 -- 172.21.15.189:0/3664820494 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4848010700 con 0x7f485813be50 2020-11-20T12:09:48.460 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.450+0000 7f48467fc700 1 -- 172.21.15.189:0/3664820494 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4848010860 con 0x7f485813be50 2020-11-20T12:09:48.460 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.450+0000 7f485cf36700 1 -- 172.21.15.189:0/3664820494 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48580a3db0 con 0x7f485813be50 2020-11-20T12:09:48.460 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.450+0000 7f48467fc700 1 -- 172.21.15.189:0/3664820494 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4848010c00 con 0x7f485813be50 2020-11-20T12:09:48.460 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.450+0000 7f485cf36700 1 -- 172.21.15.189:0/3664820494 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48580a4360 con 0x7f485813be50 2020-11-20T12:09:48.461 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.450+0000 7f48467fc700 1 -- 172.21.15.189:0/3664820494 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f4848017070 con 0x7f485813be50 2020-11-20T12:09:48.461 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.450+0000 7f485cf36700 1 -- 172.21.15.189:0/3664820494 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4858151740 con 0x7f485813be50 2020-11-20T12:09:48.461 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.450+0000 7f48467fc700 1 --2- 172.21.15.189:0/3664820494 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f482c0557c0 0x7f482c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:48.461 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.450+0000 7f48467fc700 1 -- 172.21.15.189:0/3664820494 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f48480707b0 con 0x7f485813be50 2020-11-20T12:09:48.461 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.450+0000 7f485659c700 1 --2- 172.21.15.189:0/3664820494 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f482c0557c0 0x7f482c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:48.462 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.450+0000 7f485659c700 1 --2- 172.21.15.189:0/3664820494 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f482c0557c0 0x7f482c057c80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f484c00d8f0 tx=0x7f484c008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:48.462 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.458+0000 7f48467fc700 1 -- 172.21.15.189:0/3664820494 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4848046620 con 0x7f485813be50 2020-11-20T12:09:48.664 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.662+0000 7f485cf36700 1 -- 172.21.15.189:0/3664820494 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4858044f60 con 0x7f485813be50 2020-11-20T12:09:48.665 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.662+0000 7f48467fc700 1 -- 172.21.15.189:0/3664820494 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f4848015020 con 0x7f485813be50 2020-11-20T12:09:48.665 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:09:48.666 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4676,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.47:6833/4000022090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4000022090},{"type":"v1","addr":"172.21.15.47:6833","nonce":4000022090}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:08:20.595791+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/1375880901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1375880901},{"type":"v1","addr":"172.21.15.189:6835","nonce":1375880901}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:09:48.667 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.666+0000 7f485cf36700 1 -- 172.21.15.189:0/3664820494 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f482c0557c0 msgr2=0x7f482c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:48.668 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.666+0000 7f485cf36700 1 --2- 172.21.15.189:0/3664820494 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f482c0557c0 0x7f482c057c80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f484c00d8f0 tx=0x7f484c008040).stop 2020-11-20T12:09:48.668 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.666+0000 7f485cf36700 1 -- 172.21.15.189:0/3664820494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f485813be50 msgr2=0x7f4858151a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:48.668 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.666+0000 7f485cf36700 1 --2- 172.21.15.189:0/3664820494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f485813be50 0x7f4858151a20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f4848011000 tx=0x7f4848016000).stop 2020-11-20T12:09:48.668 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.666+0000 7f485cf36700 1 -- 172.21.15.189:0/3664820494 shutdown_connections 2020-11-20T12:09:48.669 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.666+0000 7f485cf36700 1 --2- 172.21.15.189:0/3664820494 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f482c0557c0 0x7f482c057c80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:48.669 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.666+0000 7f485cf36700 1 --2- 172.21.15.189:0/3664820494 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f48580a9800 0x7f48580abf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:48.669 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.666+0000 7f485cf36700 1 --2- 172.21.15.189:0/3664820494 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f485813f260 0x7f48580a66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:48.669 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.666+0000 7f485cf36700 1 --2- 172.21.15.189:0/3664820494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f485813be50 0x7f4858151a20 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:48.669 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.666+0000 7f485cf36700 1 -- 172.21.15.189:0/3664820494 >> 172.21.15.189:0/3664820494 conn(0x7f4858061f60 msgr2=0x7f48580608c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:48.670 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.666+0000 7f485cf36700 1 -- 172.21.15.189:0/3664820494 shutdown_connections 2020-11-20T12:09:48.670 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:48.666+0000 7f485cf36700 1 -- 172.21.15.189:0/3664820494 wait complete. 2020-11-20T12:09:48.670 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 16 2020-11-20T12:09:48.679 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:09:50.681 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:50.824 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.822+0000 7fcb607b1700 1 Processor -- start 2020-11-20T12:09:50.824 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.822+0000 7fcb607b1700 1 -- start start 2020-11-20T12:09:50.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.822+0000 7fcb607b1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcb5813ab80 0x7fcb5813e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:50.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.822+0000 7fcb607b1700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fcb5813e6c0 0x7fcb5813cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:50.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.822+0000 7fcb607b1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb5813b300 con 0x7fcb58066de0 2020-11-20T12:09:50.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.822+0000 7fcb607b1700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb5813dd30 con 0x7fcb5813e6c0 2020-11-20T12:09:50.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.822+0000 7fcb607b1700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb5813aff0 con 0x7fcb5813ab80 2020-11-20T12:09:50.827 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.822+0000 7fcb5dd4c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcb5813ab80 0x7fcb5813e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:50.827 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.822+0000 7fcb5e54d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fcb58066de0 0x7fcb580671c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51472/0 (socket says 172.21.15.189:51472) 2020-11-20T12:09:50.827 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.822+0000 7fcb5e54d700 1 -- 172.21.15.189:0/1054261441 learned_addr learned my addr 172.21.15.189:0/1054261441 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:09:50.827 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.822+0000 7fcb5ed4e700 1 --2- 172.21.15.189:0/1054261441 >> v2:172.21.15.47:3300/0 conn(0x7fcb5813e6c0 0x7fcb5813cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:50.829 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.826+0000 7fcb5dd4c700 1 -- 172.21.15.189:0/1054261441 >> v2:172.21.15.47:3300/0 conn(0x7fcb5813e6c0 msgr2=0x7fcb5813cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:50.829 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.826+0000 7fcb5dd4c700 1 --2- 172.21.15.189:0/1054261441 >> v2:172.21.15.47:3300/0 conn(0x7fcb5813e6c0 0x7fcb5813cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:50.829 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.826+0000 7fcb5dd4c700 1 -- 172.21.15.189:0/1054261441 >> v1:172.21.15.189:6789/0 conn(0x7fcb58066de0 legacy=0x7fcb580671c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:50.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.826+0000 7fcb5d54b700 1 -- 172.21.15.189:0/1054261441 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313772814 0 0) 0x7fcb5813b300 con 0x7fcb58066de0 2020-11-20T12:09:50.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.826+0000 7fcb5dd4c700 1 -- 172.21.15.189:0/1054261441 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5813d390 con 0x7fcb5813ab80 2020-11-20T12:09:50.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.826+0000 7fcb5ed4e700 1 --2- 172.21.15.189:0/1054261441 >> v2:172.21.15.47:3300/0 conn(0x7fcb5813e6c0 0x7fcb5813cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:09:50.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.826+0000 7fcb5dd4c700 1 --2- 172.21.15.189:0/1054261441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb5813ab80 0x7fcb5813e1d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fcb4c00b9a0 tx=0x7fcb4c006df0).ready entity=mon.1 client_cookie=d623767d0be5a346 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:50.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.826+0000 7fcb5d54b700 1 -- 172.21.15.189:0/1054261441 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c009770 con 0x7fcb5813ab80 2020-11-20T12:09:50.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.826+0000 7fcb5d54b700 1 -- 172.21.15.189:0/1054261441 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb4c0098d0 con 0x7fcb5813ab80 2020-11-20T12:09:50.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.826+0000 7fcb5d54b700 1 -- 172.21.15.189:0/1054261441 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c009c50 con 0x7fcb5813ab80 2020-11-20T12:09:50.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.826+0000 7fcb607b1700 1 -- 172.21.15.189:0/1054261441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb5813ab80 msgr2=0x7fcb5813e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:50.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.826+0000 7fcb607b1700 1 --2- 172.21.15.189:0/1054261441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb5813ab80 0x7fcb5813e1d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fcb4c00b9a0 tx=0x7fcb4c006df0).stop 2020-11-20T12:09:50.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.826+0000 7fcb607b1700 1 -- 172.21.15.189:0/1054261441 shutdown_connections 2020-11-20T12:09:50.832 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.826+0000 7fcb607b1700 1 --2- 172.21.15.189:0/1054261441 >> v2:172.21.15.47:3300/0 conn(0x7fcb5813e6c0 0x7fcb5813cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:50.832 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.826+0000 7fcb607b1700 1 --2- 172.21.15.189:0/1054261441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb5813ab80 0x7fcb5813e1d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:50.832 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.826+0000 7fcb607b1700 1 -- 172.21.15.189:0/1054261441 >> 172.21.15.189:0/1054261441 conn(0x7fcb58061f60 msgr2=0x7fcb58062340 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:50.832 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.826+0000 7fcb607b1700 1 -- 172.21.15.189:0/1054261441 shutdown_connections 2020-11-20T12:09:50.833 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.826+0000 7fcb607b1700 1 -- 172.21.15.189:0/1054261441 wait complete. 2020-11-20T12:09:50.835 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.830+0000 7fcb607b1700 1 Processor -- start 2020-11-20T12:09:50.835 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.830+0000 7fcb607b1700 1 -- start start 2020-11-20T12:09:50.836 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.830+0000 7fcb607b1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb58066de0 0x7fcb58140ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:50.836 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.830+0000 7fcb607b1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb5813ab80 0x7fcb580a3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:50.836 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.830+0000 7fcb607b1700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcb5813e6c0 0x7fcb58137130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:50.836 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.830+0000 7fcb607b1700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb580a3c10 con 0x7fcb5813ab80 2020-11-20T12:09:50.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.830+0000 7fcb607b1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb580a2560 con 0x7fcb58066de0 2020-11-20T12:09:50.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.830+0000 7fcb607b1700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb580a2800 con 0x7fcb5813e6c0 2020-11-20T12:09:50.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.830+0000 7fcb5e54d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb58066de0 0x7fcb58140ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:50.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.830+0000 7fcb5e54d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb58066de0 0x7fcb58140ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49856/0 (socket says 172.21.15.189:49856) 2020-11-20T12:09:50.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.830+0000 7fcb5e54d700 1 -- 172.21.15.189:0/3254227486 learned_addr learned my addr 172.21.15.189:0/3254227486 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:09:50.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.830+0000 7fcb5dd4c700 1 --2- 172.21.15.189:0/3254227486 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb5813ab80 0x7fcb580a3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:50.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.830+0000 7fcb5ed4e700 1 --2- 172.21.15.189:0/3254227486 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcb5813e6c0 0x7fcb58137130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:50.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.830+0000 7fcb5e54d700 1 -- 172.21.15.189:0/3254227486 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcb5813e6c0 msgr2=0x7fcb58137130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:50.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.830+0000 7fcb5e54d700 1 --2- 172.21.15.189:0/3254227486 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcb5813e6c0 0x7fcb58137130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:50.839 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.830+0000 7fcb5e54d700 1 -- 172.21.15.189:0/3254227486 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb5813ab80 msgr2=0x7fcb580a3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:50.839 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.830+0000 7fcb5e54d700 1 --2- 172.21.15.189:0/3254227486 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb5813ab80 0x7fcb580a3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:50.839 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.830+0000 7fcb5e54d700 1 -- 172.21.15.189:0/3254227486 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb580a2aa0 con 0x7fcb58066de0 2020-11-20T12:09:50.839 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.830+0000 7fcb5ed4e700 1 --2- 172.21.15.189:0/3254227486 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcb5813e6c0 0x7fcb58137130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:09:50.839 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.830+0000 7fcb5dd4c700 1 --2- 172.21.15.189:0/3254227486 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb5813ab80 0x7fcb580a3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:09:50.840 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.834+0000 7fcb5e54d700 1 --2- 172.21.15.189:0/3254227486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb58066de0 0x7fcb58140ab0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fcb5400c720 tx=0x7fcb54007ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:50.840 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.834+0000 7fcb4a7fc700 1 -- 172.21.15.189:0/3254227486 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb54007320 con 0x7fcb58066de0 2020-11-20T12:09:50.840 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.834+0000 7fcb4a7fc700 1 -- 172.21.15.189:0/3254227486 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb54007480 con 0x7fcb58066de0 2020-11-20T12:09:50.840 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.834+0000 7fcb607b1700 1 -- 172.21.15.189:0/3254227486 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb580a1fe0 con 0x7fcb58066de0 2020-11-20T12:09:50.840 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.834+0000 7fcb4a7fc700 1 -- 172.21.15.189:0/3254227486 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb54005800 con 0x7fcb58066de0 2020-11-20T12:09:50.841 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.834+0000 7fcb607b1700 1 -- 172.21.15.189:0/3254227486 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb58137620 con 0x7fcb58066de0 2020-11-20T12:09:50.841 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.834+0000 7fcb607b1700 1 -- 172.21.15.189:0/3254227486 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb580a2c30 con 0x7fcb58066de0 2020-11-20T12:09:50.842 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.834+0000 7fcb4a7fc700 1 -- 172.21.15.189:0/3254227486 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fcb54010030 con 0x7fcb58066de0 2020-11-20T12:09:50.842 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.834+0000 7fcb4a7fc700 1 --2- 172.21.15.189:0/3254227486 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcb300557c0 0x7fcb30057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:50.842 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.838+0000 7fcb4a7fc700 1 -- 172.21.15.189:0/3254227486 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fcb5406ec10 con 0x7fcb58066de0 2020-11-20T12:09:50.842 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.838+0000 7fcb5dd4c700 1 --2- 172.21.15.189:0/3254227486 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcb300557c0 0x7fcb30057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:50.843 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.838+0000 7fcb5dd4c700 1 --2- 172.21.15.189:0/3254227486 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcb300557c0 0x7fcb30057c80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fcb4c00b150 tx=0x7fcb4c015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:50.843 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:50.842+0000 7fcb4a7fc700 1 -- 172.21.15.189:0/3254227486 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcb54044a00 con 0x7fcb58066de0 2020-11-20T12:09:51.044 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:51.042+0000 7fcb607b1700 1 -- 172.21.15.189:0/3254227486 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb58044f60 con 0x7fcb58066de0 2020-11-20T12:09:51.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:51.042+0000 7fcb4a7fc700 1 -- 172.21.15.189:0/3254227486 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fcb54048020 con 0x7fcb58066de0 2020-11-20T12:09:51.045 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:09:51.045 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4676,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.47:6833/4000022090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4000022090},{"type":"v1","addr":"172.21.15.47:6833","nonce":4000022090}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":16}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:08:20.595791+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4849},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4849":{"gid":4849,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/1375880901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1375880901},{"type":"v1","addr":"172.21.15.189:6835","nonce":1375880901}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:09:51.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:51.046+0000 7fcb607b1700 1 -- 172.21.15.189:0/3254227486 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcb300557c0 msgr2=0x7fcb30057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:51.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:51.046+0000 7fcb607b1700 1 --2- 172.21.15.189:0/3254227486 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcb300557c0 0x7fcb30057c80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fcb4c00b150 tx=0x7fcb4c015000).stop 2020-11-20T12:09:51.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:51.046+0000 7fcb607b1700 1 -- 172.21.15.189:0/3254227486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb58066de0 msgr2=0x7fcb58140ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:51.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:51.046+0000 7fcb607b1700 1 --2- 172.21.15.189:0/3254227486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb58066de0 0x7fcb58140ab0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fcb5400c720 tx=0x7fcb54007ac0).stop 2020-11-20T12:09:51.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:51.046+0000 7fcb607b1700 1 -- 172.21.15.189:0/3254227486 shutdown_connections 2020-11-20T12:09:51.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:51.046+0000 7fcb607b1700 1 --2- 172.21.15.189:0/3254227486 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcb5813e6c0 0x7fcb58137130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:51.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:51.046+0000 7fcb607b1700 1 --2- 172.21.15.189:0/3254227486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb58066de0 0x7fcb58140ab0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:51.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:51.046+0000 7fcb607b1700 1 --2- 172.21.15.189:0/3254227486 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcb300557c0 0x7fcb30057c80 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:51.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:51.046+0000 7fcb607b1700 1 --2- 172.21.15.189:0/3254227486 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcb5813ab80 0x7fcb580a3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:51.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:51.046+0000 7fcb607b1700 1 -- 172.21.15.189:0/3254227486 >> 172.21.15.189:0/3254227486 conn(0x7fcb58061f60 msgr2=0x7fcb580aecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:51.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:51.046+0000 7fcb607b1700 1 -- 172.21.15.189:0/3254227486 shutdown_connections 2020-11-20T12:09:51.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:51.046+0000 7fcb607b1700 1 -- 172.21.15.189:0/3254227486 wait complete. 2020-11-20T12:09:51.050 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 16 2020-11-20T12:09:51.064 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:09:52.881 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:09:52.887 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:09:53.065 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:53.194 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa47359e700 1 Processor -- start 2020-11-20T12:09:53.195 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa47359e700 1 -- start start 2020-11-20T12:09:53.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa47359e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa4740ad3e0 0x7fa474066d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:53.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa47359e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa47413a6e0 0x7fa47413ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:53.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa47359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa474067730 con 0x7fa4740ad000 2020-11-20T12:09:53.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa47359e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa474066050 con 0x7fa47413a6e0 2020-11-20T12:09:53.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa47359e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa474138190 con 0x7fa4740ad3e0 2020-11-20T12:09:53.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa47259c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa4740ad000 0x7fa47413e480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51486/0 (socket says 172.21.15.189:51486) 2020-11-20T12:09:53.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa47259c700 1 -- 172.21.15.189:0/3839555815 learned_addr learned my addr 172.21.15.189:0/3839555815 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:09:53.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa472d9d700 1 --2- 172.21.15.189:0/3839555815 >> v2:172.21.15.189:3300/0 conn(0x7fa47413a6e0 0x7fa47413ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:53.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa471d9b700 1 --2- 172.21.15.189:0/3839555815 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa4740ad3e0 0x7fa474066d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:53.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa47159a700 1 -- 172.21.15.189:0/3839555815 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4074480807 0 0) 0x7fa474067730 con 0x7fa4740ad000 2020-11-20T12:09:53.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa47159a700 1 -- 172.21.15.189:0/3839555815 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa454003240 con 0x7fa4740ad000 2020-11-20T12:09:53.202 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa472d9d700 1 -- 172.21.15.189:0/3839555815 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa4740ad3e0 msgr2=0x7fa474066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:53.202 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa472d9d700 1 --2- 172.21.15.189:0/3839555815 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa4740ad3e0 0x7fa474066d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:53.202 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa472d9d700 1 -- 172.21.15.189:0/3839555815 >> v1:172.21.15.189:6789/0 conn(0x7fa4740ad000 legacy=0x7fa47413e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:53.202 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa472d9d700 1 -- 172.21.15.189:0/3839555815 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa47413dd40 con 0x7fa47413a6e0 2020-11-20T12:09:53.203 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa471d9b700 1 --2- 172.21.15.189:0/3839555815 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa4740ad3e0 0x7fa474066d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:09:53.203 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa472d9d700 1 --2- 172.21.15.189:0/3839555815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa47413a6e0 0x7fa47413ad00 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fa45c00b9d0 tx=0x7fa45c006df0).ready entity=mon.1 client_cookie=26d537ec55409959 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:53.203 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa47159a700 1 -- 172.21.15.189:0/3839555815 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45c009800 con 0x7fa47413a6e0 2020-11-20T12:09:53.203 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa47159a700 1 -- 172.21.15.189:0/3839555815 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa45c009960 con 0x7fa47413a6e0 2020-11-20T12:09:53.203 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa47159a700 1 -- 172.21.15.189:0/3839555815 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45c009ce0 con 0x7fa47413a6e0 2020-11-20T12:09:53.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa47359e700 1 -- 172.21.15.189:0/3839555815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa47413a6e0 msgr2=0x7fa47413ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:53.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa47359e700 1 --2- 172.21.15.189:0/3839555815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa47413a6e0 0x7fa47413ad00 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fa45c00b9d0 tx=0x7fa45c006df0).stop 2020-11-20T12:09:53.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa47359e700 1 -- 172.21.15.189:0/3839555815 shutdown_connections 2020-11-20T12:09:53.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa47359e700 1 --2- 172.21.15.189:0/3839555815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa47413a6e0 0x7fa47413ad00 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:53.205 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa47359e700 1 --2- 172.21.15.189:0/3839555815 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa4740ad3e0 0x7fa474066d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:53.205 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa47359e700 1 -- 172.21.15.189:0/3839555815 >> 172.21.15.189:0/3839555815 conn(0x7fa474062080 msgr2=0x7fa474062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:53.205 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa47359e700 1 -- 172.21.15.189:0/3839555815 shutdown_connections 2020-11-20T12:09:53.205 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.194+0000 7fa47359e700 1 -- 172.21.15.189:0/3839555815 wait complete. 2020-11-20T12:09:53.205 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.198+0000 7fa47359e700 1 Processor -- start 2020-11-20T12:09:53.206 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.198+0000 7fa47359e700 1 -- start start 2020-11-20T12:09:53.206 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.198+0000 7fa47359e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4740ad000 0x7fa474140790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:53.206 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.198+0000 7fa47359e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4740ad3e0 0x7fa47414cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:53.206 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.198+0000 7fa47359e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa4740a69b0 0x7fa4740a7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:53.206 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.198+0000 7fa47359e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4740a6040 con 0x7fa4740ad3e0 2020-11-20T12:09:53.207 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.198+0000 7fa47359e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4740a6270 con 0x7fa4740ad000 2020-11-20T12:09:53.207 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.198+0000 7fa47359e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4740a6510 con 0x7fa4740a69b0 2020-11-20T12:09:53.207 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.198+0000 7fa47259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4740ad000 0x7fa474140790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:53.207 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.198+0000 7fa47259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4740ad000 0x7fa474140790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49870/0 (socket says 172.21.15.189:49870) 2020-11-20T12:09:53.207 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.198+0000 7fa47259c700 1 -- 172.21.15.189:0/2263819354 learned_addr learned my addr 172.21.15.189:0/2263819354 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:09:53.208 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.198+0000 7fa472d9d700 1 --2- 172.21.15.189:0/2263819354 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa4740a69b0 0x7fa4740a7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:53.208 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.198+0000 7fa471d9b700 1 --2- 172.21.15.189:0/2263819354 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4740ad3e0 0x7fa47414cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:53.208 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.198+0000 7fa47259c700 1 -- 172.21.15.189:0/2263819354 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa4740a69b0 msgr2=0x7fa4740a7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:53.208 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.198+0000 7fa47259c700 1 --2- 172.21.15.189:0/2263819354 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa4740a69b0 0x7fa4740a7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:53.209 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.198+0000 7fa47259c700 1 -- 172.21.15.189:0/2263819354 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4740ad3e0 msgr2=0x7fa47414cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:53.209 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.198+0000 7fa47259c700 1 --2- 172.21.15.189:0/2263819354 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4740ad3e0 0x7fa47414cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:53.209 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.198+0000 7fa47259c700 1 -- 172.21.15.189:0/2263819354 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa474141480 con 0x7fa4740ad000 2020-11-20T12:09:53.209 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.198+0000 7fa471d9b700 1 --2- 172.21.15.189:0/2263819354 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4740ad3e0 0x7fa47414cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:09:53.209 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.198+0000 7fa47259c700 1 --2- 172.21.15.189:0/2263819354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4740ad000 0x7fa474140790 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fa46400c6a0 tx=0x7fa464007ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:53.210 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.198+0000 7fa46a7fc700 1 -- 172.21.15.189:0/2263819354 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4640072e0 con 0x7fa4740ad000 2020-11-20T12:09:53.210 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.198+0000 7fa46a7fc700 1 -- 172.21.15.189:0/2263819354 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa464007440 con 0x7fa4740ad000 2020-11-20T12:09:53.210 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.198+0000 7fa47359e700 1 -- 172.21.15.189:0/2263819354 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa47414d2c0 con 0x7fa4740ad000 2020-11-20T12:09:53.210 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.198+0000 7fa46a7fc700 1 -- 172.21.15.189:0/2263819354 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa464005820 con 0x7fa4740ad000 2020-11-20T12:09:53.210 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.198+0000 7fa47359e700 1 -- 172.21.15.189:0/2263819354 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4740a7aa0 con 0x7fa4740ad000 2020-11-20T12:09:53.211 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.202+0000 7fa47359e700 1 -- 172.21.15.189:0/2263819354 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4740b1680 con 0x7fa4740ad000 2020-11-20T12:09:53.212 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.202+0000 7fa46a7fc700 1 -- 172.21.15.189:0/2263819354 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fa464010070 con 0x7fa4740ad000 2020-11-20T12:09:53.212 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.202+0000 7fa46a7fc700 1 --2- 172.21.15.189:0/2263819354 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa4480557c0 0x7fa448057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:53.212 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.202+0000 7fa46a7fc700 1 -- 172.21.15.189:0/2263819354 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6331+0+0 (secure 0 0 0) 0x7fa46406de80 con 0x7fa4740ad000 2020-11-20T12:09:53.212 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.202+0000 7fa471d9b700 1 --2- 172.21.15.189:0/2263819354 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa4480557c0 0x7fa448057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:53.213 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.202+0000 7fa471d9b700 1 --2- 172.21.15.189:0/2263819354 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa4480557c0 0x7fa448057c80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fa474141b30 tx=0x7fa46c008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:53.213 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.206+0000 7fa46a7fc700 1 -- 172.21.15.189:0/2263819354 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa4740b1680 con 0x7fa4740ad000 2020-11-20T12:09:53.410 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.410+0000 7fa47359e700 1 -- 172.21.15.189:0/2263819354 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa47413b2f0 con 0x7fa4740ad000 2020-11-20T12:09:53.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.410+0000 7fa46a7fc700 1 -- 172.21.15.189:0/2263819354 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+1784 (secure 0 0 0) 0x7fa47413b2f0 con 0x7fa4740ad000 2020-11-20T12:09:53.411 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:09:53.411 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:09:52.565220+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4676},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4676":{"gid":4676,"name":"b","rank":0,"incarnation":17,"state":"up:replay","state_seq":2,"addr":"172.21.15.47:6833/4000022090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4000022090},{"type":"v1","addr":"172.21.15.47:6833","nonce":4000022090}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:09:53.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.410+0000 7fa47359e700 1 -- 172.21.15.189:0/2263819354 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa4480557c0 msgr2=0x7fa448057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:53.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.410+0000 7fa47359e700 1 --2- 172.21.15.189:0/2263819354 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa4480557c0 0x7fa448057c80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fa474141b30 tx=0x7fa46c008040).stop 2020-11-20T12:09:53.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.410+0000 7fa47359e700 1 -- 172.21.15.189:0/2263819354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4740ad000 msgr2=0x7fa474140790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:53.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.410+0000 7fa47359e700 1 --2- 172.21.15.189:0/2263819354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4740ad000 0x7fa474140790 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fa46400c6a0 tx=0x7fa464007ac0).stop 2020-11-20T12:09:53.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.410+0000 7fa47359e700 1 -- 172.21.15.189:0/2263819354 shutdown_connections 2020-11-20T12:09:53.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.410+0000 7fa47359e700 1 --2- 172.21.15.189:0/2263819354 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa4740a69b0 0x7fa4740a7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:53.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.410+0000 7fa47359e700 1 --2- 172.21.15.189:0/2263819354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4740ad000 0x7fa474140790 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:53.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.410+0000 7fa47359e700 1 --2- 172.21.15.189:0/2263819354 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa4480557c0 0x7fa448057c80 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:53.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.410+0000 7fa47359e700 1 --2- 172.21.15.189:0/2263819354 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa4740ad3e0 0x7fa47414cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:53.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.410+0000 7fa47359e700 1 -- 172.21.15.189:0/2263819354 >> 172.21.15.189:0/2263819354 conn(0x7fa474062080 msgr2=0x7fa474138f10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:53.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.414+0000 7fa47359e700 1 -- 172.21.15.189:0/2263819354 shutdown_connections 2020-11-20T12:09:53.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.414+0000 7fa47359e700 1 -- 172.21.15.189:0/2263819354 wait complete. 2020-11-20T12:09:53.421 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 17 2020-11-20T12:09:53.431 INFO:tasks.mds_thrash.fs.[None]:mds.a down, removed from mdsmap 2020-11-20T12:09:53.432 INFO:tasks.mds_thrash.fs.[None]:waiting for mds cluster to stabilize... 2020-11-20T12:09:53.432 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:53.561 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.558+0000 7f916b11e700 1 Processor -- start 2020-11-20T12:09:53.562 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.558+0000 7f916b11e700 1 -- start start 2020-11-20T12:09:53.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916b11e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9164060310 0x7f91640a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:53.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916b11e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f916413fc70 0x7f9164142060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:53.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916b11e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9164063b50 con 0x7f9164134460 2020-11-20T12:09:53.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916b11e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9164067a00 con 0x7f9164060310 2020-11-20T12:09:53.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916b11e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9164066500 con 0x7f916413fc70 2020-11-20T12:09:53.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916a11c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9164060310 0x7f91640a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:53.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916991b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9164134460 0x7f91641348e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51502/0 (socket says 172.21.15.189:51502) 2020-11-20T12:09:53.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916a11c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9164060310 0x7f91640a11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49878/0 (socket says 172.21.15.189:49878) 2020-11-20T12:09:53.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916991b700 1 -- 172.21.15.189:0/1273964889 learned_addr learned my addr 172.21.15.189:0/1273964889 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:09:53.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916a91d700 1 --2- 172.21.15.189:0/1273964889 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f916413fc70 0x7f9164142060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:53.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916a11c700 1 -- 172.21.15.189:0/1273964889 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f916413fc70 msgr2=0x7f9164142060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:09:53.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916a11c700 1 --2- 172.21.15.189:0/1273964889 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f916413fc70 0x7f9164142060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:53.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916a11c700 1 -- 172.21.15.189:0/1273964889 >> v1:172.21.15.189:6789/0 conn(0x7f9164134460 legacy=0x7f91641348e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:53.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916a11c700 1 -- 172.21.15.189:0/1273964889 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91640a09a0 con 0x7f9164060310 2020-11-20T12:09:53.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916911a700 1 -- 172.21.15.189:0/1273964889 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853113338 0 0) 0x7f9164063b50 con 0x7f9164134460 2020-11-20T12:09:53.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916a11c700 1 --2- 172.21.15.189:0/1273964889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9164060310 0x7f91640a11c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f916000b9a0 tx=0x7f9160006df0).ready entity=mon.1 client_cookie=80cde27c966b7d27 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:53.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916911a700 1 -- 172.21.15.189:0/1273964889 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91600097b0 con 0x7f9164060310 2020-11-20T12:09:53.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916911a700 1 -- 172.21.15.189:0/1273964889 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9160009910 con 0x7f9164060310 2020-11-20T12:09:53.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916911a700 1 -- 172.21.15.189:0/1273964889 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9160009c90 con 0x7f9164060310 2020-11-20T12:09:53.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916b11e700 1 -- 172.21.15.189:0/1273964889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9164060310 msgr2=0x7f91640a11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:53.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916b11e700 1 --2- 172.21.15.189:0/1273964889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9164060310 0x7f91640a11c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f916000b9a0 tx=0x7f9160006df0).stop 2020-11-20T12:09:53.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916b11e700 1 -- 172.21.15.189:0/1273964889 shutdown_connections 2020-11-20T12:09:53.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916b11e700 1 --2- 172.21.15.189:0/1273964889 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f916413fc70 0x7f9164142060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:53.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916b11e700 1 --2- 172.21.15.189:0/1273964889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9164060310 0x7f91640a11c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:53.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916b11e700 1 -- 172.21.15.189:0/1273964889 >> 172.21.15.189:0/1273964889 conn(0x7f9164063210 msgr2=0x7f9164060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:53.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916b11e700 1 -- 172.21.15.189:0/1273964889 shutdown_connections 2020-11-20T12:09:53.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.562+0000 7f916b11e700 1 -- 172.21.15.189:0/1273964889 wait complete. 2020-11-20T12:09:53.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.566+0000 7f916b11e700 1 Processor -- start 2020-11-20T12:09:53.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.566+0000 7f916b11e700 1 -- start start 2020-11-20T12:09:53.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.566+0000 7f916b11e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9164060310 0x7f916409f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:53.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.566+0000 7f916b11e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9164134460 0x7f9164148b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:53.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.566+0000 7f916b11e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f916413fc70 0x7f9164136360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:53.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.566+0000 7f916b11e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9164149060 con 0x7f9164060310 2020-11-20T12:09:53.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.566+0000 7f916b11e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91640a07e0 con 0x7f916413fc70 2020-11-20T12:09:53.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.566+0000 7f916b11e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9164149280 con 0x7f9164134460 2020-11-20T12:09:53.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.566+0000 7f916a91d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f916413fc70 0x7f9164136360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:53.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.566+0000 7f916a91d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f916413fc70 0x7f9164136360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49888/0 (socket says 172.21.15.189:49888) 2020-11-20T12:09:53.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.566+0000 7f916a91d700 1 -- 172.21.15.189:0/3950361092 learned_addr learned my addr 172.21.15.189:0/3950361092 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:09:53.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.566+0000 7f916a11c700 1 --2- 172.21.15.189:0/3950361092 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9164060310 0x7f916409f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:53.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.566+0000 7f916991b700 1 --2- 172.21.15.189:0/3950361092 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9164134460 0x7f9164148b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:53.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.566+0000 7f916a91d700 1 -- 172.21.15.189:0/3950361092 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9164134460 msgr2=0x7f9164148b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:53.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.566+0000 7f916a91d700 1 --2- 172.21.15.189:0/3950361092 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9164134460 0x7f9164148b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:53.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.566+0000 7f916a91d700 1 -- 172.21.15.189:0/3950361092 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9164060310 msgr2=0x7f916409f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:53.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.566+0000 7f916a91d700 1 --2- 172.21.15.189:0/3950361092 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9164060310 0x7f916409f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:53.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.566+0000 7f916a91d700 1 -- 172.21.15.189:0/3950361092 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9164149520 con 0x7f916413fc70 2020-11-20T12:09:53.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.566+0000 7f916991b700 1 --2- 172.21.15.189:0/3950361092 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9164134460 0x7f9164148b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:09:53.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.566+0000 7f916a91d700 1 --2- 172.21.15.189:0/3950361092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f916413fc70 0x7f9164136360 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f915c00db70 tx=0x7f915c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:53.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.566+0000 7f91527fc700 1 -- 172.21.15.189:0/3950361092 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c009920 con 0x7f916413fc70 2020-11-20T12:09:53.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.566+0000 7f91527fc700 1 -- 172.21.15.189:0/3950361092 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f915c009a80 con 0x7f916413fc70 2020-11-20T12:09:53.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.566+0000 7f91527fc700 1 -- 172.21.15.189:0/3950361092 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c009e20 con 0x7f916413fc70 2020-11-20T12:09:53.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.566+0000 7f916b11e700 1 -- 172.21.15.189:0/3950361092 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91641497c0 con 0x7f916413fc70 2020-11-20T12:09:53.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.566+0000 7f916b11e700 1 -- 172.21.15.189:0/3950361092 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9164136850 con 0x7f916413fc70 2020-11-20T12:09:53.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.570+0000 7f916b11e700 1 -- 172.21.15.189:0/3950361092 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9164149950 con 0x7f916413fc70 2020-11-20T12:09:53.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.570+0000 7f91527fc700 1 -- 172.21.15.189:0/3950361092 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f915c013690 con 0x7f916413fc70 2020-11-20T12:09:53.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.570+0000 7f91527fc700 1 --2- 172.21.15.189:0/3950361092 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f913c0557c0 0x7f913c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:53.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.570+0000 7f91527fc700 1 -- 172.21.15.189:0/3950361092 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6331+0+0 (secure 0 0 0) 0x7f915c06ed70 con 0x7f916413fc70 2020-11-20T12:09:53.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.570+0000 7f916a11c700 1 --2- 172.21.15.189:0/3950361092 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f913c0557c0 0x7f913c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:53.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.570+0000 7f916a11c700 1 --2- 172.21.15.189:0/3950361092 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f913c0557c0 0x7f913c057c80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f916000b150 tx=0x7f9160015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:53.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.578+0000 7f91527fc700 1 -- 172.21.15.189:0/3950361092 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f915c044b10 con 0x7f916413fc70 2020-11-20T12:09:53.781 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.778+0000 7f916b11e700 1 -- 172.21.15.189:0/3950361092 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91640446d0 con 0x7f916413fc70 2020-11-20T12:09:53.782 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.782+0000 7f91527fc700 1 -- 172.21.15.189:0/3950361092 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+1788 (secure 0 0 0) 0x7f915c048130 con 0x7f916413fc70 2020-11-20T12:09:53.782 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:09:53.783 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:09:53.605025+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4676},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4676":{"gid":4676,"name":"b","rank":0,"incarnation":17,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.47:6833/4000022090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4000022090},{"type":"v1","addr":"172.21.15.47:6833","nonce":4000022090}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:09:53.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.782+0000 7f916b11e700 1 -- 172.21.15.189:0/3950361092 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f913c0557c0 msgr2=0x7f913c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:53.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.782+0000 7f916b11e700 1 --2- 172.21.15.189:0/3950361092 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f913c0557c0 0x7f913c057c80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f916000b150 tx=0x7f9160015000).stop 2020-11-20T12:09:53.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.782+0000 7f916b11e700 1 -- 172.21.15.189:0/3950361092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f916413fc70 msgr2=0x7f9164136360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:53.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.782+0000 7f916b11e700 1 --2- 172.21.15.189:0/3950361092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f916413fc70 0x7f9164136360 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f915c00db70 tx=0x7f915c008040).stop 2020-11-20T12:09:53.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.782+0000 7f916b11e700 1 -- 172.21.15.189:0/3950361092 shutdown_connections 2020-11-20T12:09:53.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.782+0000 7f916b11e700 1 --2- 172.21.15.189:0/3950361092 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f913c0557c0 0x7f913c057c80 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:53.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.782+0000 7f916b11e700 1 --2- 172.21.15.189:0/3950361092 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9164060310 0x7f916409f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:53.790 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.782+0000 7f916b11e700 1 --2- 172.21.15.189:0/3950361092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f916413fc70 0x7f9164136360 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:53.790 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.782+0000 7f916b11e700 1 --2- 172.21.15.189:0/3950361092 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9164134460 0x7f9164148b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:53.790 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.782+0000 7f916b11e700 1 -- 172.21.15.189:0/3950361092 >> 172.21.15.189:0/3950361092 conn(0x7f9164063210 msgr2=0x7f91640a8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:53.791 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.782+0000 7f916b11e700 1 -- 172.21.15.189:0/3950361092 shutdown_connections 2020-11-20T12:09:53.791 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:53.782+0000 7f916b11e700 1 -- 172.21.15.189:0/3950361092 wait complete. 2020-11-20T12:09:53.791 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 18 2020-11-20T12:09:53.798 INFO:tasks.mds_thrash.fs.[None]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 18, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2020-11-20T12:04:54.012110+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 18, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4676": { "addr": "172.21.15.47:6833/4000022090", "addrs": { "addrvec": [ { "addr": "172.21.15.47:6832", "nonce": 4000022090, "type": "v2" }, { "addr": "172.21.15.47:6833", "nonce": 4000022090, "type": "v1" } ] }, "export_targets": [], "features": 4540138297136906239, "flags": 0, "gid": 4676, "incarnation": 17, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 25, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2020-11-20T12:09:53.605025+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4676 } } } ], "standbys": [] } 2020-11-20T12:09:55.637 INFO:tasks.ceph.mds.b.smithi047.stderr:2020-11-20T12:09:55.639+0000 7f9fd0f77700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:09:55.800 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:09:55.932 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.930+0000 7f2b19463700 1 Processor -- start 2020-11-20T12:09:55.933 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.930+0000 7f2b19463700 1 -- start start 2020-11-20T12:09:55.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.930+0000 7f2b19463700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2b140612b0 0x7f2b140a0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:55.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.930+0000 7f2b19463700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b14068020 con 0x7f2b141376c0 2020-11-20T12:09:55.935 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.930+0000 7f2b19463700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b140682c0 con 0x7f2b14067b50 2020-11-20T12:09:55.935 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.930+0000 7f2b19463700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b14061690 con 0x7f2b140612b0 2020-11-20T12:09:55.935 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.934+0000 7f2b0bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2b14067b50 0x7f2b14136fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51518/0 (socket says 172.21.15.189:51518) 2020-11-20T12:09:55.935 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.934+0000 7f2b0bfff700 1 -- 172.21.15.189:0/3305840402 learned_addr learned my addr 172.21.15.189:0/3305840402 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:09:55.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.934+0000 7f2b13fff700 1 --2- 172.21.15.189:0/3305840402 >> v2:172.21.15.47:3300/0 conn(0x7f2b140612b0 0x7f2b140a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:55.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.934+0000 7f2b137fe700 1 -- 172.21.15.189:0/3305840402 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935703483 0 0) 0x7f2b140682c0 con 0x7f2b14067b50 2020-11-20T12:09:55.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.934+0000 7f2b137fe700 1 -- 172.21.15.189:0/3305840402 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2af4003240 con 0x7f2b14067b50 2020-11-20T12:09:55.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.934+0000 7f2b137fe700 1 -- 172.21.15.189:0/3305840402 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2916761580 0 0) 0x7f2af4003240 con 0x7f2b14067b50 2020-11-20T12:09:55.937 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.934+0000 7f2b13fff700 1 -- 172.21.15.189:0/3305840402 >> v1:172.21.15.189:6789/0 conn(0x7f2b14067b50 legacy=0x7f2b14136fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:55.937 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.934+0000 7f2b13fff700 1 -- 172.21.15.189:0/3305840402 >> v1:172.21.15.47:6789/0 conn(0x7f2b141376c0 legacy=0x7f2b14133a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:55.937 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.934+0000 7f2b13fff700 1 -- 172.21.15.189:0/3305840402 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b14136aa0 con 0x7f2b140612b0 2020-11-20T12:09:55.937 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.934+0000 7f2b13fff700 1 --2- 172.21.15.189:0/3305840402 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b140612b0 0x7f2b140a0960 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f2b0400b920 tx=0x7f2b04006df0).ready entity=mon.0 client_cookie=24862e1e3863576f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:55.941 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.934+0000 7f2b137fe700 1 -- 172.21.15.189:0/3305840402 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2afc0014e0 con 0x7f2b140612b0 2020-11-20T12:09:55.941 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.934+0000 7f2b137fe700 1 -- 172.21.15.189:0/3305840402 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b04011400 con 0x7f2b140612b0 2020-11-20T12:09:55.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.934+0000 7f2b137fe700 1 -- 172.21.15.189:0/3305840402 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b04011780 con 0x7f2b140612b0 2020-11-20T12:09:55.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.934+0000 7f2b19463700 1 -- 172.21.15.189:0/3305840402 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b140612b0 msgr2=0x7f2b140a0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:55.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.934+0000 7f2b19463700 1 --2- 172.21.15.189:0/3305840402 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b140612b0 0x7f2b140a0960 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f2b0400b920 tx=0x7f2b04006df0).stop 2020-11-20T12:09:55.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.934+0000 7f2b19463700 1 -- 172.21.15.189:0/3305840402 shutdown_connections 2020-11-20T12:09:55.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.934+0000 7f2b19463700 1 --2- 172.21.15.189:0/3305840402 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b140612b0 0x7f2b140a0960 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:55.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.934+0000 7f2b19463700 1 -- 172.21.15.189:0/3305840402 >> 172.21.15.189:0/3305840402 conn(0x7f2b14063210 msgr2=0x7f2b14060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:55.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.934+0000 7f2b19463700 1 -- 172.21.15.189:0/3305840402 shutdown_connections 2020-11-20T12:09:55.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.934+0000 7f2b19463700 1 -- 172.21.15.189:0/3305840402 wait complete. 2020-11-20T12:09:55.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.934+0000 7f2b19463700 1 Processor -- start 2020-11-20T12:09:55.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.938+0000 7f2b19463700 1 -- start start 2020-11-20T12:09:55.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.938+0000 7f2b19463700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b140612b0 0x7f2b14136100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:55.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.938+0000 7f2b19463700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2b14067b50 0x7f2b140ab280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:55.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.938+0000 7f2b19463700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b141376c0 0x7f2b140ab770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:55.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.938+0000 7f2b19463700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b140a4790 con 0x7f2b141376c0 2020-11-20T12:09:55.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.938+0000 7f2b19463700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b140a5470 con 0x7f2b140612b0 2020-11-20T12:09:55.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.938+0000 7f2b19463700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b140abc60 con 0x7f2b14067b50 2020-11-20T12:09:55.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.938+0000 7f2b13fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b140612b0 0x7f2b14136100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:55.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.938+0000 7f2b13fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b140612b0 0x7f2b14136100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49900/0 (socket says 172.21.15.189:49900) 2020-11-20T12:09:55.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.938+0000 7f2b13fff700 1 -- 172.21.15.189:0/3842558881 learned_addr learned my addr 172.21.15.189:0/3842558881 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:09:55.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.938+0000 7f2b18c62700 1 --2- 172.21.15.189:0/3842558881 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b141376c0 0x7f2b140ab770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:55.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.938+0000 7f2b0bfff700 1 --2- 172.21.15.189:0/3842558881 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2b14067b50 0x7f2b140ab280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:55.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.938+0000 7f2b13fff700 1 -- 172.21.15.189:0/3842558881 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2b14067b50 msgr2=0x7f2b140ab280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:55.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.938+0000 7f2b13fff700 1 --2- 172.21.15.189:0/3842558881 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2b14067b50 0x7f2b140ab280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:55.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.938+0000 7f2b13fff700 1 -- 172.21.15.189:0/3842558881 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b141376c0 msgr2=0x7f2b140ab770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:55.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.938+0000 7f2b13fff700 1 --2- 172.21.15.189:0/3842558881 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b141376c0 0x7f2b140ab770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:55.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.938+0000 7f2b13fff700 1 -- 172.21.15.189:0/3842558881 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b140a5660 con 0x7f2b140612b0 2020-11-20T12:09:55.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.938+0000 7f2b0bfff700 1 --2- 172.21.15.189:0/3842558881 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2b14067b50 0x7f2b140ab280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:09:55.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.938+0000 7f2b18c62700 1 --2- 172.21.15.189:0/3842558881 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b141376c0 0x7f2b140ab770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:09:55.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.938+0000 7f2b13fff700 1 --2- 172.21.15.189:0/3842558881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b140612b0 0x7f2b14136100 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f2b140625f0 tx=0x7f2b04015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:55.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.938+0000 7f2b10ff9700 1 -- 172.21.15.189:0/3842558881 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b04009ab0 con 0x7f2b140612b0 2020-11-20T12:09:55.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.938+0000 7f2b10ff9700 1 -- 172.21.15.189:0/3842558881 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b04009c10 con 0x7f2b140612b0 2020-11-20T12:09:55.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.938+0000 7f2b19463700 1 -- 172.21.15.189:0/3842558881 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b140a5900 con 0x7f2b140612b0 2020-11-20T12:09:55.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.938+0000 7f2b10ff9700 1 -- 172.21.15.189:0/3842558881 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b0401e460 con 0x7f2b140612b0 2020-11-20T12:09:55.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.938+0000 7f2b19463700 1 -- 172.21.15.189:0/3842558881 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b140ac2e0 con 0x7f2b140612b0 2020-11-20T12:09:55.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.938+0000 7f2b19463700 1 -- 172.21.15.189:0/3842558881 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b140ab0f0 con 0x7f2b140612b0 2020-11-20T12:09:55.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.942+0000 7f2b10ff9700 1 -- 172.21.15.189:0/3842558881 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f2b04025030 con 0x7f2b140612b0 2020-11-20T12:09:55.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.942+0000 7f2b10ff9700 1 --2- 172.21.15.189:0/3842558881 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2af00557c0 0x7f2af0057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:09:55.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.942+0000 7f2b10ff9700 1 -- 172.21.15.189:0/3842558881 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6331+0+0 (secure 0 0 0) 0x7f2b0406fc90 con 0x7f2b140612b0 2020-11-20T12:09:55.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.942+0000 7f2b0bfff700 1 --2- 172.21.15.189:0/3842558881 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2af00557c0 0x7f2af0057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:09:55.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.942+0000 7f2b0bfff700 1 --2- 172.21.15.189:0/3842558881 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2af00557c0 0x7f2af0057c80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f2b1413aa80 tx=0x7f2afc009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:09:55.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:55.946+0000 7f2b10ff9700 1 -- 172.21.15.189:0/3842558881 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2b04045ae0 con 0x7f2b140612b0 2020-11-20T12:09:56.169 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:56.166+0000 7f2b19463700 1 -- 172.21.15.189:0/3842558881 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b140a0e50 con 0x7f2b140612b0 2020-11-20T12:09:56.170 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:56.170+0000 7f2b10ff9700 1 -- 172.21.15.189:0/3842558881 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+1785 (secure 0 0 0) 0x7f2b04023030 con 0x7f2b140612b0 2020-11-20T12:09:56.170 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:09:56.170 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:09:55.624531+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4676},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4676":{"gid":4676,"name":"b","rank":0,"incarnation":17,"state":"up:active","state_seq":26,"addr":"172.21.15.47:6833/4000022090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4000022090},{"type":"v1","addr":"172.21.15.47:6833","nonce":4000022090}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:09:56.172 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:56.170+0000 7f2b19463700 1 -- 172.21.15.189:0/3842558881 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2af00557c0 msgr2=0x7f2af0057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:56.172 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:56.170+0000 7f2b19463700 1 --2- 172.21.15.189:0/3842558881 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2af00557c0 0x7f2af0057c80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f2b1413aa80 tx=0x7f2afc009040).stop 2020-11-20T12:09:56.172 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:56.170+0000 7f2b19463700 1 -- 172.21.15.189:0/3842558881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b140612b0 msgr2=0x7f2b14136100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:09:56.173 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:56.170+0000 7f2b19463700 1 --2- 172.21.15.189:0/3842558881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b140612b0 0x7f2b14136100 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f2b140625f0 tx=0x7f2b04015000).stop 2020-11-20T12:09:56.173 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:56.170+0000 7f2b19463700 1 -- 172.21.15.189:0/3842558881 shutdown_connections 2020-11-20T12:09:56.173 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:56.170+0000 7f2b19463700 1 --2- 172.21.15.189:0/3842558881 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2af00557c0 0x7f2af0057c80 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:56.173 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:56.170+0000 7f2b19463700 1 --2- 172.21.15.189:0/3842558881 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2b141376c0 0x7f2b140ab770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:56.173 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:56.170+0000 7f2b19463700 1 --2- 172.21.15.189:0/3842558881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b140612b0 0x7f2b14136100 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:56.174 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:56.170+0000 7f2b19463700 1 --2- 172.21.15.189:0/3842558881 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2b14067b50 0x7f2b140ab280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:09:56.174 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:56.170+0000 7f2b19463700 1 -- 172.21.15.189:0/3842558881 >> 172.21.15.189:0/3842558881 conn(0x7f2b14063210 msgr2=0x7f2b140a8370 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:09:56.174 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:56.170+0000 7f2b19463700 1 -- 172.21.15.189:0/3842558881 shutdown_connections 2020-11-20T12:09:56.174 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:09:56.170+0000 7f2b19463700 1 -- 172.21.15.189:0/3842558881 wait complete. 2020-11-20T12:09:56.175 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 20 2020-11-20T12:09:56.185 INFO:tasks.mds_thrash.fs.[None]:mds.b has gained rank=0, replacing gid=4849 2020-11-20T12:09:56.186 INFO:tasks.mds_thrash.fs.[None]:waiting for 4 secs before reviving mds.a 2020-11-20T12:10:00.187 INFO:tasks.mds_thrash.fs.[None]:reviving mds.a 2020-11-20T12:10:00.187 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:10:00.187 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:10:00.192 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:10:00.193 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:10:00.270 INFO:tasks.ceph.mds.a.smithi189.stdout:starting mds.a at 2020-11-20T12:10:00.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.318+0000 7f719e4aa700 1 Processor -- start 2020-11-20T12:10:00.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.318+0000 7f719e4aa700 1 -- start start 2020-11-20T12:10:00.320 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.318+0000 7f719e4aa700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7198066de0 0x7f719813ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:00.320 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.318+0000 7f719e4aa700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f71980671c0 0x7f719813e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:00.320 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.318+0000 7f719e4aa700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f719813b070 con 0x7f719813e6c0 2020-11-20T12:10:00.320 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.318+0000 7f719e4aa700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f719813b300 con 0x7f7198066de0 2020-11-20T12:10:00.320 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.318+0000 7f719e4aa700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f719813dd30 con 0x7f71980671c0 2020-11-20T12:10:00.321 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.318+0000 7f718ffff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f71980671c0 0x7f719813e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:00.321 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.318+0000 7f7197fff700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7198066de0 0x7f719813ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:00.321 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.318+0000 7f718ffff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f71980671c0 0x7f719813e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.189:42956/0 (socket says 172.21.15.189:42956) 2020-11-20T12:10:00.321 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.318+0000 7f719ca47700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f719813e6c0 0x7f719813cea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.189:46210/0 (socket says 172.21.15.189:46210) 2020-11-20T12:10:00.322 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.318+0000 7f718ffff700 1 -- 172.21.15.189:0/53970751 learned_addr learned my addr 172.21.15.189:0/53970751 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:10:00.326 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.318+0000 7f718ffff700 1 -- 172.21.15.189:0/53970751 >> v2:172.21.15.47:3300/0 conn(0x7f7198066de0 msgr2=0x7f719813ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:00.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.318+0000 7f718ffff700 1 --2- 172.21.15.189:0/53970751 >> v2:172.21.15.47:3300/0 conn(0x7f7198066de0 0x7f719813ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:00.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.318+0000 7f718ffff700 1 -- 172.21.15.189:0/53970751 >> v1:172.21.15.47:6789/0 conn(0x7f719813e6c0 legacy=0x7f719813cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:00.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.318+0000 7f71977fe700 1 -- 172.21.15.189:0/53970751 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988121541 0 0) 0x7f719813b070 con 0x7f719813e6c0 2020-11-20T12:10:00.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.318+0000 7f718ffff700 1 -- 172.21.15.189:0/53970751 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7198067620 con 0x7f71980671c0 2020-11-20T12:10:00.328 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.318+0000 7f7197fff700 1 --2- 172.21.15.189:0/53970751 >> v2:172.21.15.47:3300/0 conn(0x7f7198066de0 0x7f719813ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:10:00.328 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.318+0000 7f718ffff700 1 --2- 172.21.15.189:0/53970751 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f71980671c0 0x7f719813e1d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f7180006fd0 tx=0x7f7180006eb0).ready entity=mon.2 client_cookie=3c122221a4b604bf server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:00.328 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.318+0000 7f71977fe700 1 -- 172.21.15.189:0/53970751 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71800098a0 con 0x7f71980671c0 2020-11-20T12:10:00.328 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.318+0000 7f71977fe700 1 -- 172.21.15.189:0/53970751 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7180009a00 con 0x7f71980671c0 2020-11-20T12:10:00.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.318+0000 7f71977fe700 1 -- 172.21.15.189:0/53970751 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7180009d80 con 0x7f71980671c0 2020-11-20T12:10:00.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.318+0000 7f719e4aa700 1 -- 172.21.15.189:0/53970751 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f71980671c0 msgr2=0x7f719813e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:00.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.318+0000 7f719e4aa700 1 --2- 172.21.15.189:0/53970751 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f71980671c0 0x7f719813e1d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f7180006fd0 tx=0x7f7180006eb0).stop 2020-11-20T12:10:00.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f719e4aa700 1 -- 172.21.15.189:0/53970751 shutdown_connections 2020-11-20T12:10:00.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f719e4aa700 1 --2- 172.21.15.189:0/53970751 >> v2:172.21.15.47:3300/0 conn(0x7f7198066de0 0x7f719813ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:00.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f719e4aa700 1 --2- 172.21.15.189:0/53970751 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f71980671c0 0x7f719813e1d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:00.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f719e4aa700 1 -- 172.21.15.189:0/53970751 >> 172.21.15.189:0/53970751 conn(0x7f7198061f60 msgr2=0x7f7198062340 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:00.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f719e4aa700 1 -- 172.21.15.189:0/53970751 shutdown_connections 2020-11-20T12:10:00.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f719e4aa700 1 -- 172.21.15.189:0/53970751 wait complete. 2020-11-20T12:10:00.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f719e4aa700 1 Processor -- start 2020-11-20T12:10:00.331 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f719e4aa700 1 -- start start 2020-11-20T12:10:00.331 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f719e4aa700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7198066de0 0x7f71981396e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:00.331 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f719e4aa700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71980671c0 0x7f719807f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:00.331 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f719e4aa700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f719813e6c0 0x7f719807f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:00.331 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f719e4aa700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719807f9e0 con 0x7f71980671c0 2020-11-20T12:10:00.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f719e4aa700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71980a3050 con 0x7f7198066de0 2020-11-20T12:10:00.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f719e4aa700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71980a32f0 con 0x7f719813e6c0 2020-11-20T12:10:00.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f7197fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7198066de0 0x7f71981396e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:00.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f7197fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7198066de0 0x7f71981396e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49926/0 (socket says 172.21.15.189:49926) 2020-11-20T12:10:00.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f7197fff700 1 -- 172.21.15.189:0/4075956212 learned_addr learned my addr 172.21.15.189:0/4075956212 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:10:00.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f719ca47700 1 --2- 172.21.15.189:0/4075956212 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f719813e6c0 0x7f719807f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:00.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f718ffff700 1 --2- 172.21.15.189:0/4075956212 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71980671c0 0x7f719807f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:00.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f7197fff700 1 -- 172.21.15.189:0/4075956212 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f719813e6c0 msgr2=0x7f719807f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:00.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f7197fff700 1 --2- 172.21.15.189:0/4075956212 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f719813e6c0 0x7f719807f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:00.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f7197fff700 1 -- 172.21.15.189:0/4075956212 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71980671c0 msgr2=0x7f719807f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:00.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f7197fff700 1 --2- 172.21.15.189:0/4075956212 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71980671c0 0x7f719807f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:00.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f7197fff700 1 -- 172.21.15.189:0/4075956212 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71980a3590 con 0x7f7198066de0 2020-11-20T12:10:00.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f719ca47700 1 --2- 172.21.15.189:0/4075956212 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f719813e6c0 0x7f719807f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:10:00.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f7197fff700 1 --2- 172.21.15.189:0/4075956212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7198066de0 0x7f71981396e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f718800ef50 tx=0x7f7188006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:00.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f7194ff9700 1 -- 172.21.15.189:0/4075956212 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718800ca80 con 0x7f7198066de0 2020-11-20T12:10:00.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f7194ff9700 1 -- 172.21.15.189:0/4075956212 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f718800cbe0 con 0x7f7198066de0 2020-11-20T12:10:00.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f719e4aa700 1 -- 172.21.15.189:0/4075956212 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71980a2d40 con 0x7f7198066de0 2020-11-20T12:10:00.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f7194ff9700 1 -- 172.21.15.189:0/4075956212 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71880123f0 con 0x7f7198066de0 2020-11-20T12:10:00.336 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.322+0000 7f719e4aa700 1 -- 172.21.15.189:0/4075956212 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71980a1dc0 con 0x7f7198066de0 2020-11-20T12:10:00.338 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.326+0000 7f719e4aa700 1 -- 172.21.15.189:0/4075956212 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7198044f60 con 0x7f7198066de0 2020-11-20T12:10:00.338 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.326+0000 7f7194ff9700 1 -- 172.21.15.189:0/4075956212 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f7188012760 con 0x7f7198066de0 2020-11-20T12:10:00.339 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.326+0000 7f7194ff9700 1 --2- 172.21.15.189:0/4075956212 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f71740557c0 0x7f7174057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:00.339 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.326+0000 7f7194ff9700 1 -- 172.21.15.189:0/4075956212 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6331+0+0 (secure 0 0 0) 0x7f718806ed30 con 0x7f7198066de0 2020-11-20T12:10:00.339 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.326+0000 7f718ffff700 1 --2- 172.21.15.189:0/4075956212 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f71740557c0 0x7f7174057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:00.339 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.326+0000 7f718ffff700 1 --2- 172.21.15.189:0/4075956212 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f71740557c0 0x7f7174057c80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f718000b1b0 tx=0x7f7180015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:00.340 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.334+0000 7f7194ff9700 1 -- 172.21.15.189:0/4075956212 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7188044ad0 con 0x7f7198066de0 2020-11-20T12:10:00.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.570+0000 7f719e4aa700 1 -- 172.21.15.189:0/4075956212 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7198061380 con 0x7f7198066de0 2020-11-20T12:10:00.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.574+0000 7f7194ff9700 1 -- 172.21.15.189:0/4075956212 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f71880480f0 con 0x7f7198066de0 2020-11-20T12:10:00.576 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:10:00.576 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5053,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/3540904386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3540904386},{"type":"v1","addr":"172.21.15.189:6835","nonce":3540904386}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:09:55.624531+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4676},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4676":{"gid":4676,"name":"b","rank":0,"incarnation":17,"state":"up:active","state_seq":26,"addr":"172.21.15.47:6833/4000022090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4000022090},{"type":"v1","addr":"172.21.15.47:6833","nonce":4000022090}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:10:00.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.578+0000 7f719e4aa700 1 -- 172.21.15.189:0/4075956212 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f71740557c0 msgr2=0x7f7174057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:00.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.578+0000 7f719e4aa700 1 --2- 172.21.15.189:0/4075956212 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f71740557c0 0x7f7174057c80 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f718000b1b0 tx=0x7f7180015000).stop 2020-11-20T12:10:00.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.578+0000 7f719e4aa700 1 -- 172.21.15.189:0/4075956212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7198066de0 msgr2=0x7f71981396e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:00.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.578+0000 7f719e4aa700 1 --2- 172.21.15.189:0/4075956212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7198066de0 0x7f71981396e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f718800ef50 tx=0x7f7188006df0).stop 2020-11-20T12:10:00.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.578+0000 7f719e4aa700 1 -- 172.21.15.189:0/4075956212 shutdown_connections 2020-11-20T12:10:00.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.578+0000 7f719e4aa700 1 --2- 172.21.15.189:0/4075956212 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f719813e6c0 0x7f719807f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:00.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.578+0000 7f719e4aa700 1 --2- 172.21.15.189:0/4075956212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7198066de0 0x7f71981396e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:00.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.578+0000 7f719e4aa700 1 --2- 172.21.15.189:0/4075956212 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f71740557c0 0x7f7174057c80 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:00.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.578+0000 7f719e4aa700 1 --2- 172.21.15.189:0/4075956212 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f71980671c0 0x7f719807f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:00.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.578+0000 7f719e4aa700 1 -- 172.21.15.189:0/4075956212 >> 172.21.15.189:0/4075956212 conn(0x7f7198061f60 msgr2=0x7f71980afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:00.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.578+0000 7f719e4aa700 1 -- 172.21.15.189:0/4075956212 shutdown_connections 2020-11-20T12:10:00.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:00.578+0000 7f719e4aa700 1 -- 172.21.15.189:0/4075956212 wait complete. 2020-11-20T12:10:00.584 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 21 2020-11-20T12:10:00.596 INFO:tasks.mds_thrash.fs.[None]:mds.a reported in up:standby state 2020-11-20T12:10:00.596 INFO:tasks.mds_thrash.fs.[None]:waiting for 56 secs before thrashing 2020-11-20T12:10:23.122 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:10:23.125 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:10:53.158 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:10:53.188 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:10:56.597 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:10:56.726 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.725+0000 7fc58eede700 1 Processor -- start 2020-11-20T12:10:56.726 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.725+0000 7fc58eede700 1 -- start start 2020-11-20T12:10:56.728 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.725+0000 7fc58eede700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc5880612b0 0x7fc5880a0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:56.729 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.725+0000 7fc58eede700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc5881376c0 0x7fc588133a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:56.729 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.725+0000 7fc58eede700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc588068020 con 0x7fc588067b50 2020-11-20T12:10:56.729 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.725+0000 7fc58eede700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5880682c0 con 0x7fc5880612b0 2020-11-20T12:10:56.729 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.725+0000 7fc58eede700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc588061690 con 0x7fc5881376c0 2020-11-20T12:10:56.729 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.725+0000 7fc58dedc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc5880612b0 0x7fc5880a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:56.730 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.725+0000 7fc58dedc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc5880612b0 0x7fc5880a0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49936/0 (socket says 172.21.15.189:49936) 2020-11-20T12:10:56.730 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.725+0000 7fc58dedc700 1 -- 172.21.15.189:0/1306085944 learned_addr learned my addr 172.21.15.189:0/1306085944 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:10:56.730 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.725+0000 7fc58e6dd700 1 --2- 172.21.15.189:0/1306085944 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc5881376c0 0x7fc588133a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:56.730 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.725+0000 7fc58dedc700 1 -- 172.21.15.189:0/1306085944 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc5881376c0 msgr2=0x7fc588133a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:10:56.730 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.725+0000 7fc58dedc700 1 --2- 172.21.15.189:0/1306085944 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc5881376c0 0x7fc588133a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:56.731 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.725+0000 7fc58dedc700 1 -- 172.21.15.189:0/1306085944 >> v1:172.21.15.47:6789/0 conn(0x7fc588067b50 legacy=0x7fc588136fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:56.731 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.725+0000 7fc58dedc700 1 -- 172.21.15.189:0/1306085944 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc588136aa0 con 0x7fc5880612b0 2020-11-20T12:10:56.731 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.725+0000 7fc58e6dd700 1 --2- 172.21.15.189:0/1306085944 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc5881376c0 0x7fc588133a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:10:56.731 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.725+0000 7fc58dedc700 1 --2- 172.21.15.189:0/1306085944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5880612b0 0x7fc5880a0960 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fc584006fd0 tx=0x7fc584006f40).ready entity=mon.1 client_cookie=4e118f2f0d59c743 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:56.731 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.725+0000 7fc58ceda700 1 -- 172.21.15.189:0/1306085944 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58400cb30 con 0x7fc5880612b0 2020-11-20T12:10:56.732 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.725+0000 7fc58ceda700 1 -- 172.21.15.189:0/1306085944 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc58400cc90 con 0x7fc5880612b0 2020-11-20T12:10:56.732 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.725+0000 7fc58ceda700 1 -- 172.21.15.189:0/1306085944 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc584010620 con 0x7fc5880612b0 2020-11-20T12:10:56.732 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58eede700 1 -- 172.21.15.189:0/1306085944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5880612b0 msgr2=0x7fc5880a0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:56.735 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58eede700 1 --2- 172.21.15.189:0/1306085944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5880612b0 0x7fc5880a0960 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fc584006fd0 tx=0x7fc584006f40).stop 2020-11-20T12:10:56.735 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58eede700 1 -- 172.21.15.189:0/1306085944 shutdown_connections 2020-11-20T12:10:56.735 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58eede700 1 --2- 172.21.15.189:0/1306085944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5880612b0 0x7fc5880a0960 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:56.735 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58eede700 1 --2- 172.21.15.189:0/1306085944 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc5881376c0 0x7fc588133a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:56.736 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58eede700 1 -- 172.21.15.189:0/1306085944 >> 172.21.15.189:0/1306085944 conn(0x7fc588063210 msgr2=0x7fc588060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:56.736 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58eede700 1 -- 172.21.15.189:0/1306085944 shutdown_connections 2020-11-20T12:10:56.736 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58eede700 1 -- 172.21.15.189:0/1306085944 wait complete. 2020-11-20T12:10:56.736 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58eede700 1 Processor -- start 2020-11-20T12:10:56.737 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58eede700 1 -- start start 2020-11-20T12:10:56.737 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58eede700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5880612b0 0x7fc58813c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:56.737 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58eede700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc588067b50 0x7fc5881492f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:56.737 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58eede700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc5881376c0 0x7fc58813d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:56.737 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58eede700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc588134780 con 0x7fc5880612b0 2020-11-20T12:10:56.738 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58eede700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc588149b90 con 0x7fc588067b50 2020-11-20T12:10:56.738 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58eede700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc58813ce60 con 0x7fc5881376c0 2020-11-20T12:10:56.738 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58d6db700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc588067b50 0x7fc5881492f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:56.738 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58d6db700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc588067b50 0x7fc5881492f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49944/0 (socket says 172.21.15.189:49944) 2020-11-20T12:10:56.738 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58d6db700 1 -- 172.21.15.189:0/2140809689 learned_addr learned my addr 172.21.15.189:0/2140809689 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:10:56.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58dedc700 1 --2- 172.21.15.189:0/2140809689 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5880612b0 0x7fc58813c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:56.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58e6dd700 1 --2- 172.21.15.189:0/2140809689 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc5881376c0 0x7fc58813d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:56.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58d6db700 1 -- 172.21.15.189:0/2140809689 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc5881376c0 msgr2=0x7fc58813d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:56.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58d6db700 1 --2- 172.21.15.189:0/2140809689 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc5881376c0 0x7fc58813d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:56.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58d6db700 1 -- 172.21.15.189:0/2140809689 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5880612b0 msgr2=0x7fc58813c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:56.740 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58d6db700 1 --2- 172.21.15.189:0/2140809689 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5880612b0 0x7fc58813c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:56.740 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58d6db700 1 -- 172.21.15.189:0/2140809689 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58813cba0 con 0x7fc588067b50 2020-11-20T12:10:56.741 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58dedc700 1 --2- 172.21.15.189:0/2140809689 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5880612b0 0x7fc58813c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:10:56.741 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58d6db700 1 --2- 172.21.15.189:0/2140809689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc588067b50 0x7fc5881492f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fc57800c710 tx=0x7fc578007c20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:56.742 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc575ffb700 1 -- 172.21.15.189:0/2140809689 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc578007020 con 0x7fc588067b50 2020-11-20T12:10:56.742 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc575ffb700 1 -- 172.21.15.189:0/2140809689 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc578007180 con 0x7fc588067b50 2020-11-20T12:10:56.742 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58eede700 1 -- 172.21.15.189:0/2140809689 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc58813c8c0 con 0x7fc588067b50 2020-11-20T12:10:56.742 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc575ffb700 1 -- 172.21.15.189:0/2140809689 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57800e560 con 0x7fc588067b50 2020-11-20T12:10:56.742 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.729+0000 7fc58eede700 1 -- 172.21.15.189:0/2140809689 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc58813d7f0 con 0x7fc588067b50 2020-11-20T12:10:56.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.733+0000 7fc58eede700 1 -- 172.21.15.189:0/2140809689 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc58813d020 con 0x7fc588067b50 2020-11-20T12:10:56.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.733+0000 7fc575ffb700 1 -- 172.21.15.189:0/2140809689 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fc578010070 con 0x7fc588067b50 2020-11-20T12:10:56.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.733+0000 7fc575ffb700 1 --2- 172.21.15.189:0/2140809689 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc5600557c0 0x7fc560057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:56.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.733+0000 7fc575ffb700 1 -- 172.21.15.189:0/2140809689 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6331+0+0 (secure 0 0 0) 0x7fc57806dc50 con 0x7fc588067b50 2020-11-20T12:10:56.744 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.733+0000 7fc58dedc700 1 --2- 172.21.15.189:0/2140809689 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc5600557c0 0x7fc560057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:56.744 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.733+0000 7fc58dedc700 1 --2- 172.21.15.189:0/2140809689 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc5600557c0 0x7fc560057c80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fc58400c830 tx=0x7fc584006f40).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:56.744 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.741+0000 7fc575ffb700 1 -- 172.21.15.189:0/2140809689 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc578042fd0 con 0x7fc588067b50 2020-11-20T12:10:56.960 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.957+0000 7fc58eede700 1 -- 172.21.15.189:0/2140809689 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5880446d0 con 0x7fc588067b50 2020-11-20T12:10:56.960 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.957+0000 7fc575ffb700 1 -- 172.21.15.189:0/2140809689 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fc5780439f0 con 0x7fc588067b50 2020-11-20T12:10:56.961 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:10:56.961 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5053,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/3540904386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3540904386},{"type":"v1","addr":"172.21.15.189:6835","nonce":3540904386}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:09:55.624531+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4676},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4676":{"gid":4676,"name":"b","rank":0,"incarnation":17,"state":"up:active","state_seq":26,"addr":"172.21.15.47:6833/4000022090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4000022090},{"type":"v1","addr":"172.21.15.47:6833","nonce":4000022090}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:10:56.962 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.961+0000 7fc58eede700 1 -- 172.21.15.189:0/2140809689 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc5600557c0 msgr2=0x7fc560057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:56.962 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.961+0000 7fc58eede700 1 --2- 172.21.15.189:0/2140809689 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc5600557c0 0x7fc560057c80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fc58400c830 tx=0x7fc584006f40).stop 2020-11-20T12:10:56.963 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.961+0000 7fc58eede700 1 -- 172.21.15.189:0/2140809689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc588067b50 msgr2=0x7fc5881492f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:56.963 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.961+0000 7fc58eede700 1 --2- 172.21.15.189:0/2140809689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc588067b50 0x7fc5881492f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fc57800c710 tx=0x7fc578007c20).stop 2020-11-20T12:10:56.963 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.961+0000 7fc58eede700 1 -- 172.21.15.189:0/2140809689 shutdown_connections 2020-11-20T12:10:56.963 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.961+0000 7fc58eede700 1 --2- 172.21.15.189:0/2140809689 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc5600557c0 0x7fc560057c80 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:56.963 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.961+0000 7fc58eede700 1 --2- 172.21.15.189:0/2140809689 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc5880612b0 0x7fc58813c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:56.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.961+0000 7fc58eede700 1 --2- 172.21.15.189:0/2140809689 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc5881376c0 0x7fc58813d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:56.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.961+0000 7fc58eede700 1 --2- 172.21.15.189:0/2140809689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc588067b50 0x7fc5881492f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:56.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.961+0000 7fc58eede700 1 -- 172.21.15.189:0/2140809689 >> 172.21.15.189:0/2140809689 conn(0x7fc588063210 msgr2=0x7fc5880a9db0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:56.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.961+0000 7fc58eede700 1 -- 172.21.15.189:0/2140809689 shutdown_connections 2020-11-20T12:10:56.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:56.961+0000 7fc58eede700 1 -- 172.21.15.189:0/2140809689 wait complete. 2020-11-20T12:10:56.965 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 21 2020-11-20T12:10:56.975 INFO:tasks.mds_thrash.fs.[None]:kill mds.b (rank=0) 2020-11-20T12:10:56.976 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:10:56.976 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:10:56.991 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:10:56.992 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:10:57.105 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.101+0000 7f8fb8e95700 1 Processor -- start 2020-11-20T12:10:57.105 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.101+0000 7f8fb8e95700 1 -- start start 2020-11-20T12:10:57.106 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.105+0000 7f8fb8e95700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8fb40ad000 0x7f8fb413e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:57.106 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.105+0000 7f8fb8e95700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fb4067730 con 0x7f8fb413a6e0 2020-11-20T12:10:57.106 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.105+0000 7f8fb8e95700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fb413dd40 con 0x7f8fb40ad3e0 2020-11-20T12:10:57.107 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.105+0000 7f8fb8e95700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fb4138190 con 0x7f8fb40ad000 2020-11-20T12:10:57.107 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.105+0000 7f8fb37fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8fb40ad000 0x7f8fb413e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:57.107 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.105+0000 7f8fb2ffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8fb40ad3e0 0x7f8fb4066d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51576/0 (socket says 172.21.15.189:51576) 2020-11-20T12:10:57.107 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.105+0000 7f8fb37fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8fb40ad000 0x7f8fb413e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49952/0 (socket says 172.21.15.189:49952) 2020-11-20T12:10:57.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.105+0000 7f8fb2ffd700 1 -- 172.21.15.189:0/2282627289 learned_addr learned my addr 172.21.15.189:0/2282627289 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:10:57.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.105+0000 7f8fb37fe700 1 -- 172.21.15.189:0/2282627289 >> v1:172.21.15.189:6789/0 conn(0x7f8fb40ad3e0 legacy=0x7f8fb4066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:57.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.105+0000 7f8fb37fe700 1 -- 172.21.15.189:0/2282627289 >> v1:172.21.15.47:6789/0 conn(0x7f8fb413a6e0 legacy=0x7f8fb413ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:57.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.105+0000 7f8fb37fe700 1 -- 172.21.15.189:0/2282627289 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb413dfe0 con 0x7f8fb40ad000 2020-11-20T12:10:57.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.105+0000 7f8fb37fe700 1 --2- 172.21.15.189:0/2282627289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fb40ad000 0x7f8fb413e480 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f8f9c00b9a0 tx=0x7f8f9c006df0).ready entity=mon.1 client_cookie=815a5d5821e553aa server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:57.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.105+0000 7f8fb27fc700 1 -- 172.21.15.189:0/2282627289 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9c0097b0 con 0x7f8fb40ad000 2020-11-20T12:10:57.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.105+0000 7f8fb27fc700 1 -- 172.21.15.189:0/2282627289 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f9c009910 con 0x7f8fb40ad000 2020-11-20T12:10:57.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.105+0000 7f8fb27fc700 1 -- 172.21.15.189:0/2282627289 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9c009c90 con 0x7f8fb40ad000 2020-11-20T12:10:57.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.105+0000 7f8fb8e95700 1 -- 172.21.15.189:0/2282627289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fb40ad000 msgr2=0x7f8fb413e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:57.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.105+0000 7f8fb8e95700 1 --2- 172.21.15.189:0/2282627289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fb40ad000 0x7f8fb413e480 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f8f9c00b9a0 tx=0x7f8f9c006df0).stop 2020-11-20T12:10:57.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.105+0000 7f8fb8e95700 1 -- 172.21.15.189:0/2282627289 shutdown_connections 2020-11-20T12:10:57.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.105+0000 7f8fb8e95700 1 --2- 172.21.15.189:0/2282627289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fb40ad000 0x7f8fb413e480 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:57.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.105+0000 7f8fb8e95700 1 -- 172.21.15.189:0/2282627289 >> 172.21.15.189:0/2282627289 conn(0x7f8fb4062080 msgr2=0x7f8fb4062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:57.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.105+0000 7f8fb8e95700 1 -- 172.21.15.189:0/2282627289 shutdown_connections 2020-11-20T12:10:57.114 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.105+0000 7f8fb8e95700 1 -- 172.21.15.189:0/2282627289 wait complete. 2020-11-20T12:10:57.114 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.109+0000 7f8fb8e95700 1 Processor -- start 2020-11-20T12:10:57.114 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.109+0000 7f8fb8e95700 1 -- start start 2020-11-20T12:10:57.114 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.109+0000 7f8fb8e95700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8fb40ad000 0x7f8fb40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:57.114 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.109+0000 7f8fb8e95700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fb40ad3e0 0x7f8fb40a9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:57.115 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.109+0000 7f8fb8e95700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fb40aa020 0x7f8fb414d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:57.115 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.109+0000 7f8fb8e95700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb40a86d0 con 0x7f8fb40ad3e0 2020-11-20T12:10:57.115 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.109+0000 7f8fb8e95700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb40a8970 con 0x7f8fb40aa020 2020-11-20T12:10:57.115 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.109+0000 7f8fb8e95700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb40a8c10 con 0x7f8fb40ad000 2020-11-20T12:10:57.115 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.109+0000 7f8fb3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fb40aa020 0x7f8fb414d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:57.116 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.109+0000 7f8fb3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fb40aa020 0x7f8fb414d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49962/0 (socket says 172.21.15.189:49962) 2020-11-20T12:10:57.116 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.109+0000 7f8fb3fff700 1 -- 172.21.15.189:0/4008885911 learned_addr learned my addr 172.21.15.189:0/4008885911 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:10:57.116 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.109+0000 7f8fb37fe700 1 --2- 172.21.15.189:0/4008885911 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8fb40ad000 0x7f8fb40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:57.116 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.109+0000 7f8fb2ffd700 1 --2- 172.21.15.189:0/4008885911 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fb40ad3e0 0x7f8fb40a9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:57.118 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.109+0000 7f8fb3fff700 1 -- 172.21.15.189:0/4008885911 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8fb40ad000 msgr2=0x7f8fb40a4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:57.118 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.109+0000 7f8fb3fff700 1 --2- 172.21.15.189:0/4008885911 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8fb40ad000 0x7f8fb40a4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:57.118 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.109+0000 7f8fb3fff700 1 -- 172.21.15.189:0/4008885911 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fb40ad3e0 msgr2=0x7f8fb40a9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:57.119 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.109+0000 7f8fb3fff700 1 --2- 172.21.15.189:0/4008885911 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fb40ad3e0 0x7f8fb40a9b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:57.119 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.109+0000 7f8fb3fff700 1 -- 172.21.15.189:0/4008885911 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb40b08e0 con 0x7f8fb40aa020 2020-11-20T12:10:57.119 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.109+0000 7f8fb2ffd700 1 --2- 172.21.15.189:0/4008885911 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fb40ad3e0 0x7f8fb40a9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:10:57.120 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.109+0000 7f8fb3fff700 1 --2- 172.21.15.189:0/4008885911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fb40aa020 0x7f8fb414d0a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f8fac00ca90 tx=0x7f8fac007f70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:57.120 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.109+0000 7f8fab7fe700 1 -- 172.21.15.189:0/4008885911 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fac00a730 con 0x7f8fb40aa020 2020-11-20T12:10:57.120 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.109+0000 7f8fab7fe700 1 -- 172.21.15.189:0/4008885911 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fac00a890 con 0x7f8fb40aa020 2020-11-20T12:10:57.120 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.109+0000 7f8fb8e95700 1 -- 172.21.15.189:0/4008885911 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fb40a5320 con 0x7f8fb40aa020 2020-11-20T12:10:57.121 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.109+0000 7f8fab7fe700 1 -- 172.21.15.189:0/4008885911 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fac00ac10 con 0x7f8fb40aa020 2020-11-20T12:10:57.121 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.109+0000 7f8fb8e95700 1 -- 172.21.15.189:0/4008885911 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fb40a92d0 con 0x7f8fb40aa020 2020-11-20T12:10:57.121 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.113+0000 7f8fb8e95700 1 -- 172.21.15.189:0/4008885911 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fb40a8dd0 con 0x7f8fb40aa020 2020-11-20T12:10:57.121 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.113+0000 7f8fab7fe700 1 -- 172.21.15.189:0/4008885911 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f8fac019030 con 0x7f8fb40aa020 2020-11-20T12:10:57.121 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.113+0000 7f8fab7fe700 1 --2- 172.21.15.189:0/4008885911 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f8f900557c0 0x7f8f90057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:10:57.122 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.113+0000 7f8fab7fe700 1 -- 172.21.15.189:0/4008885911 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6331+0+0 (secure 0 0 0) 0x7f8fac06e720 con 0x7f8fb40aa020 2020-11-20T12:10:57.122 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.113+0000 7f8fb37fe700 1 --2- 172.21.15.189:0/4008885911 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f8f900557c0 0x7f8f90057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:10:57.122 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.113+0000 7f8fb37fe700 1 --2- 172.21.15.189:0/4008885911 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f8f900557c0 0x7f8f90057c80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f8f9c006e30 tx=0x7f8f9c015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:10:57.126 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.117+0000 7f8fab7fe700 1 -- 172.21.15.189:0/4008885911 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8fac0445f0 con 0x7f8fb40aa020 2020-11-20T12:10:57.358 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.349+0000 7f8fb8e95700 1 -- 172.21.15.189:0/4008885911 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fb40b1370 con 0x7f8fb40aa020 2020-11-20T12:10:57.918 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.349+0000 7f8fab7fe700 1 -- 172.21.15.189:0/4008885911 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f8fac021070 con 0x7f8fb40aa020 2020-11-20T12:10:57.919 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.349+0000 7f8fb8e95700 1 -- 172.21.15.189:0/4008885911 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f8f900557c0 msgr2=0x7f8f90057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:57.919 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.349+0000 7f8fb8e95700 1 --2- 172.21.15.189:0/4008885911 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f8f900557c0 0x7f8f90057c80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f8f9c006e30 tx=0x7f8f9c015000).stop 2020-11-20T12:10:57.919 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.349+0000 7f8fb8e95700 1 -- 172.21.15.189:0/4008885911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fb40aa020 msgr2=0x7f8fb414d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:10:57.920 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.349+0000 7f8fb8e95700 1 --2- 172.21.15.189:0/4008885911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fb40aa020 0x7f8fb414d0a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f8fac00ca90 tx=0x7f8fac007f70).stop 2020-11-20T12:10:57.920 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.349+0000 7f8fb8e95700 1 -- 172.21.15.189:0/4008885911 shutdown_connections 2020-11-20T12:10:57.920 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.349+0000 7f8fb8e95700 1 --2- 172.21.15.189:0/4008885911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fb40aa020 0x7f8fb414d0a0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:57.920 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.349+0000 7f8fb8e95700 1 --2- 172.21.15.189:0/4008885911 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8fb40ad000 0x7f8fb40a4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:57.920 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.349+0000 7f8fb8e95700 1 --2- 172.21.15.189:0/4008885911 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f8f900557c0 0x7f8f90057c80 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:57.921 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.349+0000 7f8fb8e95700 1 --2- 172.21.15.189:0/4008885911 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8fb40ad3e0 0x7f8fb40a9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:10:57.921 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.349+0000 7f8fb8e95700 1 -- 172.21.15.189:0/4008885911 >> 172.21.15.189:0/4008885911 conn(0x7f8fb4062080 msgr2=0x7f8fb4139a90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:10:57.921 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.349+0000 7f8fb8e95700 1 -- 172.21.15.189:0/4008885911 shutdown_connections 2020-11-20T12:10:57.921 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:10:57.349+0000 7f8fb8e95700 1 -- 172.21.15.189:0/4008885911 wait complete. 2020-11-20T12:10:57.921 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 21 2020-11-20T12:10:57.922 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:10:57.922 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5053,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/3540904386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3540904386},{"type":"v1","addr":"172.21.15.189:6835","nonce":3540904386}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:09:55.624531+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4676},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4676":{"gid":4676,"name":"b","rank":0,"incarnation":17,"state":"up:active","state_seq":26,"addr":"172.21.15.47:6833/4000022090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4000022090},{"type":"v1","addr":"172.21.15.47:6833","nonce":4000022090}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:10:57.923 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:10:59.925 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:00.070 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.069+0000 7f6d8db15700 1 Processor -- start 2020-11-20T12:11:00.071 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.069+0000 7f6d8db15700 1 -- start start 2020-11-20T12:11:00.071 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.069+0000 7f6d8db15700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6d880ad000 0x7f6d8813e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:00.072 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.069+0000 7f6d8db15700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6d8813a6e0 0x7f6d8813ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:00.072 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.069+0000 7f6d8db15700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d88067730 con 0x7f6d880ad3e0 2020-11-20T12:11:00.072 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.069+0000 7f6d8db15700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d88138190 con 0x7f6d8813a6e0 2020-11-20T12:11:00.072 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.069+0000 7f6d8db15700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8813dd40 con 0x7f6d880ad000 2020-11-20T12:11:00.077 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.069+0000 7f6d8cb13700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6d880ad000 0x7f6d8813e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:00.078 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.069+0000 7f6d8d314700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6d8813a6e0 0x7f6d8813ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:00.078 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.069+0000 7f6d8cb13700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6d880ad000 0x7f6d8813e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.189:43012/0 (socket says 172.21.15.189:43012) 2020-11-20T12:11:00.078 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.069+0000 7f6d8cb13700 1 -- 172.21.15.189:0/3908305631 learned_addr learned my addr 172.21.15.189:0/3908305631 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:11:00.078 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.069+0000 7f6d8cb13700 1 -- 172.21.15.189:0/3908305631 >> v2:172.21.15.47:3300/0 conn(0x7f6d8813a6e0 msgr2=0x7f6d8813ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:00.079 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.069+0000 7f6d8cb13700 1 --2- 172.21.15.189:0/3908305631 >> v2:172.21.15.47:3300/0 conn(0x7f6d8813a6e0 0x7f6d8813ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:00.079 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.069+0000 7f6d8cb13700 1 -- 172.21.15.189:0/3908305631 >> v1:172.21.15.47:6789/0 conn(0x7f6d880ad3e0 legacy=0x7f6d88066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:00.079 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.069+0000 7f6d8cb13700 1 -- 172.21.15.189:0/3908305631 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8813dfe0 con 0x7f6d880ad000 2020-11-20T12:11:00.079 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.069+0000 7f6d8cb13700 1 --2- 172.21.15.189:0/3908305631 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6d880ad000 0x7f6d8813e480 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f6d80006fd0 tx=0x7f6d80006eb0).ready entity=mon.2 client_cookie=31541f0e4e361b97 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:00.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.069+0000 7f6d877fe700 1 -- 172.21.15.189:0/3908305631 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d80009810 con 0x7f6d880ad000 2020-11-20T12:11:00.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.069+0000 7f6d877fe700 1 -- 172.21.15.189:0/3908305631 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d80009970 con 0x7f6d880ad000 2020-11-20T12:11:00.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.069+0000 7f6d877fe700 1 -- 172.21.15.189:0/3908305631 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d80009cf0 con 0x7f6d880ad000 2020-11-20T12:11:00.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.069+0000 7f6d8db15700 1 -- 172.21.15.189:0/3908305631 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6d880ad000 msgr2=0x7f6d8813e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:00.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.069+0000 7f6d8db15700 1 --2- 172.21.15.189:0/3908305631 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6d880ad000 0x7f6d8813e480 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f6d80006fd0 tx=0x7f6d80006eb0).stop 2020-11-20T12:11:00.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.069+0000 7f6d8db15700 1 -- 172.21.15.189:0/3908305631 shutdown_connections 2020-11-20T12:11:00.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.069+0000 7f6d8db15700 1 --2- 172.21.15.189:0/3908305631 >> v2:172.21.15.47:3300/0 conn(0x7f6d8813a6e0 0x7f6d8813ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:00.082 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.069+0000 7f6d8db15700 1 --2- 172.21.15.189:0/3908305631 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6d880ad000 0x7f6d8813e480 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:00.082 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.069+0000 7f6d8db15700 1 -- 172.21.15.189:0/3908305631 >> 172.21.15.189:0/3908305631 conn(0x7f6d88062080 msgr2=0x7f6d88062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:00.082 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d8db15700 1 -- 172.21.15.189:0/3908305631 shutdown_connections 2020-11-20T12:11:00.082 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d8db15700 1 -- 172.21.15.189:0/3908305631 wait complete. 2020-11-20T12:11:00.083 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d8db15700 1 Processor -- start 2020-11-20T12:11:00.083 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d8db15700 1 -- start start 2020-11-20T12:11:00.083 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d8db15700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6d880ad000 0x7f6d880aacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:00.083 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d8db15700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d880ad3e0 0x7f6d880a7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:00.084 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d8db15700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d88149a70 0x7f6d880a5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:00.084 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d8db15700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d881474f0 con 0x7f6d880ad3e0 2020-11-20T12:11:00.084 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d8db15700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d880a5420 con 0x7f6d88149a70 2020-11-20T12:11:00.084 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d8db15700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d880a56c0 con 0x7f6d880ad000 2020-11-20T12:11:00.085 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d8d314700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d88149a70 0x7f6d880a5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:00.085 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d8d314700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d88149a70 0x7f6d880a5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49984/0 (socket says 172.21.15.189:49984) 2020-11-20T12:11:00.085 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d8d314700 1 -- 172.21.15.189:0/1563705110 learned_addr learned my addr 172.21.15.189:0/1563705110 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:11:00.085 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d8cb13700 1 --2- 172.21.15.189:0/1563705110 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6d880ad000 0x7f6d880aacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:00.087 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d87fff700 1 --2- 172.21.15.189:0/1563705110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d880ad3e0 0x7f6d880a7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:00.087 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d8d314700 1 -- 172.21.15.189:0/1563705110 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6d880ad000 msgr2=0x7f6d880aacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:00.087 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d8d314700 1 --2- 172.21.15.189:0/1563705110 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6d880ad000 0x7f6d880aacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:00.088 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d8d314700 1 -- 172.21.15.189:0/1563705110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d880ad3e0 msgr2=0x7f6d880a7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:00.089 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d8d314700 1 --2- 172.21.15.189:0/1563705110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d880ad3e0 0x7f6d880a7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:00.090 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d8d314700 1 -- 172.21.15.189:0/1563705110 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d880a7a10 con 0x7f6d88149a70 2020-11-20T12:11:00.090 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d8d314700 1 --2- 172.21.15.189:0/1563705110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d88149a70 0x7f6d880a5b60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f6d7000e970 tx=0x7f6d70006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:00.090 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d84ff9700 1 -- 172.21.15.189:0/1563705110 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7000b6a0 con 0x7f6d88149a70 2020-11-20T12:11:00.090 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d84ff9700 1 -- 172.21.15.189:0/1563705110 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d7000b800 con 0x7f6d88149a70 2020-11-20T12:11:00.090 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d8db15700 1 -- 172.21.15.189:0/1563705110 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d880a7d10 con 0x7f6d88149a70 2020-11-20T12:11:00.091 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d84ff9700 1 -- 172.21.15.189:0/1563705110 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7000bba0 con 0x7f6d88149a70 2020-11-20T12:11:00.091 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d8db15700 1 -- 172.21.15.189:0/1563705110 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d880a9f90 con 0x7f6d88149a70 2020-11-20T12:11:00.091 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.073+0000 7f6d8db15700 1 -- 172.21.15.189:0/1563705110 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d880a5880 con 0x7f6d88149a70 2020-11-20T12:11:00.091 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.077+0000 7f6d84ff9700 1 -- 172.21.15.189:0/1563705110 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f6d7001c460 con 0x7f6d88149a70 2020-11-20T12:11:00.092 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.077+0000 7f6d84ff9700 1 --2- 172.21.15.189:0/1563705110 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6d600557c0 0x7f6d60057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:00.093 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.077+0000 7f6d8cb13700 1 --2- 172.21.15.189:0/1563705110 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6d600557c0 0x7f6d60057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:00.093 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.077+0000 7f6d84ff9700 1 -- 172.21.15.189:0/1563705110 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6331+0+0 (secure 0 0 0) 0x7f6d7006ea70 con 0x7f6d88149a70 2020-11-20T12:11:00.093 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.077+0000 7f6d8cb13700 1 --2- 172.21.15.189:0/1563705110 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6d600557c0 0x7f6d60057c80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f6d8813b680 tx=0x7f6d80015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:00.093 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.081+0000 7f6d84ff9700 1 -- 172.21.15.189:0/1563705110 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6d700446a0 con 0x7f6d88149a70 2020-11-20T12:11:00.283 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.281+0000 7f6d8db15700 1 -- 172.21.15.189:0/1563705110 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d880446d0 con 0x7f6d88149a70 2020-11-20T12:11:00.284 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.281+0000 7f6d84ff9700 1 -- 172.21.15.189:0/1563705110 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f6d70047cc0 con 0x7f6d88149a70 2020-11-20T12:11:00.284 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:11:00.284 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5053,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/3540904386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3540904386},{"type":"v1","addr":"172.21.15.189:6835","nonce":3540904386}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:09:55.624531+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4676},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4676":{"gid":4676,"name":"b","rank":0,"incarnation":17,"state":"up:active","state_seq":26,"addr":"172.21.15.47:6833/4000022090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4000022090},{"type":"v1","addr":"172.21.15.47:6833","nonce":4000022090}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:11:00.286 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.285+0000 7f6d8db15700 1 -- 172.21.15.189:0/1563705110 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6d600557c0 msgr2=0x7f6d60057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:00.286 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.285+0000 7f6d8db15700 1 --2- 172.21.15.189:0/1563705110 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6d600557c0 0x7f6d60057c80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f6d8813b680 tx=0x7f6d80015000).stop 2020-11-20T12:11:00.287 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.285+0000 7f6d8db15700 1 -- 172.21.15.189:0/1563705110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d88149a70 msgr2=0x7f6d880a5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:00.287 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.285+0000 7f6d8db15700 1 --2- 172.21.15.189:0/1563705110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d88149a70 0x7f6d880a5b60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f6d7000e970 tx=0x7f6d70006df0).stop 2020-11-20T12:11:00.287 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.285+0000 7f6d8db15700 1 -- 172.21.15.189:0/1563705110 shutdown_connections 2020-11-20T12:11:00.287 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.285+0000 7f6d8db15700 1 --2- 172.21.15.189:0/1563705110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d88149a70 0x7f6d880a5b60 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:00.288 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.285+0000 7f6d8db15700 1 --2- 172.21.15.189:0/1563705110 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6d880ad000 0x7f6d880aacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:00.288 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.285+0000 7f6d8db15700 1 --2- 172.21.15.189:0/1563705110 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6d600557c0 0x7f6d60057c80 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:00.288 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.285+0000 7f6d8db15700 1 --2- 172.21.15.189:0/1563705110 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d880ad3e0 0x7f6d880a7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:00.288 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.285+0000 7f6d8db15700 1 -- 172.21.15.189:0/1563705110 >> 172.21.15.189:0/1563705110 conn(0x7f6d88062080 msgr2=0x7f6d88136570 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:00.288 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.285+0000 7f6d8db15700 1 -- 172.21.15.189:0/1563705110 shutdown_connections 2020-11-20T12:11:00.289 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:00.285+0000 7f6d8db15700 1 -- 172.21.15.189:0/1563705110 wait complete. 2020-11-20T12:11:00.289 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 21 2020-11-20T12:11:00.299 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:11:02.300 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:02.441 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b9759e700 1 Processor -- start 2020-11-20T12:11:02.441 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b9759e700 1 -- start start 2020-11-20T12:11:02.441 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b9759e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9b98066ed0 0x7f9b9813b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:02.442 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b9759e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9b9813ba60 0x7f9b9813bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:02.442 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b9759e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b9813c3c0 con 0x7f9b980a8840 2020-11-20T12:11:02.442 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b9759e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b98066800 con 0x7f9b98066ed0 2020-11-20T12:11:02.442 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b9759e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b98066aa0 con 0x7f9b9813ba60 2020-11-20T12:11:02.442 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b95d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9b98066ed0 0x7f9b9813b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:02.443 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b95d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9b98066ed0 0x7f9b9813b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49990/0 (socket says 172.21.15.189:49990) 2020-11-20T12:11:02.443 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b95d9b700 1 -- 172.21.15.189:0/1599513812 learned_addr learned my addr 172.21.15.189:0/1599513812 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:11:02.443 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b96d9d700 1 --2- 172.21.15.189:0/1599513812 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9b9813ba60 0x7f9b9813bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:02.443 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b95d9b700 1 -- 172.21.15.189:0/1599513812 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9b9813ba60 msgr2=0x7f9b9813bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:02.443 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b95d9b700 1 --2- 172.21.15.189:0/1599513812 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9b9813ba60 0x7f9b9813bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:02.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b95d9b700 1 -- 172.21.15.189:0/1599513812 >> v1:172.21.15.47:6789/0 conn(0x7f9b980a8840 legacy=0x7f9b9813eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:02.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b95d9b700 1 -- 172.21.15.189:0/1599513812 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b98067450 con 0x7f9b98066ed0 2020-11-20T12:11:02.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b96d9d700 1 --2- 172.21.15.189:0/1599513812 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9b9813ba60 0x7f9b9813bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:11:02.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b95d9b700 1 --2- 172.21.15.189:0/1599513812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b98066ed0 0x7f9b9813b570 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f9b80006fd0 tx=0x7f9b80006eb0).ready entity=mon.1 client_cookie=8f505c870c963292 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:02.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b9559a700 1 -- 172.21.15.189:0/1599513812 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b80009960 con 0x7f9b98066ed0 2020-11-20T12:11:02.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b9559a700 1 -- 172.21.15.189:0/1599513812 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b80009ac0 con 0x7f9b98066ed0 2020-11-20T12:11:02.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b9559a700 1 -- 172.21.15.189:0/1599513812 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b80009e40 con 0x7f9b98066ed0 2020-11-20T12:11:02.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b9759e700 1 -- 172.21.15.189:0/1599513812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b98066ed0 msgr2=0x7f9b9813b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:02.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b9759e700 1 --2- 172.21.15.189:0/1599513812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b98066ed0 0x7f9b9813b570 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f9b80006fd0 tx=0x7f9b80006eb0).stop 2020-11-20T12:11:02.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b9759e700 1 -- 172.21.15.189:0/1599513812 shutdown_connections 2020-11-20T12:11:02.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b9759e700 1 --2- 172.21.15.189:0/1599513812 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9b9813ba60 0x7f9b9813bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:02.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b9759e700 1 --2- 172.21.15.189:0/1599513812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b98066ed0 0x7f9b9813b570 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:02.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b9759e700 1 -- 172.21.15.189:0/1599513812 >> 172.21.15.189:0/1599513812 conn(0x7f9b98060170 msgr2=0x7f9b98060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:02.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b9759e700 1 -- 172.21.15.189:0/1599513812 shutdown_connections 2020-11-20T12:11:02.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.433+0000 7f9b9759e700 1 -- 172.21.15.189:0/1599513812 wait complete. 2020-11-20T12:11:02.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.437+0000 7f9b9759e700 1 Processor -- start 2020-11-20T12:11:02.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.437+0000 7f9b9759e700 1 -- start start 2020-11-20T12:11:02.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.437+0000 7f9b9759e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b98066ed0 0x7f9b980a28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:02.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.437+0000 7f9b9759e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b980a8840 0x7f9b9809fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:02.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.437+0000 7f9b9759e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9b9813ba60 0x7f9b980a0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:02.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.437+0000 7f9b9759e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b98151f70 con 0x7f9b980a8840 2020-11-20T12:11:02.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.437+0000 7f9b9759e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b98152210 con 0x7f9b98066ed0 2020-11-20T12:11:02.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.437+0000 7f9b9759e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b981524b0 con 0x7f9b9813ba60 2020-11-20T12:11:02.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.437+0000 7f9b9659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b98066ed0 0x7f9b980a28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:02.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.437+0000 7f9b9659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b98066ed0 0x7f9b980a28c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49994/0 (socket says 172.21.15.189:49994) 2020-11-20T12:11:02.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.437+0000 7f9b9659c700 1 -- 172.21.15.189:0/3772452507 learned_addr learned my addr 172.21.15.189:0/3772452507 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:11:02.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.437+0000 7f9b96d9d700 1 --2- 172.21.15.189:0/3772452507 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9b9813ba60 0x7f9b980a0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:02.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.437+0000 7f9b95d9b700 1 --2- 172.21.15.189:0/3772452507 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b980a8840 0x7f9b9809fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:02.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.437+0000 7f9b9659c700 1 -- 172.21.15.189:0/3772452507 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9b9813ba60 msgr2=0x7f9b980a0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:02.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.437+0000 7f9b9659c700 1 --2- 172.21.15.189:0/3772452507 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9b9813ba60 0x7f9b980a0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:02.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.437+0000 7f9b9659c700 1 -- 172.21.15.189:0/3772452507 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b980a8840 msgr2=0x7f9b9809fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:02.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.437+0000 7f9b9659c700 1 --2- 172.21.15.189:0/3772452507 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b980a8840 0x7f9b9809fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:02.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.437+0000 7f9b9659c700 1 -- 172.21.15.189:0/3772452507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b980a2db0 con 0x7f9b98066ed0 2020-11-20T12:11:02.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.437+0000 7f9b9659c700 1 --2- 172.21.15.189:0/3772452507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b98066ed0 0x7f9b980a28c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f9b8800ca60 tx=0x7f9b88001460).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:02.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.437+0000 7f9b8e7fc700 1 -- 172.21.15.189:0/3772452507 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8800a8c0 con 0x7f9b98066ed0 2020-11-20T12:11:02.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.437+0000 7f9b9759e700 1 -- 172.21.15.189:0/3772452507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b9809c4e0 con 0x7f9b98066ed0 2020-11-20T12:11:02.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.437+0000 7f9b8e7fc700 1 -- 172.21.15.189:0/3772452507 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8800aa20 con 0x7f9b98066ed0 2020-11-20T12:11:02.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.437+0000 7f9b8e7fc700 1 -- 172.21.15.189:0/3772452507 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8800ada0 con 0x7f9b98066ed0 2020-11-20T12:11:02.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.437+0000 7f9b9759e700 1 -- 172.21.15.189:0/3772452507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b980a3640 con 0x7f9b98066ed0 2020-11-20T12:11:02.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.437+0000 7f9b9759e700 1 -- 172.21.15.189:0/3772452507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b980446d0 con 0x7f9b98066ed0 2020-11-20T12:11:02.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.441+0000 7f9b8e7fc700 1 -- 172.21.15.189:0/3772452507 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f9b88018020 con 0x7f9b98066ed0 2020-11-20T12:11:02.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.441+0000 7f9b8e7fc700 1 --2- 172.21.15.189:0/3772452507 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9b6c0557c0 0x7f9b6c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:02.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.441+0000 7f9b8e7fc700 1 -- 172.21.15.189:0/3772452507 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6331+0+0 (secure 0 0 0) 0x7f9b8806f100 con 0x7f9b98066ed0 2020-11-20T12:11:02.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.441+0000 7f9b95d9b700 1 --2- 172.21.15.189:0/3772452507 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9b6c0557c0 0x7f9b6c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:02.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.441+0000 7f9b95d9b700 1 --2- 172.21.15.189:0/3772452507 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9b6c0557c0 0x7f9b6c057c80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f9b980a6b40 tx=0x7f9b80015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:02.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.445+0000 7f9b8e7fc700 1 -- 172.21.15.189:0/3772452507 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9b88044ea0 con 0x7f9b98066ed0 2020-11-20T12:11:02.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.641+0000 7f9b9759e700 1 -- 172.21.15.189:0/3772452507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b9805cdd0 con 0x7f9b98066ed0 2020-11-20T12:11:02.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.641+0000 7f9b8e7fc700 1 -- 172.21.15.189:0/3772452507 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f9b88016070 con 0x7f9b98066ed0 2020-11-20T12:11:02.645 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:11:02.645 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5053,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/3540904386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3540904386},{"type":"v1","addr":"172.21.15.189:6835","nonce":3540904386}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:09:55.624531+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4676},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4676":{"gid":4676,"name":"b","rank":0,"incarnation":17,"state":"up:active","state_seq":26,"addr":"172.21.15.47:6833/4000022090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4000022090},{"type":"v1","addr":"172.21.15.47:6833","nonce":4000022090}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:11:02.646 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.645+0000 7f9b9759e700 1 -- 172.21.15.189:0/3772452507 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9b6c0557c0 msgr2=0x7f9b6c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:02.646 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.645+0000 7f9b9759e700 1 --2- 172.21.15.189:0/3772452507 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9b6c0557c0 0x7f9b6c057c80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f9b980a6b40 tx=0x7f9b80015000).stop 2020-11-20T12:11:02.646 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.645+0000 7f9b9759e700 1 -- 172.21.15.189:0/3772452507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b98066ed0 msgr2=0x7f9b980a28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:02.647 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.645+0000 7f9b9759e700 1 --2- 172.21.15.189:0/3772452507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b98066ed0 0x7f9b980a28c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f9b8800ca60 tx=0x7f9b88001460).stop 2020-11-20T12:11:02.647 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.645+0000 7f9b9759e700 1 -- 172.21.15.189:0/3772452507 shutdown_connections 2020-11-20T12:11:02.647 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.645+0000 7f9b9759e700 1 --2- 172.21.15.189:0/3772452507 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9b9813ba60 0x7f9b980a0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:02.647 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.645+0000 7f9b9759e700 1 --2- 172.21.15.189:0/3772452507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b98066ed0 0x7f9b980a28c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:02.647 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.645+0000 7f9b9759e700 1 --2- 172.21.15.189:0/3772452507 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9b6c0557c0 0x7f9b6c057c80 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:02.648 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.645+0000 7f9b9759e700 1 --2- 172.21.15.189:0/3772452507 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9b980a8840 0x7f9b9809fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:02.648 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.645+0000 7f9b9759e700 1 -- 172.21.15.189:0/3772452507 >> 172.21.15.189:0/3772452507 conn(0x7f9b98060170 msgr2=0x7f9b980aa850 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:02.648 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.645+0000 7f9b9759e700 1 -- 172.21.15.189:0/3772452507 shutdown_connections 2020-11-20T12:11:02.648 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:02.645+0000 7f9b9759e700 1 -- 172.21.15.189:0/3772452507 wait complete. 2020-11-20T12:11:02.649 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 21 2020-11-20T12:11:02.659 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:11:04.660 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:04.791 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.789+0000 7f0be759e700 1 Processor -- start 2020-11-20T12:11:04.792 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.789+0000 7f0be759e700 1 -- start start 2020-11-20T12:11:04.792 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.789+0000 7f0be759e700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0be813a680 0x7f0be813ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:04.793 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.789+0000 7f0be759e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0be813b000 0x7f0be81451c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:04.793 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.789+0000 7f0be759e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be813e320 con 0x7f0be80a8840 2020-11-20T12:11:04.793 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.789+0000 7f0be759e700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0be8066800 con 0x7f0be813a680 2020-11-20T12:11:04.793 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.789+0000 7f0be759e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0be8066aa0 con 0x7f0be813b000 2020-11-20T12:11:04.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.789+0000 7f0be6d9d700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0be813b000 0x7f0be81451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:04.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.789+0000 7f0be659c700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f0be80a8840 0x7f0be813dc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.189:46290/0 (socket says 172.21.15.189:46290) 2020-11-20T12:11:04.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.789+0000 7f0be659c700 1 -- 172.21.15.189:0/1371094230 learned_addr learned my addr 172.21.15.189:0/1371094230 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:11:04.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.789+0000 7f0be5d9b700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0be813a680 0x7f0be813ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:04.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.789+0000 7f0be559a700 1 -- 172.21.15.189:0/1371094230 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020025377 0 0) 0x7f0be813e320 con 0x7f0be80a8840 2020-11-20T12:11:04.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.789+0000 7f0be559a700 1 -- 172.21.15.189:0/1371094230 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bc8003240 con 0x7f0be80a8840 2020-11-20T12:11:04.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.793+0000 7f0be5d9b700 1 -- 172.21.15.189:0/1371094230 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0be813b000 msgr2=0x7f0be81451c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:04.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.793+0000 7f0be5d9b700 1 --2- 172.21.15.189:0/1371094230 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0be813b000 0x7f0be81451c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:04.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.793+0000 7f0be5d9b700 1 -- 172.21.15.189:0/1371094230 >> v1:172.21.15.47:6789/0 conn(0x7f0be80a8840 legacy=0x7f0be813dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:04.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.793+0000 7f0be5d9b700 1 -- 172.21.15.189:0/1371094230 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be80672f0 con 0x7f0be813a680 2020-11-20T12:11:04.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.793+0000 7f0be6d9d700 1 --2- 172.21.15.189:0/1371094230 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0be813b000 0x7f0be81451c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:11:04.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.793+0000 7f0be5d9b700 1 --2- 172.21.15.189:0/1371094230 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0be813a680 0x7f0be813ab10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f0bd000b9a0 tx=0x7f0bd0006df0).ready entity=mon.0 client_cookie=9b245be2f38a234c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:04.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.793+0000 7f0be559a700 1 -- 172.21.15.189:0/1371094230 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd0009680 con 0x7f0be813a680 2020-11-20T12:11:04.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.793+0000 7f0be559a700 1 -- 172.21.15.189:0/1371094230 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bd00097e0 con 0x7f0be813a680 2020-11-20T12:11:04.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.793+0000 7f0be559a700 1 -- 172.21.15.189:0/1371094230 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd0009b60 con 0x7f0be813a680 2020-11-20T12:11:04.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.793+0000 7f0be759e700 1 -- 172.21.15.189:0/1371094230 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0be813a680 msgr2=0x7f0be813ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:04.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.793+0000 7f0be759e700 1 --2- 172.21.15.189:0/1371094230 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0be813a680 0x7f0be813ab10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f0bd000b9a0 tx=0x7f0bd0006df0).stop 2020-11-20T12:11:04.801 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.793+0000 7f0be759e700 1 -- 172.21.15.189:0/1371094230 shutdown_connections 2020-11-20T12:11:04.801 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.793+0000 7f0be759e700 1 --2- 172.21.15.189:0/1371094230 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0be813b000 0x7f0be81451c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:04.801 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.793+0000 7f0be759e700 1 --2- 172.21.15.189:0/1371094230 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0be813a680 0x7f0be813ab10 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:04.801 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.793+0000 7f0be759e700 1 -- 172.21.15.189:0/1371094230 >> 172.21.15.189:0/1371094230 conn(0x7f0be8060170 msgr2=0x7f0be8060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:04.801 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.793+0000 7f0be759e700 1 -- 172.21.15.189:0/1371094230 shutdown_connections 2020-11-20T12:11:04.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.793+0000 7f0be759e700 1 -- 172.21.15.189:0/1371094230 wait complete. 2020-11-20T12:11:04.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.793+0000 7f0be759e700 1 Processor -- start 2020-11-20T12:11:04.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.793+0000 7f0be759e700 1 -- start start 2020-11-20T12:11:04.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.793+0000 7f0be759e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0be80a8840 0x7f0be813bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:04.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.793+0000 7f0be759e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0be813a680 0x7f0be809c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:04.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.793+0000 7f0be759e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0be813b000 0x7f0be81366e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:04.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.793+0000 7f0be759e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be8138370 con 0x7f0be80a8840 2020-11-20T12:11:04.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.793+0000 7f0be759e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be8149880 con 0x7f0be813b000 2020-11-20T12:11:04.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.793+0000 7f0be759e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0be813c580 con 0x7f0be813a680 2020-11-20T12:11:04.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.797+0000 7f0be6d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0be813b000 0x7f0be81366e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:04.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.797+0000 7f0be6d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0be813b000 0x7f0be81366e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50012/0 (socket says 172.21.15.189:50012) 2020-11-20T12:11:04.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.797+0000 7f0be6d9d700 1 -- 172.21.15.189:0/4048584487 learned_addr learned my addr 172.21.15.189:0/4048584487 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:11:04.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.797+0000 7f0be659c700 1 --2- 172.21.15.189:0/4048584487 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0be80a8840 0x7f0be813bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:04.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.797+0000 7f0be5d9b700 1 --2- 172.21.15.189:0/4048584487 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0be813a680 0x7f0be809c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:04.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.797+0000 7f0be6d9d700 1 -- 172.21.15.189:0/4048584487 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0be813a680 msgr2=0x7f0be809c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:04.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.797+0000 7f0be6d9d700 1 --2- 172.21.15.189:0/4048584487 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0be813a680 0x7f0be809c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:04.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.797+0000 7f0be6d9d700 1 -- 172.21.15.189:0/4048584487 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0be80a8840 msgr2=0x7f0be813bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:04.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.797+0000 7f0be6d9d700 1 --2- 172.21.15.189:0/4048584487 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0be80a8840 0x7f0be813bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:04.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.797+0000 7f0be6d9d700 1 -- 172.21.15.189:0/4048584487 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be8136240 con 0x7f0be813b000 2020-11-20T12:11:04.806 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.797+0000 7f0be6d9d700 1 --2- 172.21.15.189:0/4048584487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0be813b000 0x7f0be81366e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f0be000ef50 tx=0x7f0be00091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:04.807 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.797+0000 7f0bde7fc700 1 -- 172.21.15.189:0/4048584487 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be000cc10 con 0x7f0be813b000 2020-11-20T12:11:04.807 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.797+0000 7f0bde7fc700 1 -- 172.21.15.189:0/4048584487 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be000cd70 con 0x7f0be813b000 2020-11-20T12:11:04.807 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.797+0000 7f0be759e700 1 -- 172.21.15.189:0/4048584487 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0be8135570 con 0x7f0be813b000 2020-11-20T12:11:04.808 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.797+0000 7f0bde7fc700 1 -- 172.21.15.189:0/4048584487 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be0012620 con 0x7f0be813b000 2020-11-20T12:11:04.808 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.797+0000 7f0be759e700 1 -- 172.21.15.189:0/4048584487 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0be8136d70 con 0x7f0be813b000 2020-11-20T12:11:04.808 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.797+0000 7f0be759e700 1 -- 172.21.15.189:0/4048584487 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0be813c3f0 con 0x7f0be813b000 2020-11-20T12:11:04.808 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.797+0000 7f0bde7fc700 1 -- 172.21.15.189:0/4048584487 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f0be0022070 con 0x7f0be813b000 2020-11-20T12:11:04.808 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.797+0000 7f0bde7fc700 1 --2- 172.21.15.189:0/4048584487 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0bc40557c0 0x7f0bc4057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:04.809 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.801+0000 7f0bde7fc700 1 -- 172.21.15.189:0/4048584487 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6331+0+0 (secure 0 0 0) 0x7f0be006f080 con 0x7f0be813b000 2020-11-20T12:11:04.809 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.801+0000 7f0be659c700 1 --2- 172.21.15.189:0/4048584487 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0bc40557c0 0x7f0bc4057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:04.809 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.801+0000 7f0be659c700 1 --2- 172.21.15.189:0/4048584487 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0bc40557c0 0x7f0bc4057c80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f0bd80055a0 tx=0x7f0bd8009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:04.810 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:04.805+0000 7f0bde7fc700 1 -- 172.21.15.189:0/4048584487 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0be0044e20 con 0x7f0be813b000 2020-11-20T12:11:05.011 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:05.009+0000 7f0be759e700 1 -- 172.21.15.189:0/4048584487 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0be81363d0 con 0x7f0be813b000 2020-11-20T12:11:05.012 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:05.009+0000 7f0bde7fc700 1 -- 172.21.15.189:0/4048584487 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f0be0016070 con 0x7f0be813b000 2020-11-20T12:11:05.012 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:11:05.012 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5053,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/3540904386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3540904386},{"type":"v1","addr":"172.21.15.189:6835","nonce":3540904386}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:09:55.624531+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4676},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4676":{"gid":4676,"name":"b","rank":0,"incarnation":17,"state":"up:active","state_seq":26,"addr":"172.21.15.47:6833/4000022090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4000022090},{"type":"v1","addr":"172.21.15.47:6833","nonce":4000022090}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:11:05.014 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:05.013+0000 7f0be759e700 1 -- 172.21.15.189:0/4048584487 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0bc40557c0 msgr2=0x7f0bc4057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:05.014 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:05.013+0000 7f0be759e700 1 --2- 172.21.15.189:0/4048584487 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0bc40557c0 0x7f0bc4057c80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f0bd80055a0 tx=0x7f0bd8009040).stop 2020-11-20T12:11:05.015 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:05.013+0000 7f0be759e700 1 -- 172.21.15.189:0/4048584487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0be813b000 msgr2=0x7f0be81366e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:05.015 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:05.013+0000 7f0be759e700 1 --2- 172.21.15.189:0/4048584487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0be813b000 0x7f0be81366e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f0be000ef50 tx=0x7f0be00091d0).stop 2020-11-20T12:11:05.015 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:05.013+0000 7f0be759e700 1 -- 172.21.15.189:0/4048584487 shutdown_connections 2020-11-20T12:11:05.015 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:05.013+0000 7f0be759e700 1 --2- 172.21.15.189:0/4048584487 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0bc40557c0 0x7f0bc4057c80 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:05.015 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:05.013+0000 7f0be759e700 1 --2- 172.21.15.189:0/4048584487 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0be80a8840 0x7f0be813bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:05.016 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:05.013+0000 7f0be759e700 1 --2- 172.21.15.189:0/4048584487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0be813b000 0x7f0be81366e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:05.016 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:05.013+0000 7f0be759e700 1 --2- 172.21.15.189:0/4048584487 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0be813a680 0x7f0be809c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:05.016 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:05.013+0000 7f0be759e700 1 -- 172.21.15.189:0/4048584487 >> 172.21.15.189:0/4048584487 conn(0x7f0be8060170 msgr2=0x7f0be80acfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:05.016 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:05.013+0000 7f0be759e700 1 -- 172.21.15.189:0/4048584487 shutdown_connections 2020-11-20T12:11:05.016 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:05.013+0000 7f0be759e700 1 -- 172.21.15.189:0/4048584487 wait complete. 2020-11-20T12:11:05.018 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 21 2020-11-20T12:11:05.028 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:11:07.029 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:07.164 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.160+0000 7ff6934bb700 1 Processor -- start 2020-11-20T12:11:07.164 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.160+0000 7ff6934bb700 1 -- start start 2020-11-20T12:11:07.167 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.160+0000 7ff6934bb700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff68c062600 0x7ff68c0a94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:07.167 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.160+0000 7ff6934bb700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff68c13c390 con 0x7ff68c062220 2020-11-20T12:11:07.167 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.160+0000 7ff6934bb700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff68c142370 con 0x7ff68c141e60 2020-11-20T12:11:07.167 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.160+0000 7ff6934bb700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff68c0a99d0 con 0x7ff68c062600 2020-11-20T12:11:07.168 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.160+0000 7ff692cba700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff68c141e60 0x7ff68c0679e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51644/0 (socket says 172.21.15.189:51644) 2020-11-20T12:11:07.168 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.160+0000 7ff692cba700 1 -- 172.21.15.189:0/3974545546 learned_addr learned my addr 172.21.15.189:0/3974545546 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:11:07.168 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.160+0000 7ff691cb8700 1 --2- 172.21.15.189:0/3974545546 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff68c062600 0x7ff68c0a94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:07.168 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.160+0000 7ff6914b7700 1 -- 172.21.15.189:0/3974545546 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680542678 0 0) 0x7ff68c142370 con 0x7ff68c141e60 2020-11-20T12:11:07.169 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.160+0000 7ff6914b7700 1 -- 172.21.15.189:0/3974545546 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff674003240 con 0x7ff68c141e60 2020-11-20T12:11:07.169 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.160+0000 7ff6914b7700 1 -- 172.21.15.189:0/3974545546 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114419501 0 0) 0x7ff68c13c390 con 0x7ff68c062220 2020-11-20T12:11:07.169 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.160+0000 7ff6914b7700 1 -- 172.21.15.189:0/3974545546 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff68c142370 con 0x7ff68c062220 2020-11-20T12:11:07.170 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.164+0000 7ff691cb8700 1 -- 172.21.15.189:0/3974545546 >> v1:172.21.15.189:6789/0 conn(0x7ff68c141e60 legacy=0x7ff68c0679e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:07.170 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.164+0000 7ff691cb8700 1 -- 172.21.15.189:0/3974545546 >> v1:172.21.15.47:6789/0 conn(0x7ff68c062220 legacy=0x7ff68c0a9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:07.170 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.164+0000 7ff691cb8700 1 -- 172.21.15.189:0/3974545546 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff68c142610 con 0x7ff68c062600 2020-11-20T12:11:07.171 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.164+0000 7ff691cb8700 1 --2- 172.21.15.189:0/3974545546 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff68c062600 0x7ff68c0a94e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7ff67c00b9d0 tx=0x7ff67c006df0).ready entity=mon.2 client_cookie=b095ef5844d43b68 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:07.171 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.164+0000 7ff6914b7700 1 -- 172.21.15.189:0/3974545546 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67c009680 con 0x7ff68c062600 2020-11-20T12:11:07.171 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.164+0000 7ff6914b7700 1 -- 172.21.15.189:0/3974545546 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff67c0097e0 con 0x7ff68c062600 2020-11-20T12:11:07.171 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.164+0000 7ff6914b7700 1 -- 172.21.15.189:0/3974545546 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67c009b60 con 0x7ff68c062600 2020-11-20T12:11:07.172 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.164+0000 7ff6934bb700 1 -- 172.21.15.189:0/3974545546 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff68c062600 msgr2=0x7ff68c0a94e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:07.172 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.164+0000 7ff6934bb700 1 --2- 172.21.15.189:0/3974545546 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff68c062600 0x7ff68c0a94e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7ff67c00b9d0 tx=0x7ff67c006df0).stop 2020-11-20T12:11:07.172 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.164+0000 7ff6934bb700 1 -- 172.21.15.189:0/3974545546 shutdown_connections 2020-11-20T12:11:07.172 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.164+0000 7ff6934bb700 1 --2- 172.21.15.189:0/3974545546 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff68c062600 0x7ff68c0a94e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:07.172 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.164+0000 7ff6934bb700 1 -- 172.21.15.189:0/3974545546 >> 172.21.15.189:0/3974545546 conn(0x7ff68c065fe0 msgr2=0x7ff68c061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:07.173 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.164+0000 7ff6934bb700 1 -- 172.21.15.189:0/3974545546 shutdown_connections 2020-11-20T12:11:07.173 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.164+0000 7ff6934bb700 1 -- 172.21.15.189:0/3974545546 wait complete. 2020-11-20T12:11:07.175 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.164+0000 7ff6934bb700 1 Processor -- start 2020-11-20T12:11:07.176 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.164+0000 7ff6934bb700 1 -- start start 2020-11-20T12:11:07.176 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.164+0000 7ff6934bb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff68c062220 0x7ff68c13ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:07.176 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.164+0000 7ff6934bb700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff68c062600 0x7ff68c0a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:07.176 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.168+0000 7ff6934bb700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff68c141e60 0x7ff68c1392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:07.177 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.168+0000 7ff6934bb700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff68c09e500 con 0x7ff68c141e60 2020-11-20T12:11:07.177 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.168+0000 7ff6934bb700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff68c09d970 con 0x7ff68c062220 2020-11-20T12:11:07.177 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.168+0000 7ff6934bb700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff68c09e730 con 0x7ff68c062600 2020-11-20T12:11:07.177 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.168+0000 7ff6924b9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff68c062220 0x7ff68c13ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:07.177 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.168+0000 7ff6924b9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff68c062220 0x7ff68c13ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50024/0 (socket says 172.21.15.189:50024) 2020-11-20T12:11:07.178 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.168+0000 7ff6924b9700 1 -- 172.21.15.189:0/1960566747 learned_addr learned my addr 172.21.15.189:0/1960566747 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:11:07.178 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.168+0000 7ff691cb8700 1 --2- 172.21.15.189:0/1960566747 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff68c062600 0x7ff68c0a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:07.178 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.168+0000 7ff692cba700 1 --2- 172.21.15.189:0/1960566747 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff68c141e60 0x7ff68c1392e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:07.178 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.168+0000 7ff6924b9700 1 -- 172.21.15.189:0/1960566747 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff68c062600 msgr2=0x7ff68c0a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:07.178 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.168+0000 7ff6924b9700 1 --2- 172.21.15.189:0/1960566747 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff68c062600 0x7ff68c0a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:07.179 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.168+0000 7ff6924b9700 1 -- 172.21.15.189:0/1960566747 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff68c141e60 msgr2=0x7ff68c1392e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:07.179 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.168+0000 7ff6924b9700 1 --2- 172.21.15.189:0/1960566747 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff68c141e60 0x7ff68c1392e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:07.179 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.168+0000 7ff6924b9700 1 -- 172.21.15.189:0/1960566747 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff68c09d660 con 0x7ff68c062220 2020-11-20T12:11:07.179 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.168+0000 7ff691cb8700 1 --2- 172.21.15.189:0/1960566747 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff68c062600 0x7ff68c0a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:11:07.180 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.168+0000 7ff6924b9700 1 --2- 172.21.15.189:0/1960566747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff68c062220 0x7ff68c13ea30 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7ff68800c740 tx=0x7ff688007a90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:07.180 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.168+0000 7ff67a7fc700 1 -- 172.21.15.189:0/1960566747 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68800e490 con 0x7ff68c062220 2020-11-20T12:11:07.180 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.168+0000 7ff67a7fc700 1 -- 172.21.15.189:0/1960566747 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff68800e5f0 con 0x7ff68c062220 2020-11-20T12:11:07.180 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.168+0000 7ff67a7fc700 1 -- 172.21.15.189:0/1960566747 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68800e970 con 0x7ff68c062220 2020-11-20T12:11:07.180 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.168+0000 7ff6934bb700 1 -- 172.21.15.189:0/1960566747 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff68c0a0cf0 con 0x7ff68c062220 2020-11-20T12:11:07.181 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.168+0000 7ff6934bb700 1 -- 172.21.15.189:0/1960566747 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff68c13f710 con 0x7ff68c062220 2020-11-20T12:11:07.181 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.168+0000 7ff6934bb700 1 -- 172.21.15.189:0/1960566747 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff68c09e8f0 con 0x7ff68c062220 2020-11-20T12:11:07.181 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.168+0000 7ff67a7fc700 1 -- 172.21.15.189:0/1960566747 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7ff688019030 con 0x7ff68c062220 2020-11-20T12:11:07.181 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.168+0000 7ff67a7fc700 1 --2- 172.21.15.189:0/1960566747 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff6640557c0 0x7ff664057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:07.181 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.168+0000 7ff67a7fc700 1 -- 172.21.15.189:0/1960566747 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6331+0+0 (secure 0 0 0) 0x7ff68806da90 con 0x7ff68c062220 2020-11-20T12:11:07.182 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.168+0000 7ff691cb8700 1 --2- 172.21.15.189:0/1960566747 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff6640557c0 0x7ff664057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:07.182 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.172+0000 7ff691cb8700 1 --2- 172.21.15.189:0/1960566747 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff6640557c0 0x7ff664057c80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7ff67c00b150 tx=0x7ff67c015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:07.182 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.176+0000 7ff67a7fc700 1 -- 172.21.15.189:0/1960566747 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff688043880 con 0x7ff68c062220 2020-11-20T12:11:07.379 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.376+0000 7ff6934bb700 1 -- 172.21.15.189:0/1960566747 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff68c0663c0 con 0x7ff68c062220 2020-11-20T12:11:07.379 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.376+0000 7ff67a7fc700 1 -- 172.21.15.189:0/1960566747 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+2140 (secure 0 0 0) 0x7ff688002e60 con 0x7ff68c062220 2020-11-20T12:11:07.380 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:11:07.380 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5053,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/3540904386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3540904386},{"type":"v1","addr":"172.21.15.189:6835","nonce":3540904386}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:09:55.624531+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4676},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4676":{"gid":4676,"name":"b","rank":0,"incarnation":17,"state":"up:active","state_seq":26,"addr":"172.21.15.47:6833/4000022090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4000022090},{"type":"v1","addr":"172.21.15.47:6833","nonce":4000022090}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:11:07.383 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.376+0000 7ff6934bb700 1 -- 172.21.15.189:0/1960566747 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff6640557c0 msgr2=0x7ff664057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:07.383 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.376+0000 7ff6934bb700 1 --2- 172.21.15.189:0/1960566747 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff6640557c0 0x7ff664057c80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7ff67c00b150 tx=0x7ff67c015000).stop 2020-11-20T12:11:07.384 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.376+0000 7ff6934bb700 1 -- 172.21.15.189:0/1960566747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff68c062220 msgr2=0x7ff68c13ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:07.384 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.376+0000 7ff6934bb700 1 --2- 172.21.15.189:0/1960566747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff68c062220 0x7ff68c13ea30 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7ff68800c740 tx=0x7ff688007a90).stop 2020-11-20T12:11:07.384 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.376+0000 7ff6934bb700 1 -- 172.21.15.189:0/1960566747 shutdown_connections 2020-11-20T12:11:07.385 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.376+0000 7ff6934bb700 1 --2- 172.21.15.189:0/1960566747 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff6640557c0 0x7ff664057c80 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:07.385 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.376+0000 7ff6934bb700 1 --2- 172.21.15.189:0/1960566747 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff68c141e60 0x7ff68c1392e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:07.386 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.376+0000 7ff6934bb700 1 --2- 172.21.15.189:0/1960566747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff68c062220 0x7ff68c13ea30 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:07.386 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.376+0000 7ff6934bb700 1 --2- 172.21.15.189:0/1960566747 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff68c062600 0x7ff68c0a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:07.386 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.376+0000 7ff6934bb700 1 -- 172.21.15.189:0/1960566747 >> 172.21.15.189:0/1960566747 conn(0x7ff68c065fe0 msgr2=0x7ff68c0a91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:07.386 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.376+0000 7ff6934bb700 1 -- 172.21.15.189:0/1960566747 shutdown_connections 2020-11-20T12:11:07.386 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:07.380+0000 7ff6934bb700 1 -- 172.21.15.189:0/1960566747 wait complete. 2020-11-20T12:11:07.387 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 21 2020-11-20T12:11:07.394 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:11:09.395 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:09.543 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.532+0000 7f943b2ed700 1 Processor -- start 2020-11-20T12:11:09.543 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.532+0000 7f943b2ed700 1 -- start start 2020-11-20T12:11:09.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.532+0000 7f943b2ed700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f94340a9e20 0x7f94340a83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:09.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.532+0000 7f943b2ed700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f9434068020 0x7f94341339f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:09.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.532+0000 7f943b2ed700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94340663e0 con 0x7f94340a9290 2020-11-20T12:11:09.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.532+0000 7f943b2ed700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9434067b80 con 0x7f9434068020 2020-11-20T12:11:09.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.532+0000 7f943b2ed700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94340a9970 con 0x7f94340a9e20 2020-11-20T12:11:09.545 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.532+0000 7f9439089700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f94340a9e20 0x7f94340a83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:09.545 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.532+0000 7f9439089700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f94340a9e20 0x7f94340a83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.189:43066/0 (socket says 172.21.15.189:43066) 2020-11-20T12:11:09.548 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.532+0000 7f9439089700 1 -- 172.21.15.189:0/2546124 learned_addr learned my addr 172.21.15.189:0/2546124 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:11:09.550 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.532+0000 7f9438888700 1 --2- 172.21.15.189:0/2546124 >> v2:172.21.15.47:3300/0 conn(0x7f9434068020 0x7f94341339f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:09.559 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.532+0000 7f9439089700 1 -- 172.21.15.189:0/2546124 >> v2:172.21.15.47:3300/0 conn(0x7f9434068020 msgr2=0x7f94341339f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:09.559 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.532+0000 7f9439089700 1 --2- 172.21.15.189:0/2546124 >> v2:172.21.15.47:3300/0 conn(0x7f9434068020 0x7f94341339f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:09.559 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.532+0000 7f9433fff700 1 -- 172.21.15.189:0/2546124 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460986027 0 0) 0x7f94340663e0 con 0x7f94340a9290 2020-11-20T12:11:09.559 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.532+0000 7f9439089700 1 -- 172.21.15.189:0/2546124 >> v1:172.21.15.47:6789/0 conn(0x7f94340a9290 legacy=0x7f9434133ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:09.560 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.532+0000 7f9439089700 1 -- 172.21.15.189:0/2546124 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9434066680 con 0x7f94340a9e20 2020-11-20T12:11:09.560 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.532+0000 7f9439089700 1 --2- 172.21.15.189:0/2546124 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f94340a9e20 0x7f94340a83f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f942c00aef0 tx=0x7f942c00a430).ready entity=mon.2 client_cookie=3ac507814fcc2f84 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:09.560 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.532+0000 7f9433fff700 1 -- 172.21.15.189:0/2546124 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c00ce60 con 0x7f94340a9e20 2020-11-20T12:11:09.560 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.532+0000 7f9433fff700 1 -- 172.21.15.189:0/2546124 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f942c010410 con 0x7f94340a9e20 2020-11-20T12:11:09.560 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.532+0000 7f9433fff700 1 -- 172.21.15.189:0/2546124 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c010790 con 0x7f94340a9e20 2020-11-20T12:11:09.561 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.536+0000 7f943b2ed700 1 -- 172.21.15.189:0/2546124 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f94340a9e20 msgr2=0x7f94340a83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:09.561 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.536+0000 7f943b2ed700 1 --2- 172.21.15.189:0/2546124 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f94340a9e20 0x7f94340a83f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f942c00aef0 tx=0x7f942c00a430).stop 2020-11-20T12:11:09.561 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.536+0000 7f943b2ed700 1 -- 172.21.15.189:0/2546124 shutdown_connections 2020-11-20T12:11:09.561 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.536+0000 7f943b2ed700 1 --2- 172.21.15.189:0/2546124 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f94340a9e20 0x7f94340a83f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:09.562 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.536+0000 7f943b2ed700 1 --2- 172.21.15.189:0/2546124 >> v2:172.21.15.47:3300/0 conn(0x7f9434068020 0x7f94341339f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:09.562 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.536+0000 7f943b2ed700 1 -- 172.21.15.189:0/2546124 >> 172.21.15.189:0/2546124 conn(0x7f94340627f0 msgr2=0x7f9434065870 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:09.562 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.536+0000 7f943b2ed700 1 -- 172.21.15.189:0/2546124 shutdown_connections 2020-11-20T12:11:09.562 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.536+0000 7f943b2ed700 1 -- 172.21.15.189:0/2546124 wait complete. 2020-11-20T12:11:09.562 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.536+0000 7f943b2ed700 1 Processor -- start 2020-11-20T12:11:09.563 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.536+0000 7f943b2ed700 1 -- start start 2020-11-20T12:11:09.563 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.540+0000 7f943b2ed700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9434068020 0x7f94340ac640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:09.563 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.540+0000 7f943b2ed700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94340a9290 0x7f94340a1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:09.563 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.540+0000 7f943b2ed700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f94340a9e20 0x7f943414b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:09.563 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.540+0000 7f943b2ed700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9434136f40 con 0x7f9434068020 2020-11-20T12:11:09.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.540+0000 7f943b2ed700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94340a35b0 con 0x7f94340a9290 2020-11-20T12:11:09.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.540+0000 7f943b2ed700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94340a3070 con 0x7f94340a9e20 2020-11-20T12:11:09.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.540+0000 7f9438888700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94340a9290 0x7f94340a1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:09.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.540+0000 7f9438888700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94340a9290 0x7f94340a1370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50038/0 (socket says 172.21.15.189:50038) 2020-11-20T12:11:09.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.540+0000 7f9438888700 1 -- 172.21.15.189:0/2816932845 learned_addr learned my addr 172.21.15.189:0/2816932845 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:11:09.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.540+0000 7f9439089700 1 --2- 172.21.15.189:0/2816932845 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9434068020 0x7f94340ac640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:09.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.540+0000 7f943988a700 1 --2- 172.21.15.189:0/2816932845 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f94340a9e20 0x7f943414b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:09.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.540+0000 7f9438888700 1 -- 172.21.15.189:0/2816932845 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f94340a9e20 msgr2=0x7f943414b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:09.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.540+0000 7f9438888700 1 --2- 172.21.15.189:0/2816932845 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f94340a9e20 0x7f943414b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:09.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.540+0000 7f9438888700 1 -- 172.21.15.189:0/2816932845 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9434068020 msgr2=0x7f94340ac640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:09.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.540+0000 7f9438888700 1 --2- 172.21.15.189:0/2816932845 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9434068020 0x7f94340ac640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:09.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.540+0000 7f9438888700 1 -- 172.21.15.189:0/2816932845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94340a3310 con 0x7f94340a9290 2020-11-20T12:11:09.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.540+0000 7f9438888700 1 --2- 172.21.15.189:0/2816932845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94340a9290 0x7f94340a1370 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f9434066210 tx=0x7f9428008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:09.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.540+0000 7f94317fa700 1 -- 172.21.15.189:0/2816932845 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942800c820 con 0x7f94340a9290 2020-11-20T12:11:09.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.540+0000 7f94317fa700 1 -- 172.21.15.189:0/2816932845 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f942800c980 con 0x7f94340a9290 2020-11-20T12:11:09.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.540+0000 7f94317fa700 1 -- 172.21.15.189:0/2816932845 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942800cd00 con 0x7f94340a9290 2020-11-20T12:11:09.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.540+0000 7f943b2ed700 1 -- 172.21.15.189:0/2816932845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94340a2d60 con 0x7f94340a9290 2020-11-20T12:11:09.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.540+0000 7f943b2ed700 1 -- 172.21.15.189:0/2816932845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f943414ad00 con 0x7f94340a9290 2020-11-20T12:11:09.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.540+0000 7f943b2ed700 1 -- 172.21.15.189:0/2816932845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9434044f60 con 0x7f94340a9290 2020-11-20T12:11:09.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.544+0000 7f94317fa700 1 -- 172.21.15.189:0/2816932845 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f9428012600 con 0x7f94340a9290 2020-11-20T12:11:09.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.544+0000 7f94317fa700 1 --2- 172.21.15.189:0/2816932845 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f940c0557c0 0x7f940c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:09.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.544+0000 7f94317fa700 1 -- 172.21.15.189:0/2816932845 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6331+0+0 (secure 0 0 0) 0x7f942806e870 con 0x7f94340a9290 2020-11-20T12:11:09.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.544+0000 7f9439089700 1 --2- 172.21.15.189:0/2816932845 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f940c0557c0 0x7f940c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:09.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.544+0000 7f9439089700 1 --2- 172.21.15.189:0/2816932845 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f940c0557c0 0x7f940c057c80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f942c00cb60 tx=0x7f94340678e0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:09.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.548+0000 7f94317fa700 1 -- 172.21.15.189:0/2816932845 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9428044610 con 0x7f94340a9290 2020-11-20T12:11:09.755 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.752+0000 7f943b2ed700 1 -- 172.21.15.189:0/2816932845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94340aab70 con 0x7f94340a9290 2020-11-20T12:11:09.756 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.752+0000 7f94317fa700 1 -- 172.21.15.189:0/2816932845 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f9428016020 con 0x7f94340a9290 2020-11-20T12:11:09.757 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:11:09.757 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5053,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/3540904386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3540904386},{"type":"v1","addr":"172.21.15.189:6835","nonce":3540904386}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:09:55.624531+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4676},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4676":{"gid":4676,"name":"b","rank":0,"incarnation":17,"state":"up:active","state_seq":26,"addr":"172.21.15.47:6833/4000022090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4000022090},{"type":"v1","addr":"172.21.15.47:6833","nonce":4000022090}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:11:09.759 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.756+0000 7f943b2ed700 1 -- 172.21.15.189:0/2816932845 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f940c0557c0 msgr2=0x7f940c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:09.759 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.756+0000 7f943b2ed700 1 --2- 172.21.15.189:0/2816932845 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f940c0557c0 0x7f940c057c80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f942c00cb60 tx=0x7f94340678e0).stop 2020-11-20T12:11:09.760 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.756+0000 7f943b2ed700 1 -- 172.21.15.189:0/2816932845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94340a9290 msgr2=0x7f94340a1370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:09.760 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.756+0000 7f943b2ed700 1 --2- 172.21.15.189:0/2816932845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94340a9290 0x7f94340a1370 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f9434066210 tx=0x7f9428008040).stop 2020-11-20T12:11:09.760 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.756+0000 7f943b2ed700 1 -- 172.21.15.189:0/2816932845 shutdown_connections 2020-11-20T12:11:09.761 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.756+0000 7f943b2ed700 1 --2- 172.21.15.189:0/2816932845 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f940c0557c0 0x7f940c057c80 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:09.761 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.756+0000 7f943b2ed700 1 --2- 172.21.15.189:0/2816932845 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9434068020 0x7f94340ac640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:09.762 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.756+0000 7f943b2ed700 1 --2- 172.21.15.189:0/2816932845 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f94340a9e20 0x7f943414b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:09.762 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.756+0000 7f943b2ed700 1 --2- 172.21.15.189:0/2816932845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94340a9290 0x7f94340a1370 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:09.762 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.756+0000 7f943b2ed700 1 -- 172.21.15.189:0/2816932845 >> 172.21.15.189:0/2816932845 conn(0x7f94340627f0 msgr2=0x7f94340a8970 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:09.763 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.756+0000 7f943b2ed700 1 -- 172.21.15.189:0/2816932845 shutdown_connections 2020-11-20T12:11:09.763 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:09.756+0000 7f943b2ed700 1 -- 172.21.15.189:0/2816932845 wait complete. 2020-11-20T12:11:09.764 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 21 2020-11-20T12:11:09.773 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:11:11.774 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:11.908 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.904+0000 7fc0f7fff700 1 Processor -- start 2020-11-20T12:11:11.908 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.904+0000 7fc0f7fff700 1 -- start start 2020-11-20T12:11:11.910 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.904+0000 7fc0f7fff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc0f80612b0 0x7fc0f80a0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:11.910 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.904+0000 7fc0f7fff700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc0f81376c0 0x7fc0f8133a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:11.910 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.904+0000 7fc0f7fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f8068020 con 0x7fc0f8067b50 2020-11-20T12:11:11.910 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.904+0000 7fc0f7fff700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0f80682c0 con 0x7fc0f81376c0 2020-11-20T12:11:11.910 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.904+0000 7fc0f7fff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f8061690 con 0x7fc0f80612b0 2020-11-20T12:11:11.911 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.904+0000 7fc0f6ffd700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc0f80612b0 0x7fc0f80a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:11.911 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.904+0000 7fc0f77fe700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc0f81376c0 0x7fc0f8133a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:11.911 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.904+0000 7fc0f6ffd700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc0f80612b0 0x7fc0f80a0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.189:43084/0 (socket says 172.21.15.189:43084) 2020-11-20T12:11:11.911 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.904+0000 7fc0f6ffd700 1 -- 172.21.15.189:0/3540228735 learned_addr learned my addr 172.21.15.189:0/3540228735 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:11:11.912 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.904+0000 7fc0f67fc700 1 --1- 172.21.15.189:0/3540228735 >> v1:172.21.15.47:6789/0 conn(0x7fc0f8067b50 0x7fc0f8136fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.189:46334/0 (socket says 172.21.15.189:46334) 2020-11-20T12:11:11.914 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.908+0000 7fc0f6ffd700 1 -- 172.21.15.189:0/3540228735 >> v2:172.21.15.47:3300/0 conn(0x7fc0f81376c0 msgr2=0x7fc0f8133a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:11.914 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.908+0000 7fc0f5ffb700 1 -- 172.21.15.189:0/3540228735 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854885361 0 0) 0x7fc0f8068020 con 0x7fc0f8067b50 2020-11-20T12:11:11.914 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.908+0000 7fc0f6ffd700 1 --2- 172.21.15.189:0/3540228735 >> v2:172.21.15.47:3300/0 conn(0x7fc0f81376c0 0x7fc0f8133a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:11.914 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.908+0000 7fc0f6ffd700 1 -- 172.21.15.189:0/3540228735 >> v1:172.21.15.47:6789/0 conn(0x7fc0f8067b50 legacy=0x7fc0f8136fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:11.915 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.908+0000 7fc0f6ffd700 1 -- 172.21.15.189:0/3540228735 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f8136aa0 con 0x7fc0f80612b0 2020-11-20T12:11:11.915 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.908+0000 7fc0f6ffd700 1 --2- 172.21.15.189:0/3540228735 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc0f80612b0 0x7fc0f80a0960 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fc0e8009d50 tx=0x7fc0e8009290).ready entity=mon.2 client_cookie=d48410248272c287 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:11.915 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.908+0000 7fc0f5ffb700 1 -- 172.21.15.189:0/3540228735 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e800bbc0 con 0x7fc0f80612b0 2020-11-20T12:11:11.915 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.908+0000 7fc0f5ffb700 1 -- 172.21.15.189:0/3540228735 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e800bd20 con 0x7fc0f80612b0 2020-11-20T12:11:11.916 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.908+0000 7fc0f5ffb700 1 -- 172.21.15.189:0/3540228735 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e8011610 con 0x7fc0f80612b0 2020-11-20T12:11:11.916 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.908+0000 7fc0f7fff700 1 -- 172.21.15.189:0/3540228735 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc0f80612b0 msgr2=0x7fc0f80a0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:11.916 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.908+0000 7fc0f7fff700 1 --2- 172.21.15.189:0/3540228735 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc0f80612b0 0x7fc0f80a0960 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fc0e8009d50 tx=0x7fc0e8009290).stop 2020-11-20T12:11:11.916 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.908+0000 7fc0f7fff700 1 -- 172.21.15.189:0/3540228735 shutdown_connections 2020-11-20T12:11:11.917 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.908+0000 7fc0f7fff700 1 --2- 172.21.15.189:0/3540228735 >> v2:172.21.15.47:3300/0 conn(0x7fc0f81376c0 0x7fc0f8133a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:11.917 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.908+0000 7fc0f7fff700 1 --2- 172.21.15.189:0/3540228735 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc0f80612b0 0x7fc0f80a0960 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:11.917 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.908+0000 7fc0f7fff700 1 -- 172.21.15.189:0/3540228735 >> 172.21.15.189:0/3540228735 conn(0x7fc0f8063210 msgr2=0x7fc0f8060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:11.917 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.908+0000 7fc0f7fff700 1 -- 172.21.15.189:0/3540228735 shutdown_connections 2020-11-20T12:11:11.917 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.908+0000 7fc0f7fff700 1 -- 172.21.15.189:0/3540228735 wait complete. 2020-11-20T12:11:11.918 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.908+0000 7fc0f7fff700 1 Processor -- start 2020-11-20T12:11:11.918 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.908+0000 7fc0f7fff700 1 -- start start 2020-11-20T12:11:11.918 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.908+0000 7fc0f7fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0f80612b0 0x7fc0f80ac0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:11.918 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.908+0000 7fc0f7fff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc0f8067b50 0x7fc0f80ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:11.918 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.908+0000 7fc0f7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0f81376c0 0x7fc0f8139820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:11.919 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.908+0000 7fc0f7fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f80a6910 con 0x7fc0f80612b0 2020-11-20T12:11:11.919 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.908+0000 7fc0f7fff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f80a5790 con 0x7fc0f81376c0 2020-11-20T12:11:11.919 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.908+0000 7fc0f7fff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f80a6b40 con 0x7fc0f8067b50 2020-11-20T12:11:11.919 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.912+0000 7fc0f77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0f81376c0 0x7fc0f8139820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:11.919 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.912+0000 7fc0f77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0f81376c0 0x7fc0f8139820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50058/0 (socket says 172.21.15.189:50058) 2020-11-20T12:11:11.920 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.912+0000 7fc0f77fe700 1 -- 172.21.15.189:0/1855650822 learned_addr learned my addr 172.21.15.189:0/1855650822 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:11:11.921 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.912+0000 7fc0f77fe700 1 -- 172.21.15.189:0/1855650822 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc0f8067b50 msgr2=0x7fc0f80ac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:11.922 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.912+0000 7fc0f77fe700 1 --2- 172.21.15.189:0/1855650822 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc0f8067b50 0x7fc0f80ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:11.922 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.912+0000 7fc0f77fe700 1 -- 172.21.15.189:0/1855650822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0f80612b0 msgr2=0x7fc0f80ac0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:11.922 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.912+0000 7fc0f77fe700 1 --2- 172.21.15.189:0/1855650822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0f80612b0 0x7fc0f80ac0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:11.922 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.912+0000 7fc0f77fe700 1 -- 172.21.15.189:0/1855650822 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f813ad40 con 0x7fc0f81376c0 2020-11-20T12:11:11.923 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.912+0000 7fc0f77fe700 1 --2- 172.21.15.189:0/1855650822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0f81376c0 0x7fc0f8139820 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fc0f0006bd0 tx=0x7fc0f0006e80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:11.923 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.912+0000 7fc0deffd700 1 -- 172.21.15.189:0/1855650822 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f000d5c0 con 0x7fc0f81376c0 2020-11-20T12:11:11.923 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.912+0000 7fc0deffd700 1 -- 172.21.15.189:0/1855650822 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f0002bf0 con 0x7fc0f81376c0 2020-11-20T12:11:11.923 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.912+0000 7fc0deffd700 1 -- 172.21.15.189:0/1855650822 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f0010690 con 0x7fc0f81376c0 2020-11-20T12:11:11.923 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.912+0000 7fc0f7fff700 1 -- 172.21.15.189:0/1855650822 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0f80acdd0 con 0x7fc0f81376c0 2020-11-20T12:11:11.924 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.912+0000 7fc0f7fff700 1 -- 172.21.15.189:0/1855650822 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0f80a4bf0 con 0x7fc0f81376c0 2020-11-20T12:11:11.924 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.912+0000 7fc0deffd700 1 -- 172.21.15.189:0/1855650822 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fc0f0010a40 con 0x7fc0f81376c0 2020-11-20T12:11:11.924 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.912+0000 7fc0f7fff700 1 -- 172.21.15.189:0/1855650822 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0f80a6d00 con 0x7fc0f81376c0 2020-11-20T12:11:11.924 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.916+0000 7fc0deffd700 1 --2- 172.21.15.189:0/1855650822 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc0d40557c0 0x7fc0d4057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:11.924 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.916+0000 7fc0deffd700 1 -- 172.21.15.189:0/1855650822 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6331+0+0 (secure 0 0 0) 0x7fc0f006ee60 con 0x7fc0f81376c0 2020-11-20T12:11:11.925 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.916+0000 7fc0f6ffd700 1 --2- 172.21.15.189:0/1855650822 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc0d40557c0 0x7fc0d4057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:11.925 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.916+0000 7fc0f6ffd700 1 --2- 172.21.15.189:0/1855650822 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc0d40557c0 0x7fc0d4057c80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fc0e8009fd0 tx=0x7fc0e8016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:11.925 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:11.920+0000 7fc0deffd700 1 -- 172.21.15.189:0/1855650822 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc0f0044c00 con 0x7fc0f81376c0 2020-11-20T12:11:12.130 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:12.128+0000 7fc0f7fff700 1 -- 172.21.15.189:0/1855650822 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0f80a58d0 con 0x7fc0f81376c0 2020-11-20T12:11:12.131 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:12.128+0000 7fc0deffd700 1 -- 172.21.15.189:0/1855650822 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fc0f0014070 con 0x7fc0f81376c0 2020-11-20T12:11:12.131 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:11:12.132 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5053,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/3540904386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3540904386},{"type":"v1","addr":"172.21.15.189:6835","nonce":3540904386}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:09:55.624531+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4676},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4676":{"gid":4676,"name":"b","rank":0,"incarnation":17,"state":"up:active","state_seq":26,"addr":"172.21.15.47:6833/4000022090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4000022090},{"type":"v1","addr":"172.21.15.47:6833","nonce":4000022090}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:11:12.134 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:12.128+0000 7fc0f7fff700 1 -- 172.21.15.189:0/1855650822 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc0d40557c0 msgr2=0x7fc0d4057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:12.134 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:12.128+0000 7fc0f7fff700 1 --2- 172.21.15.189:0/1855650822 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc0d40557c0 0x7fc0d4057c80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fc0e8009fd0 tx=0x7fc0e8016000).stop 2020-11-20T12:11:12.134 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:12.128+0000 7fc0f7fff700 1 -- 172.21.15.189:0/1855650822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0f81376c0 msgr2=0x7fc0f8139820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:12.135 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:12.128+0000 7fc0f7fff700 1 --2- 172.21.15.189:0/1855650822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0f81376c0 0x7fc0f8139820 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fc0f0006bd0 tx=0x7fc0f0006e80).stop 2020-11-20T12:11:12.135 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:12.128+0000 7fc0f7fff700 1 -- 172.21.15.189:0/1855650822 shutdown_connections 2020-11-20T12:11:12.135 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:12.128+0000 7fc0f7fff700 1 --2- 172.21.15.189:0/1855650822 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc0d40557c0 0x7fc0d4057c80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:12.135 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:12.128+0000 7fc0f7fff700 1 --2- 172.21.15.189:0/1855650822 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc0f80612b0 0x7fc0f80ac0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:12.135 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:12.128+0000 7fc0f7fff700 1 --2- 172.21.15.189:0/1855650822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0f81376c0 0x7fc0f8139820 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:12.136 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:12.128+0000 7fc0f7fff700 1 --2- 172.21.15.189:0/1855650822 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc0f8067b50 0x7fc0f80ac8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T12:11:12.136 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:12.128+0000 7fc0f7fff700 1 -- 172.21.15.189:0/1855650822 >> 172.21.15.189:0/1855650822 conn(0x7fc0f8063210 msgr2=0x7fc0f812ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:12.136 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:12.128+0000 7fc0f7fff700 1 -- 172.21.15.189:0/1855650822 shutdown_connections 2020-11-20T12:11:12.136 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:12.128+0000 7fc0f7fff700 1 -- 172.21.15.189:0/1855650822 wait complete. 2020-11-20T12:11:12.137 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 21 2020-11-20T12:11:12.146 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:11:14.147 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:14.280 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.276+0000 7f5690ab5700 1 Processor -- start 2020-11-20T12:11:14.280 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.276+0000 7f5690ab5700 1 -- start start 2020-11-20T12:11:14.282 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.276+0000 7f5690ab5700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f568c0a9410 0x7f568c135450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:14.282 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.276+0000 7f5690ab5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f568c135940 0x7f568c142df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:14.282 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.276+0000 7f5690ab5700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f568c0a9a60 con 0x7f568c1386b0 2020-11-20T12:11:14.282 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.276+0000 7f5690ab5700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f568c0a9d00 con 0x7f568c0a9410 2020-11-20T12:11:14.283 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.276+0000 7f5690ab5700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f568c0a9fa0 con 0x7f568c135940 2020-11-20T12:11:14.283 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.276+0000 7f568bfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f568c135940 0x7f568c142df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:14.283 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.276+0000 7f568bfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f568c135940 0x7f568c142df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50076/0 (socket says 172.21.15.189:50076) 2020-11-20T12:11:14.283 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.276+0000 7f568bfff700 1 -- 172.21.15.189:0/3835891288 learned_addr learned my addr 172.21.15.189:0/3835891288 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:11:14.283 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.276+0000 7f568affd700 1 --2- 172.21.15.189:0/3835891288 >> v2:172.21.15.47:3300/0 conn(0x7f568c0a9410 0x7f568c135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:14.284 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.276+0000 7f568bfff700 1 -- 172.21.15.189:0/3835891288 >> v2:172.21.15.47:3300/0 conn(0x7f568c0a9410 msgr2=0x7f568c135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:14.284 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.276+0000 7f568bfff700 1 --2- 172.21.15.189:0/3835891288 >> v2:172.21.15.47:3300/0 conn(0x7f568c0a9410 0x7f568c135450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:14.284 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.276+0000 7f568bfff700 1 -- 172.21.15.189:0/3835891288 >> v1:172.21.15.47:6789/0 conn(0x7f568c1386b0 legacy=0x7f568c0a50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:14.284 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.276+0000 7f568bfff700 1 -- 172.21.15.189:0/3835891288 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f568c060ea0 con 0x7f568c135940 2020-11-20T12:11:14.284 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.276+0000 7f568bfff700 1 --2- 172.21.15.189:0/3835891288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f568c135940 0x7f568c142df0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f568c0a4fc0 tx=0x7f5684009290).ready entity=mon.1 client_cookie=c08da891a638848f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:14.285 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f568a7fc700 1 -- 172.21.15.189:0/3835891288 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568400bc50 con 0x7f568c135940 2020-11-20T12:11:14.285 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f568a7fc700 1 -- 172.21.15.189:0/3835891288 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f568400bdb0 con 0x7f568c135940 2020-11-20T12:11:14.285 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f568a7fc700 1 -- 172.21.15.189:0/3835891288 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5684011610 con 0x7f568c135940 2020-11-20T12:11:14.288 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f5690ab5700 1 -- 172.21.15.189:0/3835891288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f568c135940 msgr2=0x7f568c142df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:14.288 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f5690ab5700 1 --2- 172.21.15.189:0/3835891288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f568c135940 0x7f568c142df0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f568c0a4fc0 tx=0x7f5684009290).stop 2020-11-20T12:11:14.288 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f5690ab5700 1 -- 172.21.15.189:0/3835891288 shutdown_connections 2020-11-20T12:11:14.288 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f5690ab5700 1 --2- 172.21.15.189:0/3835891288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f568c135940 0x7f568c142df0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:14.289 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f5690ab5700 1 --2- 172.21.15.189:0/3835891288 >> v2:172.21.15.47:3300/0 conn(0x7f568c0a9410 0x7f568c135450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:14.289 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f5690ab5700 1 -- 172.21.15.189:0/3835891288 >> 172.21.15.189:0/3835891288 conn(0x7f568c060170 msgr2=0x7f568c060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:14.289 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f5690ab5700 1 -- 172.21.15.189:0/3835891288 shutdown_connections 2020-11-20T12:11:14.289 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f5690ab5700 1 -- 172.21.15.189:0/3835891288 wait complete. 2020-11-20T12:11:14.289 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f5690ab5700 1 Processor -- start 2020-11-20T12:11:14.290 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f5690ab5700 1 -- start start 2020-11-20T12:11:14.290 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f5690ab5700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f568c0a9410 0x7f568c134140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:14.290 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f5690ab5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f568c135940 0x7f568c13a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:14.290 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f5690ab5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f568c1386b0 0x7f568c0a8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:14.290 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f5690ab5700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568c13bde0 con 0x7f568c135940 2020-11-20T12:11:14.291 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f5690ab5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568c13c080 con 0x7f568c1386b0 2020-11-20T12:11:14.291 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f5690ab5700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f568c13bad0 con 0x7f568c0a9410 2020-11-20T12:11:14.291 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f568bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f568c1386b0 0x7f568c0a8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:14.291 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f568bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f568c1386b0 0x7f568c0a8440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50082/0 (socket says 172.21.15.189:50082) 2020-11-20T12:11:14.291 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f568bfff700 1 -- 172.21.15.189:0/1040833446 learned_addr learned my addr 172.21.15.189:0/1040833446 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:11:14.292 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f568b7fe700 1 --2- 172.21.15.189:0/1040833446 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f568c0a9410 0x7f568c134140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:14.292 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f568affd700 1 --2- 172.21.15.189:0/1040833446 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f568c135940 0x7f568c13a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:14.292 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f568bfff700 1 -- 172.21.15.189:0/1040833446 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f568c0a9410 msgr2=0x7f568c134140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:14.292 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f568bfff700 1 --2- 172.21.15.189:0/1040833446 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f568c0a9410 0x7f568c134140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:14.292 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f568bfff700 1 -- 172.21.15.189:0/1040833446 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f568c135940 msgr2=0x7f568c13a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:14.293 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f568bfff700 1 --2- 172.21.15.189:0/1040833446 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f568c135940 0x7f568c13a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:14.293 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.280+0000 7f568bfff700 1 -- 172.21.15.189:0/1040833446 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f568c13a520 con 0x7f568c1386b0 2020-11-20T12:11:14.293 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.284+0000 7f568bfff700 1 --2- 172.21.15.189:0/1040833446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f568c1386b0 0x7f568c0a8440 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f568c0aa0e0 tx=0x7f5684016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:14.293 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.284+0000 7f56837fe700 1 -- 172.21.15.189:0/1040833446 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568400b750 con 0x7f568c1386b0 2020-11-20T12:11:14.293 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.284+0000 7f56837fe700 1 -- 172.21.15.189:0/1040833446 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5684011e30 con 0x7f568c1386b0 2020-11-20T12:11:14.294 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.284+0000 7f5690ab5700 1 -- 172.21.15.189:0/1040833446 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f568c1347f0 con 0x7f568c1386b0 2020-11-20T12:11:14.294 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.284+0000 7f56837fe700 1 -- 172.21.15.189:0/1040833446 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5684010cf0 con 0x7f568c1386b0 2020-11-20T12:11:14.294 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.284+0000 7f5690ab5700 1 -- 172.21.15.189:0/1040833446 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f568c0a8ad0 con 0x7f568c1386b0 2020-11-20T12:11:14.294 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.284+0000 7f56837fe700 1 -- 172.21.15.189:0/1040833446 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f5684017070 con 0x7f568c1386b0 2020-11-20T12:11:14.294 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.284+0000 7f5690ab5700 1 -- 172.21.15.189:0/1040833446 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f568c13c240 con 0x7f568c1386b0 2020-11-20T12:11:14.295 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.284+0000 7f56837fe700 1 --2- 172.21.15.189:0/1040833446 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f56680557c0 0x7f5668057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:14.296 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.284+0000 7f568b7fe700 1 --2- 172.21.15.189:0/1040833446 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f56680557c0 0x7f5668057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:14.296 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.284+0000 7f56837fe700 1 -- 172.21.15.189:0/1040833446 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f5684070ab0 con 0x7f568c1386b0 2020-11-20T12:11:14.296 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.284+0000 7f568b7fe700 1 --2- 172.21.15.189:0/1040833446 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f56680557c0 0x7f5668057c80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f5674000bc0 tx=0x7f5674007b70).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:14.296 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.288+0000 7f56837fe700 1 -- 172.21.15.189:0/1040833446 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f56840467f0 con 0x7f568c1386b0 2020-11-20T12:11:14.496 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.492+0000 7f5690ab5700 1 -- 172.21.15.189:0/1040833446 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f568c0446d0 con 0x7f568c1386b0 2020-11-20T12:11:14.496 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.492+0000 7f56837fe700 1 -- 172.21.15.189:0/1040833446 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+1791 (secure 0 0 0) 0x7f5684015020 con 0x7f568c1386b0 2020-11-20T12:11:14.497 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:11:14.497 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:11:13.602884+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5053},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5053":{"gid":5053,"name":"a","rank":0,"incarnation":22,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.189:6835/3540904386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3540904386},{"type":"v1","addr":"172.21.15.189:6835","nonce":3540904386}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:11:14.498 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.492+0000 7f5690ab5700 1 -- 172.21.15.189:0/1040833446 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f56680557c0 msgr2=0x7f5668057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:14.498 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.492+0000 7f5690ab5700 1 --2- 172.21.15.189:0/1040833446 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f56680557c0 0x7f5668057c80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f5674000bc0 tx=0x7f5674007b70).stop 2020-11-20T12:11:14.498 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.492+0000 7f5690ab5700 1 -- 172.21.15.189:0/1040833446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f568c1386b0 msgr2=0x7f568c0a8440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:14.499 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.492+0000 7f5690ab5700 1 --2- 172.21.15.189:0/1040833446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f568c1386b0 0x7f568c0a8440 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f568c0aa0e0 tx=0x7f5684016000).stop 2020-11-20T12:11:14.499 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.496+0000 7f5690ab5700 1 -- 172.21.15.189:0/1040833446 shutdown_connections 2020-11-20T12:11:14.499 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.496+0000 7f5690ab5700 1 --2- 172.21.15.189:0/1040833446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f568c1386b0 0x7f568c0a8440 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:14.499 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.496+0000 7f5690ab5700 1 --2- 172.21.15.189:0/1040833446 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f568c0a9410 0x7f568c134140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:14.500 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.496+0000 7f5690ab5700 1 --2- 172.21.15.189:0/1040833446 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f56680557c0 0x7f5668057c80 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:14.500 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.496+0000 7f5690ab5700 1 --2- 172.21.15.189:0/1040833446 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f568c135940 0x7f568c13a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:14.500 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.496+0000 7f5690ab5700 1 -- 172.21.15.189:0/1040833446 >> 172.21.15.189:0/1040833446 conn(0x7f568c060170 msgr2=0x7f568c138c30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:14.500 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.496+0000 7f5690ab5700 1 -- 172.21.15.189:0/1040833446 shutdown_connections 2020-11-20T12:11:14.500 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.496+0000 7f5690ab5700 1 -- 172.21.15.189:0/1040833446 wait complete. 2020-11-20T12:11:14.501 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 23 2020-11-20T12:11:14.511 INFO:tasks.mds_thrash.fs.[None]:mds.b down, removed from mdsmap 2020-11-20T12:11:14.512 INFO:tasks.mds_thrash.fs.[None]:waiting for mds cluster to stabilize... 2020-11-20T12:11:14.512 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:14.642 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.628+0000 7fd8dac1d700 1 Processor -- start 2020-11-20T12:11:14.642 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.628+0000 7fd8dac1d700 1 -- start start 2020-11-20T12:11:14.643 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.628+0000 7fd8dac1d700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd8d40612b0 0x7fd8d40a0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:14.643 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.628+0000 7fd8dac1d700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd8d4067b50 0x7fd8d4136fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:14.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.628+0000 7fd8dac1d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd8d41374a0 0x7fd8d4133a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:14.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.628+0000 7fd8dac1d700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8d4068020 con 0x7fd8d4067b50 2020-11-20T12:11:14.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.628+0000 7fd8dac1d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8d40682c0 con 0x7fd8d41374a0 2020-11-20T12:11:14.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.628+0000 7fd8dac1d700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d41378f0 con 0x7fd8d40612b0 2020-11-20T12:11:14.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.628+0000 7fd8da41c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd8d41374a0 0x7fd8d4133a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:14.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.628+0000 7fd8da41c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd8d41374a0 0x7fd8d4133a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50090/0 (socket says 172.21.15.189:50090) 2020-11-20T12:11:14.646 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.628+0000 7fd8da41c700 1 -- 172.21.15.189:0/1794627485 learned_addr learned my addr 172.21.15.189:0/1794627485 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:11:14.646 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.628+0000 7fd8d941a700 1 --2- 172.21.15.189:0/1794627485 >> v2:172.21.15.47:3300/0 conn(0x7fd8d4067b50 0x7fd8d4136fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:14.646 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.628+0000 7fd8d9c1b700 1 --2- 172.21.15.189:0/1794627485 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd8d40612b0 0x7fd8d40a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:14.647 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.628+0000 7fd8da41c700 1 -- 172.21.15.189:0/1794627485 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd8d40612b0 msgr2=0x7fd8d40a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:14.647 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.628+0000 7fd8da41c700 1 --2- 172.21.15.189:0/1794627485 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd8d40612b0 0x7fd8d40a0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:14.648 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.628+0000 7fd8da41c700 1 -- 172.21.15.189:0/1794627485 >> v2:172.21.15.47:3300/0 conn(0x7fd8d4067b50 msgr2=0x7fd8d4136fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:14.648 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.628+0000 7fd8da41c700 1 --2- 172.21.15.189:0/1794627485 >> v2:172.21.15.47:3300/0 conn(0x7fd8d4067b50 0x7fd8d4136fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:14.648 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.628+0000 7fd8da41c700 1 -- 172.21.15.189:0/1794627485 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d4061690 con 0x7fd8d41374a0 2020-11-20T12:11:14.649 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.628+0000 7fd8da41c700 1 --2- 172.21.15.189:0/1794627485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d41374a0 0x7fd8d4133a10 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fd8d4065840 tx=0x7fd8c4006eb0).ready entity=mon.1 client_cookie=399cc42a5fb3322e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:14.649 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8d8c19700 1 -- 172.21.15.189:0/1794627485 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c400f770 con 0x7fd8d41374a0 2020-11-20T12:11:14.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8d8c19700 1 -- 172.21.15.189:0/1794627485 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c400f8d0 con 0x7fd8d41374a0 2020-11-20T12:11:14.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8d8c19700 1 -- 172.21.15.189:0/1794627485 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c400fc50 con 0x7fd8d41374a0 2020-11-20T12:11:14.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8dac1d700 1 -- 172.21.15.189:0/1794627485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d41374a0 msgr2=0x7fd8d4133a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:14.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8dac1d700 1 --2- 172.21.15.189:0/1794627485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d41374a0 0x7fd8d4133a10 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fd8d4065840 tx=0x7fd8c4006eb0).stop 2020-11-20T12:11:14.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8dac1d700 1 -- 172.21.15.189:0/1794627485 shutdown_connections 2020-11-20T12:11:14.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8dac1d700 1 --2- 172.21.15.189:0/1794627485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d41374a0 0x7fd8d4133a10 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:14.652 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8dac1d700 1 --2- 172.21.15.189:0/1794627485 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd8d40612b0 0x7fd8d40a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:14.652 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8dac1d700 1 --2- 172.21.15.189:0/1794627485 >> v2:172.21.15.47:3300/0 conn(0x7fd8d4067b50 0x7fd8d4136fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:14.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8dac1d700 1 -- 172.21.15.189:0/1794627485 >> 172.21.15.189:0/1794627485 conn(0x7fd8d4063210 msgr2=0x7fd8d4060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:14.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8dac1d700 1 -- 172.21.15.189:0/1794627485 shutdown_connections 2020-11-20T12:11:14.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8dac1d700 1 -- 172.21.15.189:0/1794627485 wait complete. 2020-11-20T12:11:14.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8dac1d700 1 Processor -- start 2020-11-20T12:11:14.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8dac1d700 1 -- start start 2020-11-20T12:11:14.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8dac1d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d40612b0 0x7fd8d40a69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:14.661 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8dac1d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8d4067b50 0x7fd8d4145bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:14.662 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8dac1d700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd8d41374a0 0x7fd8d41460e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:14.662 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8dac1d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d40a5fc0 con 0x7fd8d4067b50 2020-11-20T12:11:14.663 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8dac1d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d40a5d20 con 0x7fd8d40612b0 2020-11-20T12:11:14.663 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8dac1d700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d4145750 con 0x7fd8d41374a0 2020-11-20T12:11:14.663 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8d9c1b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d40612b0 0x7fd8d40a69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:14.664 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8d9c1b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d40612b0 0x7fd8d40a69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50092/0 (socket says 172.21.15.189:50092) 2020-11-20T12:11:14.664 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8d9c1b700 1 -- 172.21.15.189:0/807218910 learned_addr learned my addr 172.21.15.189:0/807218910 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:11:14.664 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8da41c700 1 --2- 172.21.15.189:0/807218910 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd8d41374a0 0x7fd8d41460e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:14.665 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8d941a700 1 --2- 172.21.15.189:0/807218910 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8d4067b50 0x7fd8d4145bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:14.665 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8d9c1b700 1 -- 172.21.15.189:0/807218910 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd8d41374a0 msgr2=0x7fd8d41460e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:14.665 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8d9c1b700 1 --2- 172.21.15.189:0/807218910 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd8d41374a0 0x7fd8d41460e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:14.666 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8d9c1b700 1 -- 172.21.15.189:0/807218910 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8d4067b50 msgr2=0x7fd8d4145bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:14.667 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8d9c1b700 1 --2- 172.21.15.189:0/807218910 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8d4067b50 0x7fd8d4145bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:14.667 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8d9c1b700 1 -- 172.21.15.189:0/807218910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d40a4740 con 0x7fd8d40612b0 2020-11-20T12:11:14.667 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8d9c1b700 1 --2- 172.21.15.189:0/807218910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d40612b0 0x7fd8d40a69b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7fd8cc00eda0 tx=0x7fd8cc00a370).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:14.668 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8d1ffb700 1 -- 172.21.15.189:0/807218910 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc00a7e0 con 0x7fd8d40612b0 2020-11-20T12:11:14.668 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8d1ffb700 1 -- 172.21.15.189:0/807218910 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8cc00cd70 con 0x7fd8d40612b0 2020-11-20T12:11:14.668 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8dac1d700 1 -- 172.21.15.189:0/807218910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8d40a63b0 con 0x7fd8d40612b0 2020-11-20T12:11:14.669 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8d1ffb700 1 -- 172.21.15.189:0/807218910 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc011600 con 0x7fd8d40612b0 2020-11-20T12:11:14.669 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.632+0000 7fd8dac1d700 1 -- 172.21.15.189:0/807218910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8d40a59c0 con 0x7fd8d40612b0 2020-11-20T12:11:14.669 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.636+0000 7fd8dac1d700 1 -- 172.21.15.189:0/807218910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8d40a8bd0 con 0x7fd8d40612b0 2020-11-20T12:11:14.670 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.636+0000 7fd8d1ffb700 1 -- 172.21.15.189:0/807218910 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fd8cc021070 con 0x7fd8d40612b0 2020-11-20T12:11:14.670 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.636+0000 7fd8d1ffb700 1 --2- 172.21.15.189:0/807218910 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd8ac0557c0 0x7fd8ac057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:14.670 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.636+0000 7fd8d1ffb700 1 -- 172.21.15.189:0/807218910 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fd8cc06ee90 con 0x7fd8d40612b0 2020-11-20T12:11:14.671 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.636+0000 7fd8d941a700 1 --2- 172.21.15.189:0/807218910 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd8ac0557c0 0x7fd8ac057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:14.671 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.636+0000 7fd8d941a700 1 --2- 172.21.15.189:0/807218910 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd8ac0557c0 0x7fd8ac057c80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fd8d40ae330 tx=0x7fd8c8008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:14.671 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.640+0000 7fd8d1ffb700 1 -- 172.21.15.189:0/807218910 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd8cc044bd0 con 0x7fd8d40612b0 2020-11-20T12:11:14.846 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.836+0000 7fd8dac1d700 1 -- 172.21.15.189:0/807218910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8d40446d0 con 0x7fd8d40612b0 2020-11-20T12:11:14.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.840+0000 7fd8d1ffb700 1 -- 172.21.15.189:0/807218910 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+1788 (secure 0 0 0) 0x7fd8cc015070 con 0x7fd8d40612b0 2020-11-20T12:11:14.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.840+0000 7fd8dac1d700 1 -- 172.21.15.189:0/807218910 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd8ac0557c0 msgr2=0x7fd8ac057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:14.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.840+0000 7fd8dac1d700 1 --2- 172.21.15.189:0/807218910 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd8ac0557c0 0x7fd8ac057c80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fd8d40ae330 tx=0x7fd8c8008040).stop 2020-11-20T12:11:14.848 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.840+0000 7fd8dac1d700 1 -- 172.21.15.189:0/807218910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d40612b0 msgr2=0x7fd8d40a69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:14.848 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.840+0000 7fd8dac1d700 1 --2- 172.21.15.189:0/807218910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d40612b0 0x7fd8d40a69b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7fd8cc00eda0 tx=0x7fd8cc00a370).stop 2020-11-20T12:11:14.848 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.840+0000 7fd8dac1d700 1 -- 172.21.15.189:0/807218910 shutdown_connections 2020-11-20T12:11:14.848 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.840+0000 7fd8dac1d700 1 --2- 172.21.15.189:0/807218910 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd8d41374a0 0x7fd8d41460e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:14.848 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.840+0000 7fd8dac1d700 1 --2- 172.21.15.189:0/807218910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d40612b0 0x7fd8d40a69b0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:14.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.840+0000 7fd8dac1d700 1 --2- 172.21.15.189:0/807218910 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd8ac0557c0 0x7fd8ac057c80 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:14.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.840+0000 7fd8dac1d700 1 --2- 172.21.15.189:0/807218910 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd8d4067b50 0x7fd8d4145bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:14.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.840+0000 7fd8dac1d700 1 -- 172.21.15.189:0/807218910 >> 172.21.15.189:0/807218910 conn(0x7fd8d4063210 msgr2=0x7fd8d40a7960 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:14.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.840+0000 7fd8dac1d700 1 -- 172.21.15.189:0/807218910 shutdown_connections 2020-11-20T12:11:14.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:14.840+0000 7fd8dac1d700 1 -- 172.21.15.189:0/807218910 wait complete. 2020-11-20T12:11:14.850 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 24 2020-11-20T12:11:14.850 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:11:14.850 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:11:14.610417+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5053},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5053":{"gid":5053,"name":"a","rank":0,"incarnation":22,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.189:6835/3540904386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3540904386},{"type":"v1","addr":"172.21.15.189:6835","nonce":3540904386}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:11:14.865 INFO:tasks.mds_thrash.fs.[None]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 24, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2020-11-20T12:04:54.012110+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 24, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5053": { "addr": "172.21.15.189:6835/3540904386", "addrs": { "addrvec": [ { "addr": "172.21.15.189:6834", "nonce": 3540904386, "type": "v2" }, { "addr": "172.21.15.189:6835", "nonce": 3540904386, "type": "v1" } ] }, "export_targets": [], "features": 4540138297136906239, "flags": 0, "gid": 5053, "incarnation": 22, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 26, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2020-11-20T12:11:14.610417+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5053 } } } ], "standbys": [] } 2020-11-20T12:11:16.866 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:16.997 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.992+0000 7fd2b759e700 1 Processor -- start 2020-11-20T12:11:16.997 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.992+0000 7fd2b759e700 1 -- start start 2020-11-20T12:11:16.999 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.992+0000 7fd2b759e700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd2b8067b50 0x7fd2b8136fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:16.999 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.992+0000 7fd2b759e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd2b81374a0 0x7fd2b8133a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:16.999 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.992+0000 7fd2b759e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b8068020 con 0x7fd2b80612b0 2020-11-20T12:11:16.999 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.992+0000 7fd2b759e700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2b80682c0 con 0x7fd2b8067b50 2020-11-20T12:11:16.999 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.992+0000 7fd2b759e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2b8137900 con 0x7fd2b81374a0 2020-11-20T12:11:17.000 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.992+0000 7fd2b6d9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd2b81374a0 0x7fd2b8133a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:17.000 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2b6d9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd2b81374a0 0x7fd2b8133a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50106/0 (socket says 172.21.15.189:50106) 2020-11-20T12:11:17.000 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2b6d9d700 1 -- 172.21.15.189:0/1119398057 learned_addr learned my addr 172.21.15.189:0/1119398057 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:11:17.000 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2affff700 1 --2- 172.21.15.189:0/1119398057 >> v2:172.21.15.47:3300/0 conn(0x7fd2b8067b50 0x7fd2b8136fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:17.001 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2b6d9d700 1 -- 172.21.15.189:0/1119398057 >> v2:172.21.15.47:3300/0 conn(0x7fd2b8067b50 msgr2=0x7fd2b8136fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:17.001 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2b6d9d700 1 --2- 172.21.15.189:0/1119398057 >> v2:172.21.15.47:3300/0 conn(0x7fd2b8067b50 0x7fd2b8136fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:17.001 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2b6d9d700 1 -- 172.21.15.189:0/1119398057 >> v1:172.21.15.47:6789/0 conn(0x7fd2b80612b0 legacy=0x7fd2b80a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:17.001 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2b6d9d700 1 -- 172.21.15.189:0/1119398057 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b8061690 con 0x7fd2b81374a0 2020-11-20T12:11:17.001 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2b6d9d700 1 --2- 172.21.15.189:0/1119398057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2b81374a0 0x7fd2b8133a10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fd2b80ad260 tx=0x7fd2b0009290).ready entity=mon.1 client_cookie=fcc950e4969a66a1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:17.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2b5d9b700 1 -- 172.21.15.189:0/1119398057 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b000bbc0 con 0x7fd2b81374a0 2020-11-20T12:11:17.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2b5d9b700 1 -- 172.21.15.189:0/1119398057 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b000bd20 con 0x7fd2b81374a0 2020-11-20T12:11:17.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2b5d9b700 1 -- 172.21.15.189:0/1119398057 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b0011610 con 0x7fd2b81374a0 2020-11-20T12:11:17.004 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2b759e700 1 -- 172.21.15.189:0/1119398057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2b81374a0 msgr2=0x7fd2b8133a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:17.005 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2b759e700 1 --2- 172.21.15.189:0/1119398057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2b81374a0 0x7fd2b8133a10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fd2b80ad260 tx=0x7fd2b0009290).stop 2020-11-20T12:11:17.005 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2b759e700 1 -- 172.21.15.189:0/1119398057 shutdown_connections 2020-11-20T12:11:17.005 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2b759e700 1 --2- 172.21.15.189:0/1119398057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2b81374a0 0x7fd2b8133a10 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:17.005 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2b759e700 1 --2- 172.21.15.189:0/1119398057 >> v2:172.21.15.47:3300/0 conn(0x7fd2b8067b50 0x7fd2b8136fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:17.006 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2b759e700 1 -- 172.21.15.189:0/1119398057 >> 172.21.15.189:0/1119398057 conn(0x7fd2b8063210 msgr2=0x7fd2b8060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:17.006 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2b759e700 1 -- 172.21.15.189:0/1119398057 shutdown_connections 2020-11-20T12:11:17.006 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2b759e700 1 -- 172.21.15.189:0/1119398057 wait complete. 2020-11-20T12:11:17.006 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2b759e700 1 Processor -- start 2020-11-20T12:11:17.006 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2b759e700 1 -- start start 2020-11-20T12:11:17.007 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2b759e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2b80612b0 0x7fd2b812cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:17.007 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2b759e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2b8067b50 0x7fd2b81457c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:17.007 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2b759e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd2b81374a0 0x7fd2b81294b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:17.007 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2b759e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b8132990 con 0x7fd2b80612b0 2020-11-20T12:11:17.007 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2b759e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b8132c30 con 0x7fd2b8067b50 2020-11-20T12:11:17.008 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2b759e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b8132ed0 con 0x7fd2b81374a0 2020-11-20T12:11:17.008 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2affff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2b8067b50 0x7fd2b81457c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:17.008 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:16.996+0000 7fd2affff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2b8067b50 0x7fd2b81457c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50110/0 (socket says 172.21.15.189:50110) 2020-11-20T12:11:17.008 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.000+0000 7fd2affff700 1 -- 172.21.15.189:0/2078605159 learned_addr learned my addr 172.21.15.189:0/2078605159 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:11:17.008 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.000+0000 7fd2b6d9d700 1 --2- 172.21.15.189:0/2078605159 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd2b81374a0 0x7fd2b81294b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:17.009 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.000+0000 7fd2b659c700 1 --2- 172.21.15.189:0/2078605159 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2b80612b0 0x7fd2b812cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:17.009 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.000+0000 7fd2affff700 1 -- 172.21.15.189:0/2078605159 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd2b81374a0 msgr2=0x7fd2b81294b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:17.009 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.000+0000 7fd2affff700 1 --2- 172.21.15.189:0/2078605159 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd2b81374a0 0x7fd2b81294b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:17.009 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.000+0000 7fd2affff700 1 -- 172.21.15.189:0/2078605159 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2b80612b0 msgr2=0x7fd2b812cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:17.009 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.000+0000 7fd2affff700 1 --2- 172.21.15.189:0/2078605159 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2b80612b0 0x7fd2b812cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:17.010 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.000+0000 7fd2affff700 1 -- 172.21.15.189:0/2078605159 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b8133370 con 0x7fd2b8067b50 2020-11-20T12:11:17.010 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.000+0000 7fd2affff700 1 --2- 172.21.15.189:0/2078605159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2b8067b50 0x7fd2b81457c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fd2a000db30 tx=0x7fd2a0008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:17.010 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.000+0000 7fd2ae7fc700 1 -- 172.21.15.189:0/2078605159 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a00097c0 con 0x7fd2b8067b50 2020-11-20T12:11:17.010 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.000+0000 7fd2ae7fc700 1 -- 172.21.15.189:0/2078605159 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a0009920 con 0x7fd2b8067b50 2020-11-20T12:11:17.010 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.000+0000 7fd2b759e700 1 -- 172.21.15.189:0/2078605159 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2b8132610 con 0x7fd2b8067b50 2020-11-20T12:11:17.011 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.000+0000 7fd2ae7fc700 1 -- 172.21.15.189:0/2078605159 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a0009ca0 con 0x7fd2b8067b50 2020-11-20T12:11:17.011 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.000+0000 7fd2b759e700 1 -- 172.21.15.189:0/2078605159 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2b8131bf0 con 0x7fd2b8067b50 2020-11-20T12:11:17.011 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.000+0000 7fd2b759e700 1 -- 172.21.15.189:0/2078605159 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2b8066c80 con 0x7fd2b8067b50 2020-11-20T12:11:17.011 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.000+0000 7fd2ae7fc700 1 -- 172.21.15.189:0/2078605159 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fd2a0013690 con 0x7fd2b8067b50 2020-11-20T12:11:17.012 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.000+0000 7fd2ae7fc700 1 --2- 172.21.15.189:0/2078605159 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd2940557c0 0x7fd294057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:17.012 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.000+0000 7fd2ae7fc700 1 -- 172.21.15.189:0/2078605159 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fd2a006e820 con 0x7fd2b8067b50 2020-11-20T12:11:17.013 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.000+0000 7fd2b659c700 1 --2- 172.21.15.189:0/2078605159 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd2940557c0 0x7fd294057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:17.013 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.004+0000 7fd2b659c700 1 --2- 172.21.15.189:0/2078605159 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd2940557c0 0x7fd294057c80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fd2a80034f0 tx=0x7fd2a8009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:17.013 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.008+0000 7fd2ae7fc700 1 -- 172.21.15.189:0/2078605159 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd2a0044560 con 0x7fd2b8067b50 2020-11-20T12:11:17.254 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.252+0000 7fd2b759e700 1 -- 172.21.15.189:0/2078605159 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2b8132800 con 0x7fd2b8067b50 2020-11-20T12:11:17.255 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.252+0000 7fd2ae7fc700 1 -- 172.21.15.189:0/2078605159 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+1788 (secure 0 0 0) 0x7fd2a0047b80 con 0x7fd2b8067b50 2020-11-20T12:11:17.255 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:11:17.255 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:11:15.620495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5053},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5053":{"gid":5053,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/3540904386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3540904386},{"type":"v1","addr":"172.21.15.189:6835","nonce":3540904386}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:11:17.257 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.252+0000 7fd2b759e700 1 -- 172.21.15.189:0/2078605159 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd2940557c0 msgr2=0x7fd294057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:17.257 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.252+0000 7fd2b759e700 1 --2- 172.21.15.189:0/2078605159 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd2940557c0 0x7fd294057c80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fd2a80034f0 tx=0x7fd2a8009040).stop 2020-11-20T12:11:17.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.252+0000 7fd2b759e700 1 -- 172.21.15.189:0/2078605159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2b8067b50 msgr2=0x7fd2b81457c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:17.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.252+0000 7fd2b759e700 1 --2- 172.21.15.189:0/2078605159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2b8067b50 0x7fd2b81457c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fd2a000db30 tx=0x7fd2a0008040).stop 2020-11-20T12:11:17.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.252+0000 7fd2b759e700 1 -- 172.21.15.189:0/2078605159 shutdown_connections 2020-11-20T12:11:17.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.252+0000 7fd2b759e700 1 --2- 172.21.15.189:0/2078605159 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd2940557c0 0x7fd294057c80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:17.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.252+0000 7fd2b759e700 1 --2- 172.21.15.189:0/2078605159 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd2b80612b0 0x7fd2b812cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:17.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.252+0000 7fd2b759e700 1 --2- 172.21.15.189:0/2078605159 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd2b81374a0 0x7fd2b81294b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:17.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.252+0000 7fd2b759e700 1 --2- 172.21.15.189:0/2078605159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2b8067b50 0x7fd2b81457c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:17.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.252+0000 7fd2b759e700 1 -- 172.21.15.189:0/2078605159 >> 172.21.15.189:0/2078605159 conn(0x7fd2b8063210 msgr2=0x7fd2b80a9870 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:17.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.252+0000 7fd2b759e700 1 -- 172.21.15.189:0/2078605159 shutdown_connections 2020-11-20T12:11:17.260 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:17.252+0000 7fd2b759e700 1 -- 172.21.15.189:0/2078605159 wait complete. 2020-11-20T12:11:17.261 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 25 2020-11-20T12:11:17.271 INFO:tasks.mds_thrash.fs.[None]:mds.a has gained rank=0, replacing gid=4676 2020-11-20T12:11:17.272 INFO:tasks.mds_thrash.fs.[None]:waiting for 4 secs before reviving mds.b 2020-11-20T12:11:20.586 INFO:tasks.ceph.mds.a.smithi189.stderr:2020-11-20T12:11:20.579+0000 7f1a658da700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:11:21.273 INFO:tasks.mds_thrash.fs.[None]:reviving mds.b 2020-11-20T12:11:21.273 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:11:21.274 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:11:21.287 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:11:21.288 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:21.369 INFO:tasks.ceph.mds.b.smithi047.stdout:starting mds.b at 2020-11-20T12:11:21.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.415+0000 7f9067524700 1 Processor -- start 2020-11-20T12:11:21.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.415+0000 7f9067524700 1 -- start start 2020-11-20T12:11:21.423 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.419+0000 7f9067524700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9060062220 0x7f906013a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:21.423 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.419+0000 7f9067524700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f9060062600 0x7f9060066fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:21.423 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.419+0000 7f9067524700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90600a9bb0 con 0x7f9060137750 2020-11-20T12:11:21.424 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.419+0000 7f9067524700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90600674c0 con 0x7f9060062600 2020-11-20T12:11:21.424 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.419+0000 7f9067524700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f906013ab40 con 0x7f9060062220 2020-11-20T12:11:21.424 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.419+0000 7f9066522700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9060062220 0x7f906013a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:21.424 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.419+0000 7f9066522700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9060062220 0x7f906013a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50116/0 (socket says 172.21.15.189:50116) 2020-11-20T12:11:21.425 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.419+0000 7f9066522700 1 -- 172.21.15.189:0/53425140 learned_addr learned my addr 172.21.15.189:0/53425140 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:11:21.425 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.419+0000 7f9065d21700 1 --2- 172.21.15.189:0/53425140 >> v2:172.21.15.47:3300/0 conn(0x7f9060062600 0x7f9060066fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:21.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.419+0000 7f9066522700 1 -- 172.21.15.189:0/53425140 >> v2:172.21.15.47:3300/0 conn(0x7f9060062600 msgr2=0x7f9060066fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:21.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.419+0000 7f9066522700 1 --2- 172.21.15.189:0/53425140 >> v2:172.21.15.47:3300/0 conn(0x7f9060062600 0x7f9060066fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:21.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.419+0000 7f9066522700 1 -- 172.21.15.189:0/53425140 >> v1:172.21.15.189:6789/0 conn(0x7f9060137750 legacy=0x7f9060060a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:21.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.419+0000 7f9066522700 1 -- 172.21.15.189:0/53425140 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f906013ade0 con 0x7f9060062220 2020-11-20T12:11:21.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.419+0000 7f9066522700 1 --2- 172.21.15.189:0/53425140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9060062220 0x7f906013a4a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f90600ae8f0 tx=0x7f905c006eb0).ready entity=mon.1 client_cookie=d936632f1e7d6987 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:21.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.419+0000 7f9065520700 1 -- 172.21.15.189:0/53425140 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905c009960 con 0x7f9060062220 2020-11-20T12:11:21.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.419+0000 7f9065520700 1 -- 172.21.15.189:0/53425140 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f905c009ac0 con 0x7f9060062220 2020-11-20T12:11:21.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.419+0000 7f9065520700 1 -- 172.21.15.189:0/53425140 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905c009e40 con 0x7f9060062220 2020-11-20T12:11:21.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.419+0000 7f9067524700 1 -- 172.21.15.189:0/53425140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9060062220 msgr2=0x7f906013a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:21.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.419+0000 7f9067524700 1 --2- 172.21.15.189:0/53425140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9060062220 0x7f906013a4a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f90600ae8f0 tx=0x7f905c006eb0).stop 2020-11-20T12:11:21.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.419+0000 7f9067524700 1 -- 172.21.15.189:0/53425140 shutdown_connections 2020-11-20T12:11:21.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.419+0000 7f9067524700 1 --2- 172.21.15.189:0/53425140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9060062220 0x7f906013a4a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:21.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.419+0000 7f9067524700 1 --2- 172.21.15.189:0/53425140 >> v2:172.21.15.47:3300/0 conn(0x7f9060062600 0x7f9060066fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:21.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.419+0000 7f9067524700 1 -- 172.21.15.189:0/53425140 >> 172.21.15.189:0/53425140 conn(0x7f9060065fe0 msgr2=0x7f9060061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:21.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.419+0000 7f9067524700 1 -- 172.21.15.189:0/53425140 shutdown_connections 2020-11-20T12:11:21.430 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.419+0000 7f9067524700 1 -- 172.21.15.189:0/53425140 wait complete. 2020-11-20T12:11:21.430 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.423+0000 7f9067524700 1 Processor -- start 2020-11-20T12:11:21.430 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.423+0000 7f9067524700 1 -- start start 2020-11-20T12:11:21.432 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.423+0000 7f9067524700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9060062220 0x7f906013a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:21.432 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.423+0000 7f9067524700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9060062600 0x7f90600a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:21.433 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.423+0000 7f9067524700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9060137750 0x7f906014dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:21.433 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.423+0000 7f9067524700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90600a60c0 con 0x7f9060062220 2020-11-20T12:11:21.433 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.423+0000 7f9067524700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90600a6360 con 0x7f9060062600 2020-11-20T12:11:21.433 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.423+0000 7f9067524700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90600a6600 con 0x7f9060137750 2020-11-20T12:11:21.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.423+0000 7f9065d21700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9060062600 0x7f90600a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:21.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.423+0000 7f9065d21700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9060062600 0x7f90600a4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50124/0 (socket says 172.21.15.189:50124) 2020-11-20T12:11:21.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.423+0000 7f9065d21700 1 -- 172.21.15.189:0/692707067 learned_addr learned my addr 172.21.15.189:0/692707067 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:11:21.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.423+0000 7f9066522700 1 --2- 172.21.15.189:0/692707067 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9060062220 0x7f906013a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:21.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.423+0000 7f9066d23700 1 --2- 172.21.15.189:0/692707067 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9060137750 0x7f906014dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:21.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.423+0000 7f9065d21700 1 -- 172.21.15.189:0/692707067 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9060137750 msgr2=0x7f906014dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:21.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.423+0000 7f9065d21700 1 --2- 172.21.15.189:0/692707067 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9060137750 0x7f906014dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:21.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.423+0000 7f9065d21700 1 -- 172.21.15.189:0/692707067 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9060062220 msgr2=0x7f906013a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:21.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.423+0000 7f9065d21700 1 --2- 172.21.15.189:0/692707067 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9060062220 0x7f906013a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:21.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.423+0000 7f9065d21700 1 -- 172.21.15.189:0/692707067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f906014d6b0 con 0x7f9060062600 2020-11-20T12:11:21.436 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.423+0000 7f9065d21700 1 --2- 172.21.15.189:0/692707067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9060062600 0x7f90600a4ef0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f905000db60 tx=0x7f9050008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:21.436 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.423+0000 7f904e7fc700 1 -- 172.21.15.189:0/692707067 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9050009790 con 0x7f9060062600 2020-11-20T12:11:21.436 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.423+0000 7f904e7fc700 1 -- 172.21.15.189:0/692707067 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90500098f0 con 0x7f9060062600 2020-11-20T12:11:21.436 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.423+0000 7f9067524700 1 -- 172.21.15.189:0/692707067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90601380b0 con 0x7f9060062600 2020-11-20T12:11:21.437 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.423+0000 7f904e7fc700 1 -- 172.21.15.189:0/692707067 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9050009c90 con 0x7f9060062600 2020-11-20T12:11:21.437 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.423+0000 7f9067524700 1 -- 172.21.15.189:0/692707067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90600a53e0 con 0x7f9060062600 2020-11-20T12:11:21.437 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.427+0000 7f9067524700 1 -- 172.21.15.189:0/692707067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90600663c0 con 0x7f9060062600 2020-11-20T12:11:21.437 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.427+0000 7f904e7fc700 1 -- 172.21.15.189:0/692707067 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f9050013480 con 0x7f9060062600 2020-11-20T12:11:21.437 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.427+0000 7f904e7fc700 1 --2- 172.21.15.189:0/692707067 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f90380557c0 0x7f9038057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:21.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.427+0000 7f904e7fc700 1 -- 172.21.15.189:0/692707067 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f905006f8a0 con 0x7f9060062600 2020-11-20T12:11:21.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.427+0000 7f9066522700 1 --2- 172.21.15.189:0/692707067 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f90380557c0 0x7f9038057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:21.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.427+0000 7f9066522700 1 --2- 172.21.15.189:0/692707067 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f90380557c0 0x7f9038057c80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f905c006ef0 tx=0x7f905c015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:21.439 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.431+0000 7f904e7fc700 1 -- 172.21.15.189:0/692707067 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9050045600 con 0x7f9060062600 2020-11-20T12:11:21.683 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.679+0000 7f9067524700 1 -- 172.21.15.189:0/692707067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9060137530 con 0x7f9060062600 2020-11-20T12:11:21.684 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.679+0000 7f904e7fc700 1 -- 172.21.15.189:0/692707067 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f9050048c20 con 0x7f9060062600 2020-11-20T12:11:21.684 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:11:21.685 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5122,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/4088050649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4088050649},{"type":"v1","addr":"172.21.15.47:6833","nonce":4088050649}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:11:15.620495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5053},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5053":{"gid":5053,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/3540904386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3540904386},{"type":"v1","addr":"172.21.15.189:6835","nonce":3540904386}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:11:21.686 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.679+0000 7f9067524700 1 -- 172.21.15.189:0/692707067 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f90380557c0 msgr2=0x7f9038057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:21.686 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.679+0000 7f9067524700 1 --2- 172.21.15.189:0/692707067 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f90380557c0 0x7f9038057c80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f905c006ef0 tx=0x7f905c015000).stop 2020-11-20T12:11:21.686 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.679+0000 7f9067524700 1 -- 172.21.15.189:0/692707067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9060062600 msgr2=0x7f90600a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:21.687 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.679+0000 7f9067524700 1 --2- 172.21.15.189:0/692707067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9060062600 0x7f90600a4ef0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f905000db60 tx=0x7f9050008040).stop 2020-11-20T12:11:21.687 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.679+0000 7f9067524700 1 -- 172.21.15.189:0/692707067 shutdown_connections 2020-11-20T12:11:21.687 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.679+0000 7f9067524700 1 --2- 172.21.15.189:0/692707067 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f90380557c0 0x7f9038057c80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:21.687 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.679+0000 7f9067524700 1 --2- 172.21.15.189:0/692707067 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9060062220 0x7f906013a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:21.687 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.679+0000 7f9067524700 1 --2- 172.21.15.189:0/692707067 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9060137750 0x7f906014dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:21.688 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.679+0000 7f9067524700 1 --2- 172.21.15.189:0/692707067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9060062600 0x7f90600a4ef0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:21.688 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.679+0000 7f9067524700 1 -- 172.21.15.189:0/692707067 >> 172.21.15.189:0/692707067 conn(0x7f9060065fe0 msgr2=0x7f9060136d70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:21.688 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.679+0000 7f9067524700 1 -- 172.21.15.189:0/692707067 shutdown_connections 2020-11-20T12:11:21.688 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:21.679+0000 7f9067524700 1 -- 172.21.15.189:0/692707067 wait complete. 2020-11-20T12:11:21.689 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 26 2020-11-20T12:11:21.704 INFO:tasks.mds_thrash.fs.[None]:mds.b reported in up:standby state 2020-11-20T12:11:21.704 INFO:tasks.mds_thrash.fs.[None]:waiting for 27 secs before thrashing 2020-11-20T12:11:23.292 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:11:23.296 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:11:48.706 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:48.843 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.837+0000 7f342a930700 1 Processor -- start 2020-11-20T12:11:48.843 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.837+0000 7f342a930700 1 -- start start 2020-11-20T12:11:48.844 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.837+0000 7f342a930700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f34240678c0 0x7f34240a8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:48.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.837+0000 7f342a930700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34240aeb40 con 0x7f34240ae390 2020-11-20T12:11:48.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.837+0000 7f342a930700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34240ae8a0 con 0x7f3424067ca0 2020-11-20T12:11:48.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.837+0000 7f342a930700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3424137690 con 0x7f34240678c0 2020-11-20T12:11:48.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f34237fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3424067ca0 0x7f342413b330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51752/0 (socket says 172.21.15.189:51752) 2020-11-20T12:11:48.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f34237fe700 1 -- 172.21.15.189:0/3408201106 learned_addr learned my addr 172.21.15.189:0/3408201106 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:11:48.846 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3423fff700 1 --2- 172.21.15.189:0/3408201106 >> v2:172.21.15.47:3300/0 conn(0x7f34240678c0 0x7f34240a8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:48.848 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3422ffd700 1 -- 172.21.15.189:0/3408201106 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364250290 0 0) 0x7f34240ae8a0 con 0x7f3424067ca0 2020-11-20T12:11:48.852 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3422ffd700 1 -- 172.21.15.189:0/3408201106 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3404003240 con 0x7f3424067ca0 2020-11-20T12:11:48.852 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3422ffd700 1 -- 172.21.15.189:0/3408201106 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891052259 0 0) 0x7f34240aeb40 con 0x7f34240ae390 2020-11-20T12:11:48.853 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3422ffd700 1 -- 172.21.15.189:0/3408201106 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34240ae8a0 con 0x7f34240ae390 2020-11-20T12:11:48.853 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3422ffd700 1 -- 172.21.15.189:0/3408201106 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2131472053 0 0) 0x7f3404003240 con 0x7f3424067ca0 2020-11-20T12:11:48.853 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3422ffd700 1 -- 172.21.15.189:0/3408201106 >> v2:172.21.15.47:3300/0 conn(0x7f34240678c0 msgr2=0x7f34240a8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:48.853 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3422ffd700 1 --2- 172.21.15.189:0/3408201106 >> v2:172.21.15.47:3300/0 conn(0x7f34240678c0 0x7f34240a8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:48.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3422ffd700 1 -- 172.21.15.189:0/3408201106 >> v1:172.21.15.47:6789/0 conn(0x7f34240ae390 legacy=0x7f3424137b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:48.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3422ffd700 1 -- 172.21.15.189:0/3408201106 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34240a9170 con 0x7f3424067ca0 2020-11-20T12:11:48.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3422ffd700 1 -- 172.21.15.189:0/3408201106 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f34140014e0 con 0x7f3424067ca0 2020-11-20T12:11:48.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3422ffd700 1 -- 172.21.15.189:0/3408201106 >> v1:172.21.15.189:6789/0 conn(0x7f3424067ca0 legacy=0x7f342413b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:48.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3422ffd700 1 --2- 172.21.15.189:0/3408201106 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34040038c0 0x7f3404003cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:48.855 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3422ffd700 1 --2- 172.21.15.189:0/3408201106 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f34040044e0 0x7f3404004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:48.855 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3422ffd700 1 --2- 172.21.15.189:0/3408201106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3404004e50 0x7f340400b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:48.855 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3422ffd700 1 -- 172.21.15.189:0/3408201106 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3424137690 con 0x7f34040038c0 2020-11-20T12:11:48.855 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3422ffd700 1 -- 172.21.15.189:0/3408201106 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34040053b0 con 0x7f3404004e50 2020-11-20T12:11:48.855 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3422ffd700 1 -- 172.21.15.189:0/3408201106 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f340400bca0 con 0x7f34040044e0 2020-11-20T12:11:48.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3428ecd700 1 --2- 172.21.15.189:0/3408201106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3404004e50 0x7f340400b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:48.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3423fff700 1 --2- 172.21.15.189:0/3408201106 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34040038c0 0x7f3404003cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:48.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f34237fe700 1 --2- 172.21.15.189:0/3408201106 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f34040044e0 0x7f3404004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:48.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3428ecd700 1 -- 172.21.15.189:0/3408201106 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f34040044e0 msgr2=0x7f3404004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:48.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3428ecd700 1 --2- 172.21.15.189:0/3408201106 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f34040044e0 0x7f3404004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:48.857 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3428ecd700 1 -- 172.21.15.189:0/3408201106 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34040038c0 msgr2=0x7f3404003cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:48.857 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3428ecd700 1 --2- 172.21.15.189:0/3408201106 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34040038c0 0x7f3404003cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:48.857 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3428ecd700 1 -- 172.21.15.189:0/3408201106 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3404003240 con 0x7f3404004e50 2020-11-20T12:11:48.857 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f34237fe700 1 --2- 172.21.15.189:0/3408201106 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f34040044e0 0x7f3404004960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:11:48.857 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3428ecd700 1 -- 172.21.15.189:0/3408201106 reap_dead start 2020-11-20T12:11:48.858 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3428ecd700 1 --2- 172.21.15.189:0/3408201106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3404004e50 0x7f340400b730 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f341c001fd0 tx=0x7f341c007f10).ready entity=mon.1 client_cookie=3f7da63ba66385c5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:48.858 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3422ffd700 1 -- 172.21.15.189:0/3408201106 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341c0016a0 con 0x7f3404004e50 2020-11-20T12:11:48.858 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3422ffd700 1 -- 172.21.15.189:0/3408201106 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341c00cbc0 con 0x7f3404004e50 2020-11-20T12:11:48.858 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f3422ffd700 1 -- 172.21.15.189:0/3408201106 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341c006870 con 0x7f3404004e50 2020-11-20T12:11:48.858 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f342a930700 1 -- 172.21.15.189:0/3408201106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3404004e50 msgr2=0x7f340400b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:48.859 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f342a930700 1 --2- 172.21.15.189:0/3408201106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3404004e50 0x7f340400b730 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f341c001fd0 tx=0x7f341c007f10).stop 2020-11-20T12:11:48.859 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f342a930700 1 -- 172.21.15.189:0/3408201106 shutdown_connections 2020-11-20T12:11:48.860 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f342a930700 1 --2- 172.21.15.189:0/3408201106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3404004e50 0x7f340400b730 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:48.860 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f342a930700 1 -- 172.21.15.189:0/3408201106 >> 172.21.15.189:0/3408201106 conn(0x7f34240625a0 msgr2=0x7f34240adc80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:48.860 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f342a930700 1 -- 172.21.15.189:0/3408201106 shutdown_connections 2020-11-20T12:11:48.862 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f342a930700 1 -- 172.21.15.189:0/3408201106 wait complete. 2020-11-20T12:11:48.862 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f342a930700 1 Processor -- start 2020-11-20T12:11:48.862 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.841+0000 7f342a930700 1 -- start start 2020-11-20T12:11:48.862 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.845+0000 7f342a930700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3404004e50 0x7f342409e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:48.862 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.845+0000 7f342a930700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f342409c970 0x7f34240a42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:48.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.845+0000 7f342a930700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34241460b0 0x7f3424146530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:48.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.845+0000 7f342a930700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3424066cb0 con 0x7f34241460b0 2020-11-20T12:11:48.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.845+0000 7f342a930700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34240a8390 con 0x7f342409c970 2020-11-20T12:11:48.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.845+0000 7f342a930700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f342409cdf0 con 0x7f3404004e50 2020-11-20T12:11:48.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.845+0000 7f34237fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f342409c970 0x7f34240a42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:48.864 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.845+0000 7f34237fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f342409c970 0x7f34240a42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50142/0 (socket says 172.21.15.189:50142) 2020-11-20T12:11:48.864 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.845+0000 7f34237fe700 1 -- 172.21.15.189:0/630045826 learned_addr learned my addr 172.21.15.189:0/630045826 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:11:48.864 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.845+0000 7f3428ecd700 1 --2- 172.21.15.189:0/630045826 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34241460b0 0x7f3424146530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:48.864 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.845+0000 7f3423fff700 1 --2- 172.21.15.189:0/630045826 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3404004e50 0x7f342409e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:48.864 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.845+0000 7f34237fe700 1 -- 172.21.15.189:0/630045826 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3404004e50 msgr2=0x7f342409e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:48.865 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.845+0000 7f34237fe700 1 --2- 172.21.15.189:0/630045826 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3404004e50 0x7f342409e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:48.865 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.845+0000 7f34237fe700 1 -- 172.21.15.189:0/630045826 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34241460b0 msgr2=0x7f3424146530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:48.865 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.845+0000 7f34237fe700 1 --2- 172.21.15.189:0/630045826 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34241460b0 0x7f3424146530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:48.865 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.845+0000 7f34237fe700 1 -- 172.21.15.189:0/630045826 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f342409e940 con 0x7f342409c970 2020-11-20T12:11:48.866 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.845+0000 7f34237fe700 1 --2- 172.21.15.189:0/630045826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f342409c970 0x7f34240a42a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f341400ea80 tx=0x7f3414001620).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:48.866 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.845+0000 7f341bfff700 1 -- 172.21.15.189:0/630045826 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341400a610 con 0x7f342409c970 2020-11-20T12:11:48.866 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.845+0000 7f341bfff700 1 -- 172.21.15.189:0/630045826 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3414010040 con 0x7f342409c970 2020-11-20T12:11:48.866 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.845+0000 7f341bfff700 1 -- 172.21.15.189:0/630045826 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341400a9a0 con 0x7f342409c970 2020-11-20T12:11:48.866 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.845+0000 7f342a930700 1 -- 172.21.15.189:0/630045826 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34240a3e00 con 0x7f342409c970 2020-11-20T12:11:48.867 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.845+0000 7f342a930700 1 -- 172.21.15.189:0/630045826 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34241459d0 con 0x7f342409c970 2020-11-20T12:11:48.867 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.845+0000 7f342a930700 1 -- 172.21.15.189:0/630045826 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f342409e790 con 0x7f342409c970 2020-11-20T12:11:48.867 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.845+0000 7f341bfff700 1 -- 172.21.15.189:0/630045826 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f341401b020 con 0x7f342409c970 2020-11-20T12:11:48.867 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.849+0000 7f341bfff700 1 --2- 172.21.15.189:0/630045826 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f34000557c0 0x7f3400057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:48.867 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.849+0000 7f3423fff700 1 --2- 172.21.15.189:0/630045826 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f34000557c0 0x7f3400057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:48.868 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.849+0000 7f341bfff700 1 -- 172.21.15.189:0/630045826 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f341406dd00 con 0x7f342409c970 2020-11-20T12:11:48.868 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.849+0000 7f3423fff700 1 --2- 172.21.15.189:0/630045826 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f34000557c0 0x7f3400057c80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f3424065f60 tx=0x7f340c006df0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:48.868 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:48.853+0000 7f341bfff700 1 -- 172.21.15.189:0/630045826 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3414071030 con 0x7f342409c970 2020-11-20T12:11:49.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.089+0000 7f342a930700 1 -- 172.21.15.189:0/630045826 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f342409ead0 con 0x7f342409c970 2020-11-20T12:11:49.096 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.093+0000 7f341bfff700 1 -- 172.21.15.189:0/630045826 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f3414043a40 con 0x7f342409c970 2020-11-20T12:11:49.097 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:11:49.097 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5122,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/4088050649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4088050649},{"type":"v1","addr":"172.21.15.47:6833","nonce":4088050649}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:11:15.620495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5053},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5053":{"gid":5053,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/3540904386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3540904386},{"type":"v1","addr":"172.21.15.189:6835","nonce":3540904386}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:11:49.098 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.093+0000 7f342a930700 1 -- 172.21.15.189:0/630045826 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f34000557c0 msgr2=0x7f3400057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.099 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.093+0000 7f342a930700 1 --2- 172.21.15.189:0/630045826 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f34000557c0 0x7f3400057c80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f3424065f60 tx=0x7f340c006df0).stop 2020-11-20T12:11:49.099 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.093+0000 7f342a930700 1 -- 172.21.15.189:0/630045826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f342409c970 msgr2=0x7f34240a42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.099 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.093+0000 7f342a930700 1 --2- 172.21.15.189:0/630045826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f342409c970 0x7f34240a42a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f341400ea80 tx=0x7f3414001620).stop 2020-11-20T12:11:49.099 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.093+0000 7f342a930700 1 -- 172.21.15.189:0/630045826 shutdown_connections 2020-11-20T12:11:49.100 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.093+0000 7f342a930700 1 --2- 172.21.15.189:0/630045826 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f34000557c0 0x7f3400057c80 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.100 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.093+0000 7f342a930700 1 --2- 172.21.15.189:0/630045826 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f34241460b0 0x7f3424146530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.100 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.093+0000 7f342a930700 1 --2- 172.21.15.189:0/630045826 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3404004e50 0x7f342409e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.100 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.093+0000 7f342a930700 1 --2- 172.21.15.189:0/630045826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f342409c970 0x7f34240a42a0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.100 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.093+0000 7f342a930700 1 -- 172.21.15.189:0/630045826 >> 172.21.15.189:0/630045826 conn(0x7f34240625a0 msgr2=0x7f3424063410 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:49.101 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.093+0000 7f342a930700 1 -- 172.21.15.189:0/630045826 shutdown_connections 2020-11-20T12:11:49.101 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.093+0000 7f342a930700 1 -- 172.21.15.189:0/630045826 wait complete. 2020-11-20T12:11:49.101 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 26 2020-11-20T12:11:49.112 INFO:tasks.mds_thrash.fs.[None]:kill mds.a (rank=0) 2020-11-20T12:11:49.113 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T12:11:49.113 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:11:49.128 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T12:11:49.128 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:49.296 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f015359e700 1 Processor -- start 2020-11-20T12:11:49.297 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f015359e700 1 -- start start 2020-11-20T12:11:49.298 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f015359e700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0154060310 0x7f01540a1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.299 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f015359e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f01540a2440 0x7f01540a7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.299 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f015359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01540a8fa0 con 0x7f01540a8230 2020-11-20T12:11:49.299 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f015359e700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01540618b0 con 0x7f0154060310 2020-11-20T12:11:49.299 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f015359e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01540a9240 con 0x7f01540a2440 2020-11-20T12:11:49.299 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f0152d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f01540a8230 0x7f0154135110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51776/0 (socket says 172.21.15.189:51776) 2020-11-20T12:11:49.300 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f0152d9d700 1 -- 172.21.15.189:0/226556917 learned_addr learned my addr 172.21.15.189:0/226556917 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:11:49.300 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f015259c700 1 --2- 172.21.15.189:0/226556917 >> v2:172.21.15.47:3300/0 conn(0x7f0154060310 0x7f01540a1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.300 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f0151d9b700 1 --2- 172.21.15.189:0/226556917 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f01540a2440 0x7f01540a7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.300 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f015159a700 1 -- 172.21.15.189:0/226556917 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3758979145 0 0) 0x7f01540a8fa0 con 0x7f01540a8230 2020-11-20T12:11:49.301 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f015159a700 1 -- 172.21.15.189:0/226556917 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0138003240 con 0x7f01540a8230 2020-11-20T12:11:49.301 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f015259c700 1 -- 172.21.15.189:0/226556917 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f01540a2440 msgr2=0x7f01540a7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:49.301 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f015259c700 1 --2- 172.21.15.189:0/226556917 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f01540a2440 0x7f01540a7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.301 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f015259c700 1 -- 172.21.15.189:0/226556917 >> v1:172.21.15.189:6789/0 conn(0x7f01540a8230 legacy=0x7f0154135110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.301 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f015259c700 1 -- 172.21.15.189:0/226556917 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01540a94e0 con 0x7f0154060310 2020-11-20T12:11:49.302 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f015259c700 1 --2- 172.21.15.189:0/226556917 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0154060310 0x7f01540a1f50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f014400b9a0 tx=0x7f0144006df0).ready entity=mon.0 client_cookie=b24a802a29214e36 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:49.302 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f015159a700 1 -- 172.21.15.189:0/226556917 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0144009680 con 0x7f0154060310 2020-11-20T12:11:49.302 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f015159a700 1 -- 172.21.15.189:0/226556917 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01440097e0 con 0x7f0154060310 2020-11-20T12:11:49.302 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f015159a700 1 -- 172.21.15.189:0/226556917 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0144009b60 con 0x7f0154060310 2020-11-20T12:11:49.305 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f015359e700 1 -- 172.21.15.189:0/226556917 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0154060310 msgr2=0x7f01540a1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.305 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f015359e700 1 --2- 172.21.15.189:0/226556917 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0154060310 0x7f01540a1f50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f014400b9a0 tx=0x7f0144006df0).stop 2020-11-20T12:11:49.305 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f015359e700 1 -- 172.21.15.189:0/226556917 shutdown_connections 2020-11-20T12:11:49.306 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f015359e700 1 --2- 172.21.15.189:0/226556917 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0154060310 0x7f01540a1f50 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.306 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f015359e700 1 --2- 172.21.15.189:0/226556917 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f01540a2440 0x7f01540a7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.306 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f015359e700 1 -- 172.21.15.189:0/226556917 >> 172.21.15.189:0/226556917 conn(0x7f0154063210 msgr2=0x7f0154060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:49.306 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f015359e700 1 -- 172.21.15.189:0/226556917 shutdown_connections 2020-11-20T12:11:49.307 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.293+0000 7f015359e700 1 -- 172.21.15.189:0/226556917 wait complete. 2020-11-20T12:11:49.307 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.297+0000 7f015359e700 1 Processor -- start 2020-11-20T12:11:49.307 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.297+0000 7f015359e700 1 -- start start 2020-11-20T12:11:49.307 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.297+0000 7f015359e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0154060310 0x7f015409c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.307 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.297+0000 7f015359e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01540a2440 0x7f015414ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.308 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.297+0000 7f015359e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01540a8230 0x7f015413aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.308 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.297+0000 7f015359e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0154139b60 con 0x7f01540a8230 2020-11-20T12:11:49.308 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.297+0000 7f015359e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0154139d90 con 0x7f01540a2440 2020-11-20T12:11:49.308 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.297+0000 7f015359e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015413a030 con 0x7f0154060310 2020-11-20T12:11:49.308 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.297+0000 7f0151d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01540a2440 0x7f015414ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.309 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.297+0000 7f0151d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01540a2440 0x7f015414ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50158/0 (socket says 172.21.15.189:50158) 2020-11-20T12:11:49.309 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.297+0000 7f0151d9b700 1 -- 172.21.15.189:0/998258864 learned_addr learned my addr 172.21.15.189:0/998258864 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:11:49.310 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.297+0000 7f015259c700 1 --2- 172.21.15.189:0/998258864 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0154060310 0x7f015409c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.310 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.297+0000 7f0152d9d700 1 --2- 172.21.15.189:0/998258864 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01540a8230 0x7f015413aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.297+0000 7f0151d9b700 1 -- 172.21.15.189:0/998258864 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0154060310 msgr2=0x7f015409c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.297+0000 7f0151d9b700 1 --2- 172.21.15.189:0/998258864 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0154060310 0x7f015409c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.297+0000 7f0151d9b700 1 -- 172.21.15.189:0/998258864 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01540a8230 msgr2=0x7f015413aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.297+0000 7f0151d9b700 1 --2- 172.21.15.189:0/998258864 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01540a8230 0x7f015413aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.297+0000 7f0151d9b700 1 -- 172.21.15.189:0/998258864 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015413a2d0 con 0x7f01540a2440 2020-11-20T12:11:49.312 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.297+0000 7f0151d9b700 1 --2- 172.21.15.189:0/998258864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01540a2440 0x7f015414ab00 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f014800ebd0 tx=0x7f0148006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:49.312 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.297+0000 7f01427fc700 1 -- 172.21.15.189:0/998258864 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014800c8e0 con 0x7f01540a2440 2020-11-20T12:11:49.312 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.297+0000 7f01427fc700 1 -- 172.21.15.189:0/998258864 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014800ca40 con 0x7f01540a2440 2020-11-20T12:11:49.312 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.297+0000 7f01427fc700 1 -- 172.21.15.189:0/998258864 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014800cdc0 con 0x7f01540a2440 2020-11-20T12:11:49.312 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.297+0000 7f015359e700 1 -- 172.21.15.189:0/998258864 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f015413a770 con 0x7f01540a2440 2020-11-20T12:11:49.313 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.297+0000 7f015359e700 1 -- 172.21.15.189:0/998258864 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f015414b210 con 0x7f01540a2440 2020-11-20T12:11:49.313 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.301+0000 7f01427fc700 1 -- 172.21.15.189:0/998258864 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f0148012600 con 0x7f01540a2440 2020-11-20T12:11:49.313 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.301+0000 7f01427fc700 1 --2- 172.21.15.189:0/998258864 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f01280557c0 0x7f0128057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:49.313 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.301+0000 7f01427fc700 1 -- 172.21.15.189:0/998258864 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f014806eb00 con 0x7f01540a2440 2020-11-20T12:11:49.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.301+0000 7f015259c700 1 --2- 172.21.15.189:0/998258864 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f01280557c0 0x7f0128057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:49.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.301+0000 7f015359e700 1 -- 172.21.15.189:0/998258864 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01540446d0 con 0x7f01540a2440 2020-11-20T12:11:49.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.301+0000 7f015259c700 1 --2- 172.21.15.189:0/998258864 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f01280557c0 0x7f0128057c80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f014400c010 tx=0x7f0144015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:49.315 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.305+0000 7f01427fc700 1 -- 172.21.15.189:0/998258864 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f014806e070 con 0x7f01540a2440 2020-11-20T12:11:49.527 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.521+0000 7f015359e700 1 -- 172.21.15.189:0/998258864 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f015405cdd0 con 0x7f01540a2440 2020-11-20T12:11:49.528 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.525+0000 7f01427fc700 1 -- 172.21.15.189:0/998258864 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f0148044840 con 0x7f01540a2440 2020-11-20T12:11:49.529 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:11:49.529 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5122,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/4088050649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4088050649},{"type":"v1","addr":"172.21.15.47:6833","nonce":4088050649}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:11:15.620495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5053},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5053":{"gid":5053,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/3540904386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3540904386},{"type":"v1","addr":"172.21.15.189:6835","nonce":3540904386}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:11:49.530 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.525+0000 7f015359e700 1 -- 172.21.15.189:0/998258864 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f01280557c0 msgr2=0x7f0128057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.531 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.525+0000 7f015359e700 1 --2- 172.21.15.189:0/998258864 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f01280557c0 0x7f0128057c80 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f014400c010 tx=0x7f0144015000).stop 2020-11-20T12:11:49.531 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.525+0000 7f015359e700 1 -- 172.21.15.189:0/998258864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01540a2440 msgr2=0x7f015414ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:49.531 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.525+0000 7f015359e700 1 --2- 172.21.15.189:0/998258864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01540a2440 0x7f015414ab00 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f014800ebd0 tx=0x7f0148006df0).stop 2020-11-20T12:11:49.531 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.525+0000 7f015359e700 1 -- 172.21.15.189:0/998258864 shutdown_connections 2020-11-20T12:11:49.531 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.525+0000 7f015359e700 1 --2- 172.21.15.189:0/998258864 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f01280557c0 0x7f0128057c80 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.532 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.525+0000 7f015359e700 1 --2- 172.21.15.189:0/998258864 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f01540a8230 0x7f015413aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.532 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.525+0000 7f015359e700 1 --2- 172.21.15.189:0/998258864 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0154060310 0x7f015409c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.532 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.525+0000 7f015359e700 1 --2- 172.21.15.189:0/998258864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01540a2440 0x7f015414ab00 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:49.532 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.525+0000 7f015359e700 1 -- 172.21.15.189:0/998258864 >> 172.21.15.189:0/998258864 conn(0x7f0154063210 msgr2=0x7f01540ab0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:49.533 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.525+0000 7f015359e700 1 -- 172.21.15.189:0/998258864 shutdown_connections 2020-11-20T12:11:49.533 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:49.525+0000 7f015359e700 1 -- 172.21.15.189:0/998258864 wait complete. 2020-11-20T12:11:49.533 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 26 2020-11-20T12:11:49.543 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:11:51.544 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:51.673 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.669+0000 7fb7cea2f700 1 Processor -- start 2020-11-20T12:11:51.673 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.669+0000 7fb7cea2f700 1 -- start start 2020-11-20T12:11:51.675 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.669+0000 7fb7cea2f700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb7c8066de0 0x7fb7c813ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:51.676 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.669+0000 7fb7cea2f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb7c80671c0 0x7fb7c813e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:51.676 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.669+0000 7fb7cea2f700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb7c813e6c0 0x7fb7c813cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:51.676 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.669+0000 7fb7cea2f700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c813b070 con 0x7fb7c813e6c0 2020-11-20T12:11:51.676 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.669+0000 7fb7cea2f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c813b300 con 0x7fb7c80671c0 2020-11-20T12:11:51.677 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.669+0000 7fb7cea2f700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c813dd30 con 0x7fb7c8066de0 2020-11-20T12:11:51.677 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.669+0000 7fb7c77fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb7c80671c0 0x7fb7c813e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:51.677 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.669+0000 7fb7c77fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb7c80671c0 0x7fb7c813e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50172/0 (socket says 172.21.15.189:50172) 2020-11-20T12:11:51.677 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.669+0000 7fb7c77fe700 1 -- 172.21.15.189:0/3370560248 learned_addr learned my addr 172.21.15.189:0/3370560248 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:11:51.677 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.669+0000 7fb7c7fff700 1 --2- 172.21.15.189:0/3370560248 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb7c8066de0 0x7fb7c813ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:51.678 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.669+0000 7fb7ccfcc700 1 --2- 172.21.15.189:0/3370560248 >> v2:172.21.15.47:3300/0 conn(0x7fb7c813e6c0 0x7fb7c813cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:51.678 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.669+0000 7fb7c77fe700 1 -- 172.21.15.189:0/3370560248 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb7c8066de0 msgr2=0x7fb7c813ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:51.678 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.669+0000 7fb7c77fe700 1 --2- 172.21.15.189:0/3370560248 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb7c8066de0 0x7fb7c813ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:51.678 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.669+0000 7fb7c77fe700 1 -- 172.21.15.189:0/3370560248 >> v2:172.21.15.47:3300/0 conn(0x7fb7c813e6c0 msgr2=0x7fb7c813cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:51.679 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.669+0000 7fb7c77fe700 1 --2- 172.21.15.189:0/3370560248 >> v2:172.21.15.47:3300/0 conn(0x7fb7c813e6c0 0x7fb7c813cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:51.679 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.669+0000 7fb7c77fe700 1 -- 172.21.15.189:0/3370560248 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c8067620 con 0x7fb7c80671c0 2020-11-20T12:11:51.679 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.669+0000 7fb7c77fe700 1 --2- 172.21.15.189:0/3370560248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7c80671c0 0x7fb7c813e1d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fb7c805c7d0 tx=0x7fb7b0006eb0).ready entity=mon.1 client_cookie=30a2f654f32968c2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:51.679 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.669+0000 7fb7c6ffd700 1 -- 172.21.15.189:0/3370560248 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b00097b0 con 0x7fb7c80671c0 2020-11-20T12:11:51.680 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.669+0000 7fb7c6ffd700 1 -- 172.21.15.189:0/3370560248 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b0009910 con 0x7fb7c80671c0 2020-11-20T12:11:51.680 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.669+0000 7fb7c6ffd700 1 -- 172.21.15.189:0/3370560248 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0009c90 con 0x7fb7c80671c0 2020-11-20T12:11:51.682 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7cea2f700 1 -- 172.21.15.189:0/3370560248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7c80671c0 msgr2=0x7fb7c813e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:51.682 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7cea2f700 1 --2- 172.21.15.189:0/3370560248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7c80671c0 0x7fb7c813e1d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fb7c805c7d0 tx=0x7fb7b0006eb0).stop 2020-11-20T12:11:51.683 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7cea2f700 1 -- 172.21.15.189:0/3370560248 shutdown_connections 2020-11-20T12:11:51.683 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7cea2f700 1 --2- 172.21.15.189:0/3370560248 >> v2:172.21.15.47:3300/0 conn(0x7fb7c813e6c0 0x7fb7c813cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:51.683 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7cea2f700 1 --2- 172.21.15.189:0/3370560248 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb7c8066de0 0x7fb7c813ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:51.683 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7cea2f700 1 --2- 172.21.15.189:0/3370560248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7c80671c0 0x7fb7c813e1d0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:51.683 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7cea2f700 1 -- 172.21.15.189:0/3370560248 >> 172.21.15.189:0/3370560248 conn(0x7fb7c8061f60 msgr2=0x7fb7c8062340 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:51.684 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7cea2f700 1 -- 172.21.15.189:0/3370560248 shutdown_connections 2020-11-20T12:11:51.684 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7cea2f700 1 -- 172.21.15.189:0/3370560248 wait complete. 2020-11-20T12:11:51.684 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7cea2f700 1 Processor -- start 2020-11-20T12:11:51.684 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7cea2f700 1 -- start start 2020-11-20T12:11:51.685 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7cea2f700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb7c8066de0 0x7fb7c8149c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:51.685 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7cea2f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb7c813e6c0 0x7fb7c80a3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:51.685 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7cea2f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7c8138a50 0x7fb7c8138e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:51.685 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7cea2f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c80a0960 con 0x7fb7c813e6c0 2020-11-20T12:11:51.686 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7cea2f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c814a6a0 con 0x7fb7c8138a50 2020-11-20T12:11:51.686 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7cea2f700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c814a1a0 con 0x7fb7c8066de0 2020-11-20T12:11:51.686 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7ccfcc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7c8138a50 0x7fb7c8138e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:51.686 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7ccfcc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7c8138a50 0x7fb7c8138e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50180/0 (socket says 172.21.15.189:50180) 2020-11-20T12:11:51.686 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7ccfcc700 1 -- 172.21.15.189:0/4082355970 learned_addr learned my addr 172.21.15.189:0/4082355970 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:11:51.687 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7c77fe700 1 --2- 172.21.15.189:0/4082355970 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb7c813e6c0 0x7fb7c80a3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:51.687 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7c7fff700 1 --2- 172.21.15.189:0/4082355970 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb7c8066de0 0x7fb7c8149c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:51.687 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7ccfcc700 1 -- 172.21.15.189:0/4082355970 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb7c8066de0 msgr2=0x7fb7c8149c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:51.687 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7ccfcc700 1 --2- 172.21.15.189:0/4082355970 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb7c8066de0 0x7fb7c8149c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:51.687 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7ccfcc700 1 -- 172.21.15.189:0/4082355970 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb7c813e6c0 msgr2=0x7fb7c80a3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:51.688 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7ccfcc700 1 --2- 172.21.15.189:0/4082355970 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb7c813e6c0 0x7fb7c80a3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:51.688 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.673+0000 7fb7ccfcc700 1 -- 172.21.15.189:0/4082355970 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c814a400 con 0x7fb7c8138a50 2020-11-20T12:11:51.688 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.677+0000 7fb7ccfcc700 1 --2- 172.21.15.189:0/4082355970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7c8138a50 0x7fb7c8138e90 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fb7c000eba0 tx=0x7fb7c0006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:51.688 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.677+0000 7fb7bffff700 1 -- 172.21.15.189:0/4082355970 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c000b8a0 con 0x7fb7c8138a50 2020-11-20T12:11:51.688 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.677+0000 7fb7cea2f700 1 -- 172.21.15.189:0/4082355970 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c8139380 con 0x7fb7c8138a50 2020-11-20T12:11:51.691 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.677+0000 7fb7bffff700 1 -- 172.21.15.189:0/4082355970 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c000ba00 con 0x7fb7c8138a50 2020-11-20T12:11:51.691 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.677+0000 7fb7bffff700 1 -- 172.21.15.189:0/4082355970 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c000bda0 con 0x7fb7c8138a50 2020-11-20T12:11:51.691 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.677+0000 7fb7cea2f700 1 -- 172.21.15.189:0/4082355970 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c80a4220 con 0x7fb7c8138a50 2020-11-20T12:11:51.692 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.677+0000 7fb7cea2f700 1 -- 172.21.15.189:0/4082355970 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c8138780 con 0x7fb7c8138a50 2020-11-20T12:11:51.693 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.677+0000 7fb7bffff700 1 -- 172.21.15.189:0/4082355970 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fb7c0012600 con 0x7fb7c8138a50 2020-11-20T12:11:51.693 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.681+0000 7fb7bffff700 1 --2- 172.21.15.189:0/4082355970 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb7a40557f0 0x7fb7a4057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:51.694 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.681+0000 7fb7bffff700 1 -- 172.21.15.189:0/4082355970 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fb7c006f5c0 con 0x7fb7c8138a50 2020-11-20T12:11:51.694 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.681+0000 7fb7c7fff700 1 --2- 172.21.15.189:0/4082355970 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb7a40557f0 0x7fb7a4057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:51.694 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.681+0000 7fb7c7fff700 1 --2- 172.21.15.189:0/4082355970 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb7a40557f0 0x7fb7a4057cb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fb7b8009e00 tx=0x7fb7b80091d0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:51.694 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.685+0000 7fb7bffff700 1 -- 172.21.15.189:0/4082355970 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb7c0045300 con 0x7fb7c8138a50 2020-11-20T12:11:51.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.885+0000 7fb7cea2f700 1 -- 172.21.15.189:0/4082355970 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7c8044f60 con 0x7fb7c8138a50 2020-11-20T12:11:51.894 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.889+0000 7fb7bffff700 1 -- 172.21.15.189:0/4082355970 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fb7c0048920 con 0x7fb7c8138a50 2020-11-20T12:11:51.894 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:11:51.895 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5122,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/4088050649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4088050649},{"type":"v1","addr":"172.21.15.47:6833","nonce":4088050649}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:11:15.620495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5053},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5053":{"gid":5053,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/3540904386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3540904386},{"type":"v1","addr":"172.21.15.189:6835","nonce":3540904386}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:11:51.897 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.889+0000 7fb7cea2f700 1 -- 172.21.15.189:0/4082355970 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb7a40557f0 msgr2=0x7fb7a4057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:51.897 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.889+0000 7fb7cea2f700 1 --2- 172.21.15.189:0/4082355970 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb7a40557f0 0x7fb7a4057cb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fb7b8009e00 tx=0x7fb7b80091d0).stop 2020-11-20T12:11:51.897 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.889+0000 7fb7cea2f700 1 -- 172.21.15.189:0/4082355970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7c8138a50 msgr2=0x7fb7c8138e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:51.898 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.889+0000 7fb7cea2f700 1 --2- 172.21.15.189:0/4082355970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7c8138a50 0x7fb7c8138e90 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fb7c000eba0 tx=0x7fb7c0006df0).stop 2020-11-20T12:11:51.898 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.889+0000 7fb7cea2f700 1 -- 172.21.15.189:0/4082355970 shutdown_connections 2020-11-20T12:11:51.898 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.889+0000 7fb7cea2f700 1 --2- 172.21.15.189:0/4082355970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7c8138a50 0x7fb7c8138e90 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:51.899 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.889+0000 7fb7cea2f700 1 --2- 172.21.15.189:0/4082355970 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb7c8066de0 0x7fb7c8149c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:51.899 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.889+0000 7fb7cea2f700 1 --2- 172.21.15.189:0/4082355970 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb7a40557f0 0x7fb7a4057cb0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:51.900 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.889+0000 7fb7cea2f700 1 --2- 172.21.15.189:0/4082355970 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb7c813e6c0 0x7fb7c80a3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:51.901 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.889+0000 7fb7cea2f700 1 -- 172.21.15.189:0/4082355970 >> 172.21.15.189:0/4082355970 conn(0x7fb7c8061f60 msgr2=0x7fb7c813bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:51.901 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.889+0000 7fb7cea2f700 1 -- 172.21.15.189:0/4082355970 shutdown_connections 2020-11-20T12:11:51.901 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:51.889+0000 7fb7cea2f700 1 -- 172.21.15.189:0/4082355970 wait complete. 2020-11-20T12:11:51.901 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 26 2020-11-20T12:11:51.908 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:11:53.508 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:11:53.515 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:11:53.909 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:54.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.044+0000 7fb36e3e4700 1 Processor -- start 2020-11-20T12:11:54.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.044+0000 7fb36e3e4700 1 -- start start 2020-11-20T12:11:54.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.044+0000 7fb36e3e4700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb3680ad3e0 0x7fb368066d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:54.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.044+0000 7fb36e3e4700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb36813a6e0 0x7fb36813ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:54.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.044+0000 7fb36e3e4700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb368067730 con 0x7fb3680ad000 2020-11-20T12:11:54.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.044+0000 7fb36e3e4700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb368066050 con 0x7fb36813a6e0 2020-11-20T12:11:54.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.044+0000 7fb36e3e4700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb368138190 con 0x7fb3680ad3e0 2020-11-20T12:11:54.052 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.044+0000 7fb36d3e2700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb3680ad000 0x7fb36813e480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51808/0 (socket says 172.21.15.189:51808) 2020-11-20T12:11:54.052 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.044+0000 7fb36d3e2700 1 -- 172.21.15.189:0/4233950896 learned_addr learned my addr 172.21.15.189:0/4233950896 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:11:54.052 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.044+0000 7fb36dbe3700 1 --2- 172.21.15.189:0/4233950896 >> v2:172.21.15.189:3300/0 conn(0x7fb36813a6e0 0x7fb36813ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:54.052 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.044+0000 7fb36cbe1700 1 --2- 172.21.15.189:0/4233950896 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb3680ad3e0 0x7fb368066d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:54.053 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.044+0000 7fb36dbe3700 1 -- 172.21.15.189:0/4233950896 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb3680ad3e0 msgr2=0x7fb368066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:54.056 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.044+0000 7fb36dbe3700 1 --2- 172.21.15.189:0/4233950896 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb3680ad3e0 0x7fb368066d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:54.056 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.044+0000 7fb36dbe3700 1 -- 172.21.15.189:0/4233950896 >> v1:172.21.15.189:6789/0 conn(0x7fb3680ad000 legacy=0x7fb36813e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:54.057 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.044+0000 7fb36dbe3700 1 -- 172.21.15.189:0/4233950896 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36813dd40 con 0x7fb36813a6e0 2020-11-20T12:11:54.057 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.044+0000 7fb35ffff700 1 -- 172.21.15.189:0/4233950896 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2518157150 0 0) 0x7fb368067730 con 0x7fb3680ad000 2020-11-20T12:11:54.057 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.044+0000 7fb36cbe1700 1 --2- 172.21.15.189:0/4233950896 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb3680ad3e0 0x7fb368066d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:11:54.057 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.044+0000 7fb36dbe3700 1 --2- 172.21.15.189:0/4233950896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb36813a6e0 0x7fb36813ad00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fb36400b9d0 tx=0x7fb364006df0).ready entity=mon.1 client_cookie=cf4c29b7e5609c50 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:54.058 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.044+0000 7fb35ffff700 1 -- 172.21.15.189:0/4233950896 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb364009800 con 0x7fb36813a6e0 2020-11-20T12:11:54.058 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.044+0000 7fb35ffff700 1 -- 172.21.15.189:0/4233950896 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb364009960 con 0x7fb36813a6e0 2020-11-20T12:11:54.058 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.044+0000 7fb35ffff700 1 -- 172.21.15.189:0/4233950896 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb364009ce0 con 0x7fb36813a6e0 2020-11-20T12:11:54.058 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.044+0000 7fb36e3e4700 1 -- 172.21.15.189:0/4233950896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb36813a6e0 msgr2=0x7fb36813ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:54.058 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.044+0000 7fb36e3e4700 1 --2- 172.21.15.189:0/4233950896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb36813a6e0 0x7fb36813ad00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fb36400b9d0 tx=0x7fb364006df0).stop 2020-11-20T12:11:54.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.048+0000 7fb36e3e4700 1 -- 172.21.15.189:0/4233950896 shutdown_connections 2020-11-20T12:11:54.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.048+0000 7fb36e3e4700 1 --2- 172.21.15.189:0/4233950896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb36813a6e0 0x7fb36813ad00 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:54.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.048+0000 7fb36e3e4700 1 --2- 172.21.15.189:0/4233950896 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb3680ad3e0 0x7fb368066d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:54.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.048+0000 7fb36e3e4700 1 -- 172.21.15.189:0/4233950896 >> 172.21.15.189:0/4233950896 conn(0x7fb368062080 msgr2=0x7fb368062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:54.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.048+0000 7fb36e3e4700 1 -- 172.21.15.189:0/4233950896 shutdown_connections 2020-11-20T12:11:54.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.048+0000 7fb36e3e4700 1 -- 172.21.15.189:0/4233950896 wait complete. 2020-11-20T12:11:54.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.048+0000 7fb36e3e4700 1 Processor -- start 2020-11-20T12:11:54.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.048+0000 7fb36e3e4700 1 -- start start 2020-11-20T12:11:54.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.048+0000 7fb36e3e4700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3680ad000 0x7fb368140820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:54.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.048+0000 7fb36e3e4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3680ad3e0 0x7fb3680aa860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:54.061 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.048+0000 7fb36e3e4700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb3680a9480 0x7fb3680a98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:54.061 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.048+0000 7fb36e3e4700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3680a5bc0 con 0x7fb3680ad000 2020-11-20T12:11:54.061 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.048+0000 7fb36e3e4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb368141360 con 0x7fb3680ad3e0 2020-11-20T12:11:54.061 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.048+0000 7fb36e3e4700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3680a5db0 con 0x7fb3680a9480 2020-11-20T12:11:54.061 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.048+0000 7fb36cbe1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3680ad3e0 0x7fb3680aa860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:54.062 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.048+0000 7fb36cbe1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3680ad3e0 0x7fb3680aa860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50194/0 (socket says 172.21.15.189:50194) 2020-11-20T12:11:54.062 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.048+0000 7fb36cbe1700 1 -- 172.21.15.189:0/3475425424 learned_addr learned my addr 172.21.15.189:0/3475425424 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:11:54.062 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.048+0000 7fb36d3e2700 1 --2- 172.21.15.189:0/3475425424 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3680ad000 0x7fb368140820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:54.062 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.048+0000 7fb36dbe3700 1 --2- 172.21.15.189:0/3475425424 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb3680a9480 0x7fb3680a98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:54.062 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.048+0000 7fb36cbe1700 1 -- 172.21.15.189:0/3475425424 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb3680a9480 msgr2=0x7fb3680a98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:54.063 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.048+0000 7fb36cbe1700 1 --2- 172.21.15.189:0/3475425424 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb3680a9480 0x7fb3680a98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:54.063 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.048+0000 7fb36cbe1700 1 -- 172.21.15.189:0/3475425424 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3680ad000 msgr2=0x7fb368140820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:54.063 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.048+0000 7fb36cbe1700 1 --2- 172.21.15.189:0/3475425424 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3680ad000 0x7fb368140820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:54.065 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.048+0000 7fb36cbe1700 1 -- 172.21.15.189:0/3475425424 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3680a6050 con 0x7fb3680ad3e0 2020-11-20T12:11:54.065 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.048+0000 7fb36cbe1700 1 --2- 172.21.15.189:0/3475425424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3680ad3e0 0x7fb3680aa860 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fb35800ef80 tx=0x7fb358006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:54.065 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.048+0000 7fb35d7fa700 1 -- 172.21.15.189:0/3475425424 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35800cd00 con 0x7fb3680ad3e0 2020-11-20T12:11:54.066 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.052+0000 7fb35d7fa700 1 -- 172.21.15.189:0/3475425424 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35800ce60 con 0x7fb3680ad3e0 2020-11-20T12:11:54.066 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.052+0000 7fb36e3e4700 1 -- 172.21.15.189:0/3475425424 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3680a62f0 con 0x7fb3680ad3e0 2020-11-20T12:11:54.066 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.052+0000 7fb35d7fa700 1 -- 172.21.15.189:0/3475425424 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb358012600 con 0x7fb3680ad3e0 2020-11-20T12:11:54.068 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.052+0000 7fb36e3e4700 1 -- 172.21.15.189:0/3475425424 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3680aaf70 con 0x7fb3680ad3e0 2020-11-20T12:11:54.068 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.052+0000 7fb35d7fa700 1 -- 172.21.15.189:0/3475425424 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fb3580129b0 con 0x7fb3680ad3e0 2020-11-20T12:11:54.068 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.052+0000 7fb36e3e4700 1 -- 172.21.15.189:0/3475425424 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3680a92f0 con 0x7fb3680ad3e0 2020-11-20T12:11:54.068 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.052+0000 7fb35d7fa700 1 --2- 172.21.15.189:0/3475425424 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb3400557f0 0x7fb340057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:54.068 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.052+0000 7fb35d7fa700 1 -- 172.21.15.189:0/3475425424 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6417+0+0 (secure 0 0 0) 0x7fb35806ee50 con 0x7fb3680ad3e0 2020-11-20T12:11:54.069 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.052+0000 7fb36d3e2700 1 --2- 172.21.15.189:0/3475425424 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb3400557f0 0x7fb340057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:54.069 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.052+0000 7fb36d3e2700 1 --2- 172.21.15.189:0/3475425424 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb3400557f0 0x7fb340057cb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fb360001520 tx=0x7fb360009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:54.069 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.056+0000 7fb35d7fa700 1 -- 172.21.15.189:0/3475425424 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb358044a80 con 0x7fb3680ad3e0 2020-11-20T12:11:54.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.256+0000 7fb36e3e4700 1 -- 172.21.15.189:0/3475425424 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3680aaf70 con 0x7fb3680ad3e0 2020-11-20T12:11:54.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.256+0000 7fb35d7fa700 1 -- 172.21.15.189:0/3475425424 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fb3680aaf70 con 0x7fb3680ad3e0 2020-11-20T12:11:54.264 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:11:54.265 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5122,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/4088050649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4088050649},{"type":"v1","addr":"172.21.15.47:6833","nonce":4088050649}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:11:15.620495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5053},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5053":{"gid":5053,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/3540904386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3540904386},{"type":"v1","addr":"172.21.15.189:6835","nonce":3540904386}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:11:54.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.260+0000 7fb36e3e4700 1 -- 172.21.15.189:0/3475425424 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb3400557f0 msgr2=0x7fb340057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:54.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.260+0000 7fb36e3e4700 1 --2- 172.21.15.189:0/3475425424 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb3400557f0 0x7fb340057cb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fb360001520 tx=0x7fb360009040).stop 2020-11-20T12:11:54.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.260+0000 7fb36e3e4700 1 -- 172.21.15.189:0/3475425424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3680ad3e0 msgr2=0x7fb3680aa860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:54.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.260+0000 7fb36e3e4700 1 --2- 172.21.15.189:0/3475425424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3680ad3e0 0x7fb3680aa860 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fb35800ef80 tx=0x7fb358006df0).stop 2020-11-20T12:11:54.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.260+0000 7fb36e3e4700 1 -- 172.21.15.189:0/3475425424 shutdown_connections 2020-11-20T12:11:54.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.260+0000 7fb36e3e4700 1 --2- 172.21.15.189:0/3475425424 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb3400557f0 0x7fb340057cb0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:54.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.260+0000 7fb36e3e4700 1 --2- 172.21.15.189:0/3475425424 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb3680ad000 0x7fb368140820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:54.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.260+0000 7fb36e3e4700 1 --2- 172.21.15.189:0/3475425424 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb3680a9480 0x7fb3680a98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:54.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.260+0000 7fb36e3e4700 1 --2- 172.21.15.189:0/3475425424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3680ad3e0 0x7fb3680aa860 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:54.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.260+0000 7fb36e3e4700 1 -- 172.21.15.189:0/3475425424 >> 172.21.15.189:0/3475425424 conn(0x7fb368062080 msgr2=0x7fb36805ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:54.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.260+0000 7fb36e3e4700 1 -- 172.21.15.189:0/3475425424 shutdown_connections 2020-11-20T12:11:54.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:54.260+0000 7fb36e3e4700 1 -- 172.21.15.189:0/3475425424 wait complete. 2020-11-20T12:11:54.269 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 26 2020-11-20T12:11:54.279 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:11:56.280 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:56.413 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.408+0000 7f3ad347e700 1 Processor -- start 2020-11-20T12:11:56.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.408+0000 7f3ad347e700 1 -- start start 2020-11-20T12:11:56.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.408+0000 7f3ad347e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3acc0612b0 0x7f3acc0a0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:56.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.408+0000 7f3ad347e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3acc067b50 0x7f3acc136fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:56.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.408+0000 7f3ad347e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3acc068020 con 0x7f3acc1374a0 2020-11-20T12:11:56.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.408+0000 7f3ad347e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3acc060880 con 0x7f3acc067b50 2020-11-20T12:11:56.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.408+0000 7f3ad347e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc0682c0 con 0x7f3acc0612b0 2020-11-20T12:11:56.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.408+0000 7f3ad1c7b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3acc067b50 0x7f3acc136fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:56.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.408+0000 7f3ad1c7b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3acc067b50 0x7f3acc136fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50204/0 (socket says 172.21.15.189:50204) 2020-11-20T12:11:56.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.408+0000 7f3ad247c700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3acc0612b0 0x7f3acc0a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:56.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.408+0000 7f3ad1c7b700 1 -- 172.21.15.189:0/2139931170 learned_addr learned my addr 172.21.15.189:0/2139931170 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:11:56.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.408+0000 7f3ad2c7d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3acc1374a0 0x7f3acc060170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51828/0 (socket says 172.21.15.189:51828) 2020-11-20T12:11:56.418 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.408+0000 7f3ad1c7b700 1 -- 172.21.15.189:0/2139931170 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3acc0612b0 msgr2=0x7f3acc0a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:56.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.408+0000 7f3ad1c7b700 1 --2- 172.21.15.189:0/2139931170 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3acc0612b0 0x7f3acc0a0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:56.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.408+0000 7f3ad1c7b700 1 -- 172.21.15.189:0/2139931170 >> v1:172.21.15.189:6789/0 conn(0x7f3acc1374a0 legacy=0x7f3acc060170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:56.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.408+0000 7f3ad1c7b700 1 -- 172.21.15.189:0/2139931170 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3acc1378f0 con 0x7f3acc067b50 2020-11-20T12:11:56.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.408+0000 7f3ad1c7b700 1 --2- 172.21.15.189:0/2139931170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3acc067b50 0x7f3acc136fb0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f3ac0006fd0 tx=0x7f3ac0006eb0).ready entity=mon.1 client_cookie=4e93e90166fe7622 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:56.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.408+0000 7f3ad147a700 1 -- 172.21.15.189:0/2139931170 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac0009810 con 0x7f3acc067b50 2020-11-20T12:11:56.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.408+0000 7f3ad147a700 1 -- 172.21.15.189:0/2139931170 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac0009970 con 0x7f3acc067b50 2020-11-20T12:11:56.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.408+0000 7f3ad147a700 1 -- 172.21.15.189:0/2139931170 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac0009cf0 con 0x7f3acc067b50 2020-11-20T12:11:56.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.408+0000 7f3ad347e700 1 -- 172.21.15.189:0/2139931170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3acc067b50 msgr2=0x7f3acc136fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:56.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.408+0000 7f3ad347e700 1 --2- 172.21.15.189:0/2139931170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3acc067b50 0x7f3acc136fb0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f3ac0006fd0 tx=0x7f3ac0006eb0).stop 2020-11-20T12:11:56.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.408+0000 7f3ad347e700 1 -- 172.21.15.189:0/2139931170 shutdown_connections 2020-11-20T12:11:56.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.408+0000 7f3ad347e700 1 --2- 172.21.15.189:0/2139931170 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3acc0612b0 0x7f3acc0a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:56.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.408+0000 7f3ad347e700 1 --2- 172.21.15.189:0/2139931170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3acc067b50 0x7f3acc136fb0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:56.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.408+0000 7f3ad347e700 1 -- 172.21.15.189:0/2139931170 >> 172.21.15.189:0/2139931170 conn(0x7f3acc063210 msgr2=0x7f3acc060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:56.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.412+0000 7f3ad347e700 1 -- 172.21.15.189:0/2139931170 shutdown_connections 2020-11-20T12:11:56.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.412+0000 7f3ad347e700 1 -- 172.21.15.189:0/2139931170 wait complete. 2020-11-20T12:11:56.424 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.412+0000 7f3ad347e700 1 Processor -- start 2020-11-20T12:11:56.424 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.412+0000 7f3ad347e700 1 -- start start 2020-11-20T12:11:56.425 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.416+0000 7f3ad347e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3acc0612b0 0x7f3acc13a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:56.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.416+0000 7f3ad347e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3acc067b50 0x7f3acc0a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:56.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.416+0000 7f3ad347e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3acc1374a0 0x7f3acc0a1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:56.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.416+0000 7f3ad347e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc139a30 con 0x7f3acc0612b0 2020-11-20T12:11:56.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.416+0000 7f3ad347e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc0a20b0 con 0x7f3acc1374a0 2020-11-20T12:11:56.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.416+0000 7f3ad347e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc0a1480 con 0x7f3acc067b50 2020-11-20T12:11:56.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.416+0000 7f3ad2c7d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3acc1374a0 0x7f3acc0a1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:56.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.416+0000 7f3ad2c7d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3acc1374a0 0x7f3acc0a1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50212/0 (socket says 172.21.15.189:50212) 2020-11-20T12:11:56.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.416+0000 7f3ad2c7d700 1 -- 172.21.15.189:0/3970621231 learned_addr learned my addr 172.21.15.189:0/3970621231 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:11:56.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.416+0000 7f3ad1c7b700 1 --2- 172.21.15.189:0/3970621231 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3acc067b50 0x7f3acc0a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:56.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.416+0000 7f3ad247c700 1 --2- 172.21.15.189:0/3970621231 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3acc0612b0 0x7f3acc13a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:56.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.416+0000 7f3ad2c7d700 1 -- 172.21.15.189:0/3970621231 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3acc067b50 msgr2=0x7f3acc0a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:56.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.416+0000 7f3ad2c7d700 1 --2- 172.21.15.189:0/3970621231 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3acc067b50 0x7f3acc0a36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:56.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.416+0000 7f3ad2c7d700 1 -- 172.21.15.189:0/3970621231 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3acc0612b0 msgr2=0x7f3acc13a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:56.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.416+0000 7f3ad2c7d700 1 --2- 172.21.15.189:0/3970621231 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3acc0612b0 0x7f3acc13a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:56.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.416+0000 7f3ad2c7d700 1 -- 172.21.15.189:0/3970621231 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3acc0a1720 con 0x7f3acc1374a0 2020-11-20T12:11:56.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.416+0000 7f3ad1c7b700 1 --2- 172.21.15.189:0/3970621231 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3acc067b50 0x7f3acc0a36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:11:56.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.416+0000 7f3ad2c7d700 1 --2- 172.21.15.189:0/3970621231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3acc1374a0 0x7f3acc0a1bc0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f3ac400c710 tx=0x7f3ac4007be0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:56.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.416+0000 7f3aca7fc700 1 -- 172.21.15.189:0/3970621231 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac40072e0 con 0x7f3acc1374a0 2020-11-20T12:11:56.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.416+0000 7f3aca7fc700 1 -- 172.21.15.189:0/3970621231 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac4005530 con 0x7f3acc1374a0 2020-11-20T12:11:56.430 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.416+0000 7f3aca7fc700 1 -- 172.21.15.189:0/3970621231 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac40058d0 con 0x7f3acc1374a0 2020-11-20T12:11:56.431 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.416+0000 7f3ad347e700 1 -- 172.21.15.189:0/3970621231 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3acc0a3f30 con 0x7f3acc1374a0 2020-11-20T12:11:56.431 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.416+0000 7f3ad347e700 1 -- 172.21.15.189:0/3970621231 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3acc14b280 con 0x7f3acc1374a0 2020-11-20T12:11:56.431 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.416+0000 7f3ad347e700 1 -- 172.21.15.189:0/3970621231 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3acc0a18b0 con 0x7f3acc1374a0 2020-11-20T12:11:56.431 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.416+0000 7f3aca7fc700 1 -- 172.21.15.189:0/3970621231 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f3ac4010070 con 0x7f3acc1374a0 2020-11-20T12:11:56.432 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.420+0000 7f3aca7fc700 1 --2- 172.21.15.189:0/3970621231 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3aa40557c0 0x7f3aa4057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:56.432 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.420+0000 7f3aca7fc700 1 -- 172.21.15.189:0/3970621231 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f3ac406de10 con 0x7f3acc1374a0 2020-11-20T12:11:56.432 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.420+0000 7f3ad247c700 1 --2- 172.21.15.189:0/3970621231 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3aa40557c0 0x7f3aa4057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:56.433 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.420+0000 7f3ad247c700 1 --2- 172.21.15.189:0/3970621231 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3aa40557c0 0x7f3aa4057c80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f3abc009d90 tx=0x7f3abc008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:56.433 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.424+0000 7f3aca7fc700 1 -- 172.21.15.189:0/3970621231 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3ac4071030 con 0x7f3acc1374a0 2020-11-20T12:11:56.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.620+0000 7f3ad347e700 1 -- 172.21.15.189:0/3970621231 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3acc061690 con 0x7f3acc1374a0 2020-11-20T12:11:56.630 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.624+0000 7f3aca7fc700 1 -- 172.21.15.189:0/3970621231 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f3ac4018020 con 0x7f3acc1374a0 2020-11-20T12:11:56.630 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:11:56.630 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5122,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/4088050649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4088050649},{"type":"v1","addr":"172.21.15.47:6833","nonce":4088050649}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:11:15.620495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5053},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5053":{"gid":5053,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/3540904386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3540904386},{"type":"v1","addr":"172.21.15.189:6835","nonce":3540904386}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:11:56.631 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.624+0000 7f3ad347e700 1 -- 172.21.15.189:0/3970621231 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3aa40557c0 msgr2=0x7f3aa4057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:56.632 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.624+0000 7f3ad347e700 1 --2- 172.21.15.189:0/3970621231 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3aa40557c0 0x7f3aa4057c80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f3abc009d90 tx=0x7f3abc008040).stop 2020-11-20T12:11:56.632 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.624+0000 7f3ad347e700 1 -- 172.21.15.189:0/3970621231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3acc1374a0 msgr2=0x7f3acc0a1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:56.632 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.624+0000 7f3ad347e700 1 --2- 172.21.15.189:0/3970621231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3acc1374a0 0x7f3acc0a1bc0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f3ac400c710 tx=0x7f3ac4007be0).stop 2020-11-20T12:11:56.632 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.624+0000 7f3ad347e700 1 -- 172.21.15.189:0/3970621231 shutdown_connections 2020-11-20T12:11:56.632 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.624+0000 7f3ad347e700 1 --2- 172.21.15.189:0/3970621231 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3aa40557c0 0x7f3aa4057c80 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:56.633 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.624+0000 7f3ad347e700 1 --2- 172.21.15.189:0/3970621231 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3acc0612b0 0x7f3acc13a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:56.633 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.624+0000 7f3ad347e700 1 --2- 172.21.15.189:0/3970621231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3acc1374a0 0x7f3acc0a1bc0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:56.633 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.624+0000 7f3ad347e700 1 --2- 172.21.15.189:0/3970621231 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3acc067b50 0x7f3acc0a36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:56.633 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.624+0000 7f3ad347e700 1 -- 172.21.15.189:0/3970621231 >> 172.21.15.189:0/3970621231 conn(0x7f3acc063210 msgr2=0x7f3acc133f30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:56.634 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.624+0000 7f3ad347e700 1 -- 172.21.15.189:0/3970621231 shutdown_connections 2020-11-20T12:11:56.634 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:56.624+0000 7f3ad347e700 1 -- 172.21.15.189:0/3970621231 wait complete. 2020-11-20T12:11:56.635 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 26 2020-11-20T12:11:56.643 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:11:58.645 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:11:59.038 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.032+0000 7f044cd34700 1 Processor -- start 2020-11-20T12:11:59.038 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.032+0000 7f044cd34700 1 -- start start 2020-11-20T12:11:59.039 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.032+0000 7f044cd34700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f04480a9660 0x7f044813e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:59.040 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.032+0000 7f044cd34700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04480a9a40 0x7f0448060170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:59.040 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.032+0000 7f044cd34700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f044813dc10 con 0x7f04480a7760 2020-11-20T12:11:59.040 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.032+0000 7f044cd34700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f044813deb0 con 0x7f04480a9660 2020-11-20T12:11:59.040 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.032+0000 7f044cd34700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044813e150 con 0x7f04480a9a40 2020-11-20T12:11:59.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.032+0000 7f0445d9b700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04480a9a40 0x7f0448060170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:59.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.032+0000 7f044659c700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f04480a9660 0x7f044813e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:59.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.032+0000 7f0446d9d700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f04480a7760 0x7f0448060660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.189:46506/0 (socket says 172.21.15.189:46506) 2020-11-20T12:11:59.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.032+0000 7f0446d9d700 1 -- 172.21.15.189:0/3601913148 learned_addr learned my addr 172.21.15.189:0/3601913148 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:11:59.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.032+0000 7f0445d9b700 1 -- 172.21.15.189:0/3601913148 >> v2:172.21.15.47:3300/0 conn(0x7f04480a9660 msgr2=0x7f044813e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:11:59.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.032+0000 7f0445d9b700 1 --2- 172.21.15.189:0/3601913148 >> v2:172.21.15.47:3300/0 conn(0x7f04480a9660 0x7f044813e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:59.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.032+0000 7f0445d9b700 1 -- 172.21.15.189:0/3601913148 >> v1:172.21.15.47:6789/0 conn(0x7f04480a7760 legacy=0x7f0448060660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:59.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.032+0000 7f044559a700 1 -- 172.21.15.189:0/3601913148 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3556235964 0 0) 0x7f044813dc10 con 0x7f04480a7760 2020-11-20T12:11:59.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.032+0000 7f0445d9b700 1 -- 172.21.15.189:0/3601913148 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0448065a30 con 0x7f04480a9a40 2020-11-20T12:11:59.044 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.032+0000 7f044659c700 1 --2- 172.21.15.189:0/3601913148 >> v2:172.21.15.47:3300/0 conn(0x7f04480a9660 0x7f044813e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:11:59.044 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.036+0000 7f0445d9b700 1 --2- 172.21.15.189:0/3601913148 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04480a9a40 0x7f0448060170 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f0438009c90 tx=0x7f04380091d0).ready entity=mon.2 client_cookie=1d3f8fd87afbbdbc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:59.044 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.036+0000 7f044559a700 1 -- 172.21.15.189:0/3601913148 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043800ba70 con 0x7f04480a9a40 2020-11-20T12:11:59.044 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.036+0000 7f044559a700 1 -- 172.21.15.189:0/3601913148 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043800bbd0 con 0x7f04480a9a40 2020-11-20T12:11:59.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.036+0000 7f044559a700 1 -- 172.21.15.189:0/3601913148 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0438011400 con 0x7f04480a9a40 2020-11-20T12:11:59.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.036+0000 7f044cd34700 1 -- 172.21.15.189:0/3601913148 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04480a9a40 msgr2=0x7f0448060170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:59.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.036+0000 7f044cd34700 1 --2- 172.21.15.189:0/3601913148 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04480a9a40 0x7f0448060170 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f0438009c90 tx=0x7f04380091d0).stop 2020-11-20T12:11:59.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.036+0000 7f044cd34700 1 -- 172.21.15.189:0/3601913148 shutdown_connections 2020-11-20T12:11:59.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.036+0000 7f044cd34700 1 --2- 172.21.15.189:0/3601913148 >> v2:172.21.15.47:3300/0 conn(0x7f04480a9660 0x7f044813e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:59.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.036+0000 7f044cd34700 1 --2- 172.21.15.189:0/3601913148 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04480a9a40 0x7f0448060170 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:59.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.036+0000 7f044cd34700 1 -- 172.21.15.189:0/3601913148 >> 172.21.15.189:0/3601913148 conn(0x7f0448061f60 msgr2=0x7f0448062370 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:59.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.036+0000 7f044cd34700 1 -- 172.21.15.189:0/3601913148 shutdown_connections 2020-11-20T12:11:59.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.036+0000 7f044cd34700 1 -- 172.21.15.189:0/3601913148 wait complete. 2020-11-20T12:11:59.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.036+0000 7f044cd34700 1 Processor -- start 2020-11-20T12:11:59.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.036+0000 7f044cd34700 1 -- start start 2020-11-20T12:11:59.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.036+0000 7f044cd34700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04480a7760 0x7f04480a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:59.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.036+0000 7f044cd34700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04480a9660 0x7f04480a6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:59.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.040+0000 7f044cd34700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04480a9a40 0x7f0448149710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:59.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.040+0000 7f044cd34700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04480a5b70 con 0x7f04480a7760 2020-11-20T12:11:59.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.040+0000 7f044cd34700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04480a5190 con 0x7f04480a9660 2020-11-20T12:11:59.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.040+0000 7f044cd34700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04480a5430 con 0x7f04480a9a40 2020-11-20T12:11:59.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.040+0000 7f0445d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04480a9660 0x7f04480a6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:59.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.040+0000 7f0445d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04480a9660 0x7f04480a6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50224/0 (socket says 172.21.15.189:50224) 2020-11-20T12:11:59.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.040+0000 7f0445d9b700 1 -- 172.21.15.189:0/3565312708 learned_addr learned my addr 172.21.15.189:0/3565312708 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:11:59.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.040+0000 7f044659c700 1 --2- 172.21.15.189:0/3565312708 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04480a7760 0x7f04480a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:59.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.040+0000 7f0446d9d700 1 --2- 172.21.15.189:0/3565312708 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04480a9a40 0x7f0448149710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:59.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.040+0000 7f0445d9b700 1 -- 172.21.15.189:0/3565312708 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04480a9a40 msgr2=0x7f0448149710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:59.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.040+0000 7f0445d9b700 1 --2- 172.21.15.189:0/3565312708 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04480a9a40 0x7f0448149710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:59.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.040+0000 7f0445d9b700 1 -- 172.21.15.189:0/3565312708 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04480a7760 msgr2=0x7f04480a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:59.052 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.040+0000 7f0445d9b700 1 --2- 172.21.15.189:0/3565312708 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04480a7760 0x7f04480a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:59.052 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.040+0000 7f0445d9b700 1 -- 172.21.15.189:0/3565312708 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04480a4e80 con 0x7f04480a9660 2020-11-20T12:11:59.052 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.040+0000 7f0445d9b700 1 --2- 172.21.15.189:0/3565312708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04480a9660 0x7f04480a6480 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f0438011530 tx=0x7f0438016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:59.053 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.040+0000 7f043e7fc700 1 -- 172.21.15.189:0/3565312708 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043801c5f0 con 0x7f04480a9660 2020-11-20T12:11:59.053 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.040+0000 7f043e7fc700 1 -- 172.21.15.189:0/3565312708 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043801c750 con 0x7f04480a9660 2020-11-20T12:11:59.053 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.040+0000 7f044cd34700 1 -- 172.21.15.189:0/3565312708 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04480a1c70 con 0x7f04480a9660 2020-11-20T12:11:59.053 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.040+0000 7f043e7fc700 1 -- 172.21.15.189:0/3565312708 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043801caf0 con 0x7f04480a9660 2020-11-20T12:11:59.054 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.040+0000 7f044cd34700 1 -- 172.21.15.189:0/3565312708 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04480a6b10 con 0x7f04480a9660 2020-11-20T12:11:59.055 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.044+0000 7f044cd34700 1 -- 172.21.15.189:0/3565312708 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0448044f60 con 0x7f04480a9660 2020-11-20T12:11:59.055 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.044+0000 7f043e7fc700 1 -- 172.21.15.189:0/3565312708 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f0438024030 con 0x7f04480a9660 2020-11-20T12:11:59.055 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.044+0000 7f043e7fc700 1 --2- 172.21.15.189:0/3565312708 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f04240557c0 0x7f0424057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:11:59.056 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.044+0000 7f043e7fc700 1 -- 172.21.15.189:0/3565312708 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f043806fc70 con 0x7f04480a9660 2020-11-20T12:11:59.056 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.044+0000 7f044659c700 1 --2- 172.21.15.189:0/3565312708 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f04240557c0 0x7f0424057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:11:59.056 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.044+0000 7f044659c700 1 --2- 172.21.15.189:0/3565312708 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f04240557c0 0x7f0424057c80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f0430006fd0 tx=0x7f0430006df0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:11:59.056 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.052+0000 7f043e7fc700 1 -- 172.21.15.189:0/3565312708 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0438045a60 con 0x7f04480a9660 2020-11-20T12:11:59.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.252+0000 7f044cd34700 1 -- 172.21.15.189:0/3565312708 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0448061380 con 0x7f04480a9660 2020-11-20T12:11:59.260 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.252+0000 7f043e7fc700 1 -- 172.21.15.189:0/3565312708 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f0438022070 con 0x7f04480a9660 2020-11-20T12:11:59.261 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:11:59.261 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5122,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/4088050649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4088050649},{"type":"v1","addr":"172.21.15.47:6833","nonce":4088050649}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:11:15.620495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5053},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5053":{"gid":5053,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/3540904386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3540904386},{"type":"v1","addr":"172.21.15.189:6835","nonce":3540904386}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:11:59.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.256+0000 7f044cd34700 1 -- 172.21.15.189:0/3565312708 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f04240557c0 msgr2=0x7f0424057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:59.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.256+0000 7f044cd34700 1 --2- 172.21.15.189:0/3565312708 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f04240557c0 0x7f0424057c80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f0430006fd0 tx=0x7f0430006df0).stop 2020-11-20T12:11:59.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.256+0000 7f044cd34700 1 -- 172.21.15.189:0/3565312708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04480a9660 msgr2=0x7f04480a6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:11:59.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.256+0000 7f044cd34700 1 --2- 172.21.15.189:0/3565312708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04480a9660 0x7f04480a6480 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f0438011530 tx=0x7f0438016000).stop 2020-11-20T12:11:59.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.256+0000 7f044cd34700 1 -- 172.21.15.189:0/3565312708 shutdown_connections 2020-11-20T12:11:59.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.256+0000 7f044cd34700 1 --2- 172.21.15.189:0/3565312708 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f04240557c0 0x7f0424057c80 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:59.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.256+0000 7f044cd34700 1 --2- 172.21.15.189:0/3565312708 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04480a7760 0x7f04480a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:59.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.256+0000 7f044cd34700 1 --2- 172.21.15.189:0/3565312708 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04480a9a40 0x7f0448149710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:59.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.256+0000 7f044cd34700 1 --2- 172.21.15.189:0/3565312708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04480a9660 0x7f04480a6480 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:11:59.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.256+0000 7f044cd34700 1 -- 172.21.15.189:0/3565312708 >> 172.21.15.189:0/3565312708 conn(0x7f0448061f60 msgr2=0x7f044813d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:11:59.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.256+0000 7f044cd34700 1 -- 172.21.15.189:0/3565312708 shutdown_connections 2020-11-20T12:11:59.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:11:59.256+0000 7f044cd34700 1 -- 172.21.15.189:0/3565312708 wait complete. 2020-11-20T12:11:59.266 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 26 2020-11-20T12:11:59.283 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:12:01.284 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:01.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.408+0000 7f4081284700 1 Processor -- start 2020-11-20T12:12:01.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.412+0000 7f4081284700 1 -- start start 2020-11-20T12:12:01.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.412+0000 7f4081284700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f407c0a8840 0x7f407c13eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:01.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.412+0000 7f4081284700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f407c142eb0 con 0x7f407c066ed0 2020-11-20T12:12:01.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.412+0000 7f4081284700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f407c066800 con 0x7f407c13f030 2020-11-20T12:12:01.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.412+0000 7f4081284700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f407c13f4a0 con 0x7f407c0a8840 2020-11-20T12:12:01.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.412+0000 7f4080a83700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f407c13f030 0x7f407c1427a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51856/0 (socket says 172.21.15.189:51856) 2020-11-20T12:12:01.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.412+0000 7f4080a83700 1 -- 172.21.15.189:0/1331380398 learned_addr learned my addr 172.21.15.189:0/1331380398 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:12:01.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.412+0000 7f407bfff700 1 --2- 172.21.15.189:0/1331380398 >> v2:172.21.15.189:3300/0 conn(0x7f407c0a8840 0x7f407c13eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:01.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.412+0000 7f407b7fe700 1 -- 172.21.15.189:0/1331380398 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 219798627 0 0) 0x7f407c142eb0 con 0x7f407c066ed0 2020-11-20T12:12:01.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.412+0000 7f407b7fe700 1 -- 172.21.15.189:0/1331380398 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f405c003240 con 0x7f407c066ed0 2020-11-20T12:12:01.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.412+0000 7f407bfff700 1 -- 172.21.15.189:0/1331380398 >> v1:172.21.15.189:6789/0 conn(0x7f407c13f030 legacy=0x7f407c1427a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:01.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.412+0000 7f407bfff700 1 -- 172.21.15.189:0/1331380398 >> v1:172.21.15.47:6789/0 conn(0x7f407c066ed0 legacy=0x7f407c142090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:01.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.412+0000 7f407b7fe700 1 -- 172.21.15.189:0/1331380398 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806379846 0 0) 0x7f407c066800 con 0x7f407c13f030 2020-11-20T12:12:01.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.412+0000 7f407bfff700 1 -- 172.21.15.189:0/1331380398 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f407c066aa0 con 0x7f407c0a8840 2020-11-20T12:12:01.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.412+0000 7f407bfff700 1 --2- 172.21.15.189:0/1331380398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f407c0a8840 0x7f407c13eb40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f406400b9a0 tx=0x7f4064006df0).ready entity=mon.1 client_cookie=c28586e88ea550ba server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:01.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.412+0000 7f407b7fe700 1 -- 172.21.15.189:0/1331380398 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40640097b0 con 0x7f407c0a8840 2020-11-20T12:12:01.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.412+0000 7f407b7fe700 1 -- 172.21.15.189:0/1331380398 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4064009910 con 0x7f407c0a8840 2020-11-20T12:12:01.423 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.412+0000 7f407b7fe700 1 -- 172.21.15.189:0/1331380398 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4064009c90 con 0x7f407c0a8840 2020-11-20T12:12:01.425 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.412+0000 7f4081284700 1 -- 172.21.15.189:0/1331380398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f407c0a8840 msgr2=0x7f407c13eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:01.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.412+0000 7f4081284700 1 --2- 172.21.15.189:0/1331380398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f407c0a8840 0x7f407c13eb40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f406400b9a0 tx=0x7f4064006df0).stop 2020-11-20T12:12:01.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.412+0000 7f4081284700 1 -- 172.21.15.189:0/1331380398 shutdown_connections 2020-11-20T12:12:01.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.412+0000 7f4081284700 1 --2- 172.21.15.189:0/1331380398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f407c0a8840 0x7f407c13eb40 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:01.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.412+0000 7f4081284700 1 -- 172.21.15.189:0/1331380398 >> 172.21.15.189:0/1331380398 conn(0x7f407c060170 msgr2=0x7f407c060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:01.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.412+0000 7f4081284700 1 -- 172.21.15.189:0/1331380398 shutdown_connections 2020-11-20T12:12:01.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.412+0000 7f4081284700 1 -- 172.21.15.189:0/1331380398 wait complete. 2020-11-20T12:12:01.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.416+0000 7f4081284700 1 Processor -- start 2020-11-20T12:12:01.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.416+0000 7f4081284700 1 -- start start 2020-11-20T12:12:01.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.416+0000 7f4081284700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f407c066ed0 0x7f407c0a7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:01.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.416+0000 7f4081284700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f407c0a8840 0x7f407c0a3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:01.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.416+0000 7f4081284700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f407c13f030 0x7f407c0a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:01.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.416+0000 7f4081284700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407c0a49c0 con 0x7f407c13f030 2020-11-20T12:12:01.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.416+0000 7f4081284700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407c0a5300 con 0x7f407c0a8840 2020-11-20T12:12:01.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.416+0000 7f4081284700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f407c0a55a0 con 0x7f407c066ed0 2020-11-20T12:12:01.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.416+0000 7f4073fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f407c0a8840 0x7f407c0a3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:01.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.416+0000 7f4073fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f407c0a8840 0x7f407c0a3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50238/0 (socket says 172.21.15.189:50238) 2020-11-20T12:12:01.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.416+0000 7f4073fff700 1 -- 172.21.15.189:0/131661459 learned_addr learned my addr 172.21.15.189:0/131661459 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:01.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.416+0000 7f407bfff700 1 --2- 172.21.15.189:0/131661459 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f407c066ed0 0x7f407c0a7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:01.430 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.416+0000 7f4080a83700 1 --2- 172.21.15.189:0/131661459 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f407c13f030 0x7f407c0a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:01.430 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.416+0000 7f4073fff700 1 -- 172.21.15.189:0/131661459 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f407c066ed0 msgr2=0x7f407c0a7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:01.430 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.416+0000 7f4073fff700 1 --2- 172.21.15.189:0/131661459 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f407c066ed0 0x7f407c0a7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:01.430 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.416+0000 7f4073fff700 1 -- 172.21.15.189:0/131661459 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f407c13f030 msgr2=0x7f407c0a44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:01.430 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.416+0000 7f4073fff700 1 --2- 172.21.15.189:0/131661459 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f407c13f030 0x7f407c0a44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:01.431 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.416+0000 7f4073fff700 1 -- 172.21.15.189:0/131661459 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f407c0a5840 con 0x7f407c0a8840 2020-11-20T12:12:01.431 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.416+0000 7f4073fff700 1 --2- 172.21.15.189:0/131661459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f407c0a8840 0x7f407c0a3fe0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f406c00c700 tx=0x7f406c007ae0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:01.431 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.416+0000 7f4078ff9700 1 -- 172.21.15.189:0/131661459 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406c00e490 con 0x7f407c0a8840 2020-11-20T12:12:01.431 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.416+0000 7f4078ff9700 1 -- 172.21.15.189:0/131661459 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f406c00e5f0 con 0x7f407c0a8840 2020-11-20T12:12:01.432 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.416+0000 7f4081284700 1 -- 172.21.15.189:0/131661459 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f407c13c750 con 0x7f407c0a8840 2020-11-20T12:12:01.432 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.416+0000 7f4078ff9700 1 -- 172.21.15.189:0/131661459 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406c00e990 con 0x7f407c0a8840 2020-11-20T12:12:01.432 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.416+0000 7f4081284700 1 -- 172.21.15.189:0/131661459 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f407c0a4d80 con 0x7f407c0a8840 2020-11-20T12:12:01.433 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.420+0000 7f4078ff9700 1 -- 172.21.15.189:0/131661459 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f406c019030 con 0x7f407c0a8840 2020-11-20T12:12:01.433 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.420+0000 7f4081284700 1 -- 172.21.15.189:0/131661459 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f407c0446d0 con 0x7f407c0a8840 2020-11-20T12:12:01.433 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.420+0000 7f4078ff9700 1 --2- 172.21.15.189:0/131661459 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f40580557f0 0x7f4058057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:01.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.420+0000 7f4078ff9700 1 -- 172.21.15.189:0/131661459 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f406c06dc30 con 0x7f407c0a8840 2020-11-20T12:12:01.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.420+0000 7f407bfff700 1 --2- 172.21.15.189:0/131661459 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f40580557f0 0x7f4058057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:01.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.420+0000 7f407bfff700 1 --2- 172.21.15.189:0/131661459 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f40580557f0 0x7f4058057cb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f406400be60 tx=0x7f4064015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:01.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.424+0000 7f4078ff9700 1 -- 172.21.15.189:0/131661459 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f406c043970 con 0x7f407c0a8840 2020-11-20T12:12:01.631 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.624+0000 7f4081284700 1 -- 172.21.15.189:0/131661459 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f407c05cdd0 con 0x7f407c0a8840 2020-11-20T12:12:01.632 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.624+0000 7f4078ff9700 1 -- 172.21.15.189:0/131661459 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f406c046f90 con 0x7f407c0a8840 2020-11-20T12:12:01.633 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:12:01.633 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5122,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/4088050649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4088050649},{"type":"v1","addr":"172.21.15.47:6833","nonce":4088050649}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:11:15.620495+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":26,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5053},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5053":{"gid":5053,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/3540904386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3540904386},{"type":"v1","addr":"172.21.15.189:6835","nonce":3540904386}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:12:01.634 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.628+0000 7f4081284700 1 -- 172.21.15.189:0/131661459 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f40580557f0 msgr2=0x7f4058057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:01.635 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.628+0000 7f4081284700 1 --2- 172.21.15.189:0/131661459 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f40580557f0 0x7f4058057cb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f406400be60 tx=0x7f4064015000).stop 2020-11-20T12:12:01.635 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.628+0000 7f4081284700 1 -- 172.21.15.189:0/131661459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f407c0a8840 msgr2=0x7f407c0a3fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:01.635 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.628+0000 7f4081284700 1 --2- 172.21.15.189:0/131661459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f407c0a8840 0x7f407c0a3fe0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f406c00c700 tx=0x7f406c007ae0).stop 2020-11-20T12:12:01.636 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.628+0000 7f4081284700 1 -- 172.21.15.189:0/131661459 shutdown_connections 2020-11-20T12:12:01.636 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.628+0000 7f4081284700 1 --2- 172.21.15.189:0/131661459 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f40580557f0 0x7f4058057cb0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:01.636 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.628+0000 7f4081284700 1 --2- 172.21.15.189:0/131661459 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f407c13f030 0x7f407c0a44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:01.636 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.628+0000 7f4081284700 1 --2- 172.21.15.189:0/131661459 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f407c066ed0 0x7f407c0a7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:01.637 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.628+0000 7f4081284700 1 --2- 172.21.15.189:0/131661459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f407c0a8840 0x7f407c0a3fe0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:01.637 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.628+0000 7f4081284700 1 -- 172.21.15.189:0/131661459 >> 172.21.15.189:0/131661459 conn(0x7f407c060170 msgr2=0x7f407c05ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:01.637 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.628+0000 7f4081284700 1 -- 172.21.15.189:0/131661459 shutdown_connections 2020-11-20T12:12:01.637 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:01.628+0000 7f4081284700 1 -- 172.21.15.189:0/131661459 wait complete. 2020-11-20T12:12:01.637 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 26 2020-11-20T12:12:01.651 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:12:03.653 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:03.778 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.772+0000 7f50af59e700 1 Processor -- start 2020-11-20T12:12:03.779 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.772+0000 7f50af59e700 1 -- start start 2020-11-20T12:12:03.780 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.772+0000 7f50af59e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f50b0065d20 0x7f50b0067460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:03.781 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.772+0000 7f50af59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b013c290 con 0x7f50b0142820 2020-11-20T12:12:03.781 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.772+0000 7f50af59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b0141e40 con 0x7f50b00a9440 2020-11-20T12:12:03.781 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.772+0000 7f50af59e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b00670e0 con 0x7f50b0065d20 2020-11-20T12:12:03.781 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.772+0000 7f50ae59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f50b00a9440 0x7f50b013b470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51872/0 (socket says 172.21.15.189:51872) 2020-11-20T12:12:03.782 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.772+0000 7f50add9b700 1 --2- 172.21.15.189:0/2986741736 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f50b0065d20 0x7f50b0067460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:03.782 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.772+0000 7f50ae59c700 1 -- 172.21.15.189:0/2986741736 learned_addr learned my addr 172.21.15.189:0/2986741736 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:12:03.782 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.772+0000 7f50ad59a700 1 -- 172.21.15.189:0/2986741736 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3654412617 0 0) 0x7f50b0141e40 con 0x7f50b00a9440 2020-11-20T12:12:03.782 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.772+0000 7f50ad59a700 1 -- 172.21.15.189:0/2986741736 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5090003240 con 0x7f50b00a9440 2020-11-20T12:12:03.782 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.772+0000 7f50ad59a700 1 -- 172.21.15.189:0/2986741736 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269370933 0 0) 0x7f50b013c290 con 0x7f50b0142820 2020-11-20T12:12:03.783 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.772+0000 7f50ad59a700 1 -- 172.21.15.189:0/2986741736 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b0141e40 con 0x7f50b0142820 2020-11-20T12:12:03.783 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.772+0000 7f50add9b700 1 -- 172.21.15.189:0/2986741736 >> v1:172.21.15.189:6789/0 conn(0x7f50b00a9440 legacy=0x7f50b013b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:03.783 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.772+0000 7f50add9b700 1 -- 172.21.15.189:0/2986741736 >> v1:172.21.15.47:6789/0 conn(0x7f50b0142820 legacy=0x7f50b013bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:03.783 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.772+0000 7f50add9b700 1 -- 172.21.15.189:0/2986741736 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b01420e0 con 0x7f50b0065d20 2020-11-20T12:12:03.783 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.772+0000 7f50add9b700 1 --2- 172.21.15.189:0/2986741736 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f50b0065d20 0x7f50b0067460 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f50a8009cc0 tx=0x7f50a80091d0).ready entity=mon.2 client_cookie=d32b64a7bf1079a9 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:03.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.772+0000 7f50ad59a700 1 -- 172.21.15.189:0/2986741736 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a800bab0 con 0x7f50b0065d20 2020-11-20T12:12:03.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.772+0000 7f50ad59a700 1 -- 172.21.15.189:0/2986741736 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a800bc10 con 0x7f50b0065d20 2020-11-20T12:12:03.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.772+0000 7f50ad59a700 1 -- 172.21.15.189:0/2986741736 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a8011400 con 0x7f50b0065d20 2020-11-20T12:12:03.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50af59e700 1 -- 172.21.15.189:0/2986741736 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f50b0065d20 msgr2=0x7f50b0067460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:03.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50af59e700 1 --2- 172.21.15.189:0/2986741736 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f50b0065d20 0x7f50b0067460 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f50a8009cc0 tx=0x7f50a80091d0).stop 2020-11-20T12:12:03.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50af59e700 1 -- 172.21.15.189:0/2986741736 shutdown_connections 2020-11-20T12:12:03.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50af59e700 1 --2- 172.21.15.189:0/2986741736 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f50b0065d20 0x7f50b0067460 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:03.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50af59e700 1 -- 172.21.15.189:0/2986741736 >> 172.21.15.189:0/2986741736 conn(0x7f50b0060170 msgr2=0x7f50b0060550 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:03.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50af59e700 1 -- 172.21.15.189:0/2986741736 shutdown_connections 2020-11-20T12:12:03.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50af59e700 1 -- 172.21.15.189:0/2986741736 wait complete. 2020-11-20T12:12:03.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50af59e700 1 Processor -- start 2020-11-20T12:12:03.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50af59e700 1 -- start start 2020-11-20T12:12:03.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50af59e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50b0065d20 0x7f50b00ab7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:03.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50af59e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f50b00a9440 0x7f50b014dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:03.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50af59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50b0142820 0x7f50b00a3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:03.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50af59e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b00aa750 con 0x7f50b0065d20 2020-11-20T12:12:03.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50af59e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b00aa9f0 con 0x7f50b0142820 2020-11-20T12:12:03.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50af59e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b00aac90 con 0x7f50b00a9440 2020-11-20T12:12:03.790 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50aed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50b0142820 0x7f50b00a3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:03.790 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50aed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50b0142820 0x7f50b00a3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50258/0 (socket says 172.21.15.189:50258) 2020-11-20T12:12:03.790 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50aed9d700 1 -- 172.21.15.189:0/402824236 learned_addr learned my addr 172.21.15.189:0/402824236 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:03.790 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50add9b700 1 --2- 172.21.15.189:0/402824236 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f50b00a9440 0x7f50b014dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:03.791 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50ae59c700 1 --2- 172.21.15.189:0/402824236 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50b0065d20 0x7f50b00ab7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:03.791 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50add9b700 1 -- 172.21.15.189:0/402824236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50b0142820 msgr2=0x7f50b00a3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:03.791 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50add9b700 1 --2- 172.21.15.189:0/402824236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50b0142820 0x7f50b00a3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:03.791 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50add9b700 1 -- 172.21.15.189:0/402824236 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50b0065d20 msgr2=0x7f50b00ab7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:03.791 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50add9b700 1 --2- 172.21.15.189:0/402824236 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50b0065d20 0x7f50b00ab7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:03.792 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50add9b700 1 -- 172.21.15.189:0/402824236 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b014e170 con 0x7f50b00a9440 2020-11-20T12:12:03.792 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50aed9d700 1 --2- 172.21.15.189:0/402824236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50b0142820 0x7f50b00a3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:12:03.792 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.776+0000 7f50add9b700 1 --2- 172.21.15.189:0/402824236 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f50b00a9440 0x7f50b014dc80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f50a8009cc0 tx=0x7f50a8016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:03.792 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.780+0000 7f50a67fc700 1 -- 172.21.15.189:0/402824236 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a8011e70 con 0x7f50b00a9440 2020-11-20T12:12:03.792 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.780+0000 7f50a67fc700 1 -- 172.21.15.189:0/402824236 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a801c5f0 con 0x7f50b00a9440 2020-11-20T12:12:03.793 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.780+0000 7f50af59e700 1 -- 172.21.15.189:0/402824236 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b013c590 con 0x7f50b00a9440 2020-11-20T12:12:03.793 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.780+0000 7f50a67fc700 1 -- 172.21.15.189:0/402824236 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a801c970 con 0x7f50b00a9440 2020-11-20T12:12:03.793 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.780+0000 7f50af59e700 1 -- 172.21.15.189:0/402824236 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b00a43d0 con 0x7f50b00a9440 2020-11-20T12:12:03.793 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.780+0000 7f50af59e700 1 -- 172.21.15.189:0/402824236 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b00aae50 con 0x7f50b00a9440 2020-11-20T12:12:03.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.780+0000 7f50a67fc700 1 -- 172.21.15.189:0/402824236 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f50a8024030 con 0x7f50b00a9440 2020-11-20T12:12:03.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.780+0000 7f50a67fc700 1 --2- 172.21.15.189:0/402824236 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f508c0557c0 0x7f508c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:03.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.780+0000 7f50ae59c700 1 --2- 172.21.15.189:0/402824236 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f508c0557c0 0x7f508c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:03.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.780+0000 7f50a67fc700 1 -- 172.21.15.189:0/402824236 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f50a8070270 con 0x7f50b00a9440 2020-11-20T12:12:03.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.780+0000 7f50ae59c700 1 --2- 172.21.15.189:0/402824236 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f508c0557c0 0x7f508c057c80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f50980055a0 tx=0x7f5098009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:03.797 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.792+0000 7f50a67fc700 1 -- 172.21.15.189:0/402824236 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f50a8045f60 con 0x7f50b00a9440 2020-11-20T12:12:03.997 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.992+0000 7f50af59e700 1 -- 172.21.15.189:0/402824236 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50b00a39b0 con 0x7f50b00a9440 2020-11-20T12:12:03.998 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.992+0000 7f50a67fc700 1 -- 172.21.15.189:0/402824236 <== mon.2 v2:172.21.15.47:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+1788 (secure 0 0 0) 0x7f50a8022090 con 0x7f50b00a9440 2020-11-20T12:12:03.998 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:12:03.999 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:03.665053+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5122":{"gid":5122,"name":"b","rank":0,"incarnation":27,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.47:6833/4088050649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4088050649},{"type":"v1","addr":"172.21.15.47:6833","nonce":4088050649}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:12:04.000 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.992+0000 7f50af59e700 1 -- 172.21.15.189:0/402824236 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f508c0557c0 msgr2=0x7f508c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:04.000 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.992+0000 7f50af59e700 1 --2- 172.21.15.189:0/402824236 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f508c0557c0 0x7f508c057c80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f50980055a0 tx=0x7f5098009040).stop 2020-11-20T12:12:04.001 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.992+0000 7f50af59e700 1 -- 172.21.15.189:0/402824236 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f50b00a9440 msgr2=0x7f50b014dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:04.001 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.992+0000 7f50af59e700 1 --2- 172.21.15.189:0/402824236 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f50b00a9440 0x7f50b014dc80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f50a8009cc0 tx=0x7f50a8016000).stop 2020-11-20T12:12:04.001 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.992+0000 7f50af59e700 1 -- 172.21.15.189:0/402824236 shutdown_connections 2020-11-20T12:12:04.001 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.992+0000 7f50af59e700 1 --2- 172.21.15.189:0/402824236 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f508c0557c0 0x7f508c057c80 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:04.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.992+0000 7f50af59e700 1 --2- 172.21.15.189:0/402824236 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50b0065d20 0x7f50b00ab7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:04.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.992+0000 7f50af59e700 1 --2- 172.21.15.189:0/402824236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50b0142820 0x7f50b00a3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:04.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.992+0000 7f50af59e700 1 --2- 172.21.15.189:0/402824236 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f50b00a9440 0x7f50b014dc80 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:04.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.992+0000 7f50af59e700 1 -- 172.21.15.189:0/402824236 >> 172.21.15.189:0/402824236 conn(0x7f50b0060170 msgr2=0x7f50b00a7880 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:04.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.992+0000 7f50af59e700 1 -- 172.21.15.189:0/402824236 shutdown_connections 2020-11-20T12:12:04.003 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:03.992+0000 7f50af59e700 1 -- 172.21.15.189:0/402824236 wait complete. 2020-11-20T12:12:04.003 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 28 2020-11-20T12:12:04.014 INFO:tasks.mds_thrash.fs.[None]:mds.a down, removed from mdsmap 2020-11-20T12:12:04.014 INFO:tasks.mds_thrash.fs.[None]:waiting for mds cluster to stabilize... 2020-11-20T12:12:04.015 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:04.138 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.132+0000 7fef6a79e700 1 Processor -- start 2020-11-20T12:12:04.139 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.132+0000 7fef6a79e700 1 -- start start 2020-11-20T12:12:04.140 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.132+0000 7fef6a79e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fef64060310 0x7fef640a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:04.140 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.132+0000 7fef6a79e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef64063b50 con 0x7fef6413fc70 2020-11-20T12:12:04.141 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.132+0000 7fef6a79e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef640a09a0 con 0x7fef64134460 2020-11-20T12:12:04.141 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.132+0000 7fef6a79e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef64067a00 con 0x7fef64060310 2020-11-20T12:12:04.141 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.132+0000 7fef6979c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fef64060310 0x7fef640a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:04.141 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.132+0000 7fef6979c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fef64060310 0x7fef640a11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50262/0 (socket says 172.21.15.189:50262) 2020-11-20T12:12:04.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.132+0000 7fef6979c700 1 -- 172.21.15.189:0/2513542386 learned_addr learned my addr 172.21.15.189:0/2513542386 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:04.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.132+0000 7fef6979c700 1 -- 172.21.15.189:0/2513542386 >> v1:172.21.15.189:6789/0 conn(0x7fef64134460 legacy=0x7fef641348e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:04.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.132+0000 7fef6979c700 1 -- 172.21.15.189:0/2513542386 >> v1:172.21.15.47:6789/0 conn(0x7fef6413fc70 legacy=0x7fef64142060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:04.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.132+0000 7fef6979c700 1 -- 172.21.15.189:0/2513542386 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef64066500 con 0x7fef64060310 2020-11-20T12:12:04.143 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.132+0000 7fef6979c700 1 --2- 172.21.15.189:0/2513542386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef64060310 0x7fef640a11c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fef54009d80 tx=0x7fef54009290).ready entity=mon.1 client_cookie=59514a47ea83b307 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:04.144 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.132+0000 7fef63fff700 1 -- 172.21.15.189:0/2513542386 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5400bc40 con 0x7fef64060310 2020-11-20T12:12:04.144 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.132+0000 7fef63fff700 1 -- 172.21.15.189:0/2513542386 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef5400bda0 con 0x7fef64060310 2020-11-20T12:12:04.144 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.132+0000 7fef63fff700 1 -- 172.21.15.189:0/2513542386 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef54011610 con 0x7fef64060310 2020-11-20T12:12:04.146 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.136+0000 7fef6a79e700 1 -- 172.21.15.189:0/2513542386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef64060310 msgr2=0x7fef640a11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:04.147 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.136+0000 7fef6a79e700 1 --2- 172.21.15.189:0/2513542386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef64060310 0x7fef640a11c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fef54009d80 tx=0x7fef54009290).stop 2020-11-20T12:12:04.147 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.136+0000 7fef6a79e700 1 -- 172.21.15.189:0/2513542386 shutdown_connections 2020-11-20T12:12:04.147 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.136+0000 7fef6a79e700 1 --2- 172.21.15.189:0/2513542386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef64060310 0x7fef640a11c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:04.147 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.136+0000 7fef6a79e700 1 -- 172.21.15.189:0/2513542386 >> 172.21.15.189:0/2513542386 conn(0x7fef64063210 msgr2=0x7fef64060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:04.148 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.136+0000 7fef6a79e700 1 -- 172.21.15.189:0/2513542386 shutdown_connections 2020-11-20T12:12:04.148 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.136+0000 7fef6a79e700 1 -- 172.21.15.189:0/2513542386 wait complete. 2020-11-20T12:12:04.148 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.136+0000 7fef6a79e700 1 Processor -- start 2020-11-20T12:12:04.148 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.136+0000 7fef6a79e700 1 -- start start 2020-11-20T12:12:04.148 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef6a79e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef64060310 0x7fef640a02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:04.149 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef6a79e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef64134460 0x7fef6409d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:04.149 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef6a79e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef6413fc70 0x7fef64147170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:04.149 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef6a79e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6409e1d0 con 0x7fef64060310 2020-11-20T12:12:04.149 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef6a79e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef64146cd0 con 0x7fef64134460 2020-11-20T12:12:04.150 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef6a79e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef641469c0 con 0x7fef6413fc70 2020-11-20T12:12:04.150 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef68f9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef64134460 0x7fef6409d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:04.150 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef68f9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef64134460 0x7fef6409d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50270/0 (socket says 172.21.15.189:50270) 2020-11-20T12:12:04.150 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef68f9b700 1 -- 172.21.15.189:0/2341450273 learned_addr learned my addr 172.21.15.189:0/2341450273 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:04.150 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef6979c700 1 --2- 172.21.15.189:0/2341450273 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef64060310 0x7fef640a02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:04.151 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef69f9d700 1 --2- 172.21.15.189:0/2341450273 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef6413fc70 0x7fef64147170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:04.151 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef68f9b700 1 -- 172.21.15.189:0/2341450273 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef6413fc70 msgr2=0x7fef64147170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:04.151 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef68f9b700 1 --2- 172.21.15.189:0/2341450273 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef6413fc70 0x7fef64147170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:04.151 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef68f9b700 1 -- 172.21.15.189:0/2341450273 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef64060310 msgr2=0x7fef640a02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:04.151 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef68f9b700 1 --2- 172.21.15.189:0/2341450273 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef64060310 0x7fef640a02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:04.152 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef68f9b700 1 -- 172.21.15.189:0/2341450273 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef6409fd90 con 0x7fef64134460 2020-11-20T12:12:04.152 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef6979c700 1 --2- 172.21.15.189:0/2341450273 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef64060310 0x7fef640a02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:12:04.152 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef68f9b700 1 --2- 172.21.15.189:0/2341450273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef64134460 0x7fef6409d8c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fef58001e70 tx=0x7fef58007c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:04.153 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef617fa700 1 -- 172.21.15.189:0/2341450273 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef58007020 con 0x7fef64134460 2020-11-20T12:12:04.153 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef617fa700 1 -- 172.21.15.189:0/2341450273 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef58007180 con 0x7fef64134460 2020-11-20T12:12:04.154 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef6a79e700 1 -- 172.21.15.189:0/2341450273 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef6409ddb0 con 0x7fef64134460 2020-11-20T12:12:04.154 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef617fa700 1 -- 172.21.15.189:0/2341450273 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5800e560 con 0x7fef64134460 2020-11-20T12:12:04.154 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef6a79e700 1 -- 172.21.15.189:0/2341450273 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef64147880 con 0x7fef64134460 2020-11-20T12:12:04.155 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef6a79e700 1 -- 172.21.15.189:0/2341450273 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef64146b30 con 0x7fef64134460 2020-11-20T12:12:04.155 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef617fa700 1 -- 172.21.15.189:0/2341450273 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fef58010070 con 0x7fef64134460 2020-11-20T12:12:04.156 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef617fa700 1 --2- 172.21.15.189:0/2341450273 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fef3c0557c0 0x7fef3c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:04.156 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef617fa700 1 -- 172.21.15.189:0/2341450273 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6503+0+0 (secure 0 0 0) 0x7fef5806dcf0 con 0x7fef64134460 2020-11-20T12:12:04.156 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.140+0000 7fef6979c700 1 --2- 172.21.15.189:0/2341450273 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fef3c0557c0 0x7fef3c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:04.156 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.144+0000 7fef6979c700 1 --2- 172.21.15.189:0/2341450273 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fef3c0557c0 0x7fef3c057c80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fef540090f0 tx=0x7fef54016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:04.156 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.148+0000 7fef617fa700 1 -- 172.21.15.189:0/2341450273 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fef58042fd0 con 0x7fef64134460 2020-11-20T12:12:04.351 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.344+0000 7fef6a79e700 1 -- 172.21.15.189:0/2341450273 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef640446d0 con 0x7fef64134460 2020-11-20T12:12:04.352 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.344+0000 7fef617fa700 1 -- 172.21.15.189:0/2341450273 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+1788 (secure 0 0 0) 0x7fef58020710 con 0x7fef64134460 2020-11-20T12:12:04.353 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:12:04.353 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:03.665053+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5122":{"gid":5122,"name":"b","rank":0,"incarnation":27,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.47:6833/4088050649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4088050649},{"type":"v1","addr":"172.21.15.47:6833","nonce":4088050649}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:12:04.355 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.348+0000 7fef6a79e700 1 -- 172.21.15.189:0/2341450273 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fef3c0557c0 msgr2=0x7fef3c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:04.355 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.348+0000 7fef6a79e700 1 --2- 172.21.15.189:0/2341450273 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fef3c0557c0 0x7fef3c057c80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fef540090f0 tx=0x7fef54016000).stop 2020-11-20T12:12:04.355 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.348+0000 7fef6a79e700 1 -- 172.21.15.189:0/2341450273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef64134460 msgr2=0x7fef6409d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:04.356 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.348+0000 7fef6a79e700 1 --2- 172.21.15.189:0/2341450273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef64134460 0x7fef6409d8c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fef58001e70 tx=0x7fef58007c50).stop 2020-11-20T12:12:04.356 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.348+0000 7fef6a79e700 1 -- 172.21.15.189:0/2341450273 shutdown_connections 2020-11-20T12:12:04.356 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.348+0000 7fef6a79e700 1 --2- 172.21.15.189:0/2341450273 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fef3c0557c0 0x7fef3c057c80 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:04.356 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.348+0000 7fef6a79e700 1 --2- 172.21.15.189:0/2341450273 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef64060310 0x7fef640a02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:04.357 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.348+0000 7fef6a79e700 1 --2- 172.21.15.189:0/2341450273 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef6413fc70 0x7fef64147170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:04.357 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.348+0000 7fef6a79e700 1 --2- 172.21.15.189:0/2341450273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef64134460 0x7fef6409d8c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:04.357 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.348+0000 7fef6a79e700 1 -- 172.21.15.189:0/2341450273 >> 172.21.15.189:0/2341450273 conn(0x7fef64063210 msgr2=0x7fef641348e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:04.357 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.348+0000 7fef6a79e700 1 -- 172.21.15.189:0/2341450273 shutdown_connections 2020-11-20T12:12:04.357 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:04.348+0000 7fef6a79e700 1 -- 172.21.15.189:0/2341450273 wait complete. 2020-11-20T12:12:04.358 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 28 2020-11-20T12:12:04.368 INFO:tasks.mds_thrash.fs.[None]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 28, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2020-11-20T12:04:54.012110+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 28, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5122": { "addr": "172.21.15.47:6833/4088050649", "addrs": { "addrvec": [ { "addr": "172.21.15.47:6832", "nonce": 4088050649, "type": "v2" }, { "addr": "172.21.15.47:6833", "nonce": 4088050649, "type": "v1" } ] }, "export_targets": [], "features": 4540138297136906239, "flags": 0, "gid": 5122, "incarnation": 27, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 12 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2020-11-20T12:12:03.665053+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5122 } } } ], "standbys": [] } 2020-11-20T12:12:06.370 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:06.482 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a21af8700 1 Processor -- start 2020-11-20T12:12:06.482 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a21af8700 1 -- start start 2020-11-20T12:12:06.484 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a21af8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0a1c0a7d40 0x7f0a1c0a81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:06.485 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a21af8700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0a1c0a9cc0 0x7f0a1c0aa140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:06.485 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a21af8700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a1c0a8fa0 con 0x7f0a1c060310 2020-11-20T12:12:06.485 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a21af8700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a1c0618b0 con 0x7f0a1c0a7d40 2020-11-20T12:12:06.485 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a21af8700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1c0a9240 con 0x7f0a1c0a9cc0 2020-11-20T12:12:06.485 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a1bfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0a1c0a7d40 0x7f0a1c0a81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:06.486 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a1bfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0a1c0a7d40 0x7f0a1c0a81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50278/0 (socket says 172.21.15.189:50278) 2020-11-20T12:12:06.486 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a1bfff700 1 -- 172.21.15.189:0/2750534303 learned_addr learned my addr 172.21.15.189:0/2750534303 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:06.486 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a212f7700 1 --2- 172.21.15.189:0/2750534303 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0a1c0a9cc0 0x7f0a1c0aa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:06.486 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a1bfff700 1 -- 172.21.15.189:0/2750534303 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0a1c0a9cc0 msgr2=0x7f0a1c0aa140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:06.486 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a1bfff700 1 --2- 172.21.15.189:0/2750534303 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0a1c0a9cc0 0x7f0a1c0aa140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:06.487 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a1bfff700 1 -- 172.21.15.189:0/2750534303 >> v1:172.21.15.47:6789/0 conn(0x7f0a1c060310 legacy=0x7f0a1c0a1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:06.487 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a1bfff700 1 -- 172.21.15.189:0/2750534303 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a1c0a94e0 con 0x7f0a1c0a7d40 2020-11-20T12:12:06.487 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a212f7700 1 --2- 172.21.15.189:0/2750534303 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0a1c0a9cc0 0x7f0a1c0aa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:12:06.487 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a1bfff700 1 --2- 172.21.15.189:0/2750534303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a1c0a7d40 0x7f0a1c0a81b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f0a08006fd0 tx=0x7f0a08006eb0).ready entity=mon.1 client_cookie=94c75c1a85c49350 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:06.487 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a1b7fe700 1 -- 172.21.15.189:0/2750534303 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a08009960 con 0x7f0a1c0a7d40 2020-11-20T12:12:06.488 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a1b7fe700 1 -- 172.21.15.189:0/2750534303 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a08009ac0 con 0x7f0a1c0a7d40 2020-11-20T12:12:06.488 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a1b7fe700 1 -- 172.21.15.189:0/2750534303 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a08009e40 con 0x7f0a1c0a7d40 2020-11-20T12:12:06.488 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a21af8700 1 -- 172.21.15.189:0/2750534303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a1c0a7d40 msgr2=0x7f0a1c0a81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:06.488 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a21af8700 1 --2- 172.21.15.189:0/2750534303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a1c0a7d40 0x7f0a1c0a81b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f0a08006fd0 tx=0x7f0a08006eb0).stop 2020-11-20T12:12:06.488 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a21af8700 1 -- 172.21.15.189:0/2750534303 shutdown_connections 2020-11-20T12:12:06.489 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a21af8700 1 --2- 172.21.15.189:0/2750534303 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0a1c0a9cc0 0x7f0a1c0aa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:06.489 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a21af8700 1 --2- 172.21.15.189:0/2750534303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a1c0a7d40 0x7f0a1c0a81b0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:06.489 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a21af8700 1 -- 172.21.15.189:0/2750534303 >> 172.21.15.189:0/2750534303 conn(0x7f0a1c063210 msgr2=0x7f0a1c060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:06.489 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a21af8700 1 -- 172.21.15.189:0/2750534303 shutdown_connections 2020-11-20T12:12:06.490 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.475+0000 7f0a21af8700 1 -- 172.21.15.189:0/2750534303 wait complete. 2020-11-20T12:12:06.492 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.479+0000 7f0a21af8700 1 Processor -- start 2020-11-20T12:12:06.492 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.479+0000 7f0a21af8700 1 -- start start 2020-11-20T12:12:06.492 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.479+0000 7f0a21af8700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a1c060310 0x7f0a1c09c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:06.492 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.479+0000 7f0a21af8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a1c0a7d40 0x7f0a1c14b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:06.493 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.479+0000 7f0a21af8700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0a1c0a9cc0 0x7f0a1c13e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:06.493 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.479+0000 7f0a21af8700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1c14be00 con 0x7f0a1c060310 2020-11-20T12:12:06.493 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.479+0000 7f0a21af8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1c14af30 con 0x7f0a1c0a7d40 2020-11-20T12:12:06.493 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.479+0000 7f0a21af8700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1c14b1d0 con 0x7f0a1c0a9cc0 2020-11-20T12:12:06.494 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.479+0000 7f0a1bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a1c0a7d40 0x7f0a1c14b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:06.494 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.479+0000 7f0a1bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a1c0a7d40 0x7f0a1c14b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50284/0 (socket says 172.21.15.189:50284) 2020-11-20T12:12:06.494 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.479+0000 7f0a1bfff700 1 -- 172.21.15.189:0/390763797 learned_addr learned my addr 172.21.15.189:0/390763797 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:06.494 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.479+0000 7f0a20af6700 1 --2- 172.21.15.189:0/390763797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a1c060310 0x7f0a1c09c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:06.494 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.479+0000 7f0a212f7700 1 --2- 172.21.15.189:0/390763797 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0a1c0a9cc0 0x7f0a1c13e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:06.495 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.479+0000 7f0a1bfff700 1 -- 172.21.15.189:0/390763797 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0a1c0a9cc0 msgr2=0x7f0a1c13e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:06.495 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.479+0000 7f0a1bfff700 1 --2- 172.21.15.189:0/390763797 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0a1c0a9cc0 0x7f0a1c13e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:06.495 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.479+0000 7f0a1bfff700 1 -- 172.21.15.189:0/390763797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a1c060310 msgr2=0x7f0a1c09c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:06.495 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.479+0000 7f0a1bfff700 1 --2- 172.21.15.189:0/390763797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a1c060310 0x7f0a1c09c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:06.495 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.479+0000 7f0a1bfff700 1 -- 172.21.15.189:0/390763797 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a1c14b470 con 0x7f0a1c0a7d40 2020-11-20T12:12:06.495 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.479+0000 7f0a20af6700 1 --2- 172.21.15.189:0/390763797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a1c060310 0x7f0a1c09c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:12:06.496 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.479+0000 7f0a1bfff700 1 --2- 172.21.15.189:0/390763797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a1c0a7d40 0x7f0a1c14b910 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f0a1c060980 tx=0x7f0a08015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:06.496 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.479+0000 7f0a18ff9700 1 -- 172.21.15.189:0/390763797 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a08006890 con 0x7f0a1c0a7d40 2020-11-20T12:12:06.496 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.479+0000 7f0a18ff9700 1 -- 172.21.15.189:0/390763797 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a08011dd0 con 0x7f0a1c0a7d40 2020-11-20T12:12:06.496 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.479+0000 7f0a21af8700 1 -- 172.21.15.189:0/390763797 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a1c13e7c0 con 0x7f0a1c0a7d40 2020-11-20T12:12:06.496 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.479+0000 7f0a18ff9700 1 -- 172.21.15.189:0/390763797 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0801e670 con 0x7f0a1c0a7d40 2020-11-20T12:12:06.497 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.479+0000 7f0a21af8700 1 -- 172.21.15.189:0/390763797 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a1c13a8f0 con 0x7f0a1c0a7d40 2020-11-20T12:12:06.497 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.479+0000 7f0a21af8700 1 -- 172.21.15.189:0/390763797 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a1c063c30 con 0x7f0a1c0a7d40 2020-11-20T12:12:06.497 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.483+0000 7f0a18ff9700 1 -- 172.21.15.189:0/390763797 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f0a08025030 con 0x7f0a1c0a7d40 2020-11-20T12:12:06.497 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.483+0000 7f0a18ff9700 1 --2- 172.21.15.189:0/390763797 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f09f40557c0 0x7f09f4057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:06.497 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.483+0000 7f0a18ff9700 1 -- 172.21.15.189:0/390763797 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f0a08070120 con 0x7f0a1c0a7d40 2020-11-20T12:12:06.498 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.483+0000 7f0a20af6700 1 --2- 172.21.15.189:0/390763797 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f09f40557c0 0x7f09f4057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:06.498 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.483+0000 7f0a20af6700 1 --2- 172.21.15.189:0/390763797 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f09f40557c0 0x7f09f4057c80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f0a1c135180 tx=0x7f0a10009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:06.498 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.491+0000 7f0a18ff9700 1 -- 172.21.15.189:0/390763797 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0a1c063c30 con 0x7f0a1c0a7d40 2020-11-20T12:12:06.671 INFO:tasks.ceph.mds.b.smithi047.stderr:2020-11-20T12:12:06.668+0000 7f90a98a9700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:12:06.712 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.703+0000 7f0a21af8700 1 -- 172.21.15.189:0/390763797 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a1c14b650 con 0x7f0a1c0a7d40 2020-11-20T12:12:06.713 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.703+0000 7f0a18ff9700 1 -- 172.21.15.189:0/390763797 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+1785 (secure 0 0 0) 0x7f0a1c14b650 con 0x7f0a1c0a7d40 2020-11-20T12:12:06.713 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:12:06.713 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:05.684503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5122":{"gid":5122,"name":"b","rank":0,"incarnation":27,"state":"up:active","state_seq":14,"addr":"172.21.15.47:6833/4088050649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4088050649},{"type":"v1","addr":"172.21.15.47:6833","nonce":4088050649}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:12:06.715 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.707+0000 7f0a21af8700 1 -- 172.21.15.189:0/390763797 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f09f40557c0 msgr2=0x7f09f4057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:06.715 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.707+0000 7f0a21af8700 1 --2- 172.21.15.189:0/390763797 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f09f40557c0 0x7f09f4057c80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f0a1c135180 tx=0x7f0a10009040).stop 2020-11-20T12:12:06.715 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.707+0000 7f0a21af8700 1 -- 172.21.15.189:0/390763797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a1c0a7d40 msgr2=0x7f0a1c14b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:06.715 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.707+0000 7f0a21af8700 1 --2- 172.21.15.189:0/390763797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a1c0a7d40 0x7f0a1c14b910 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f0a1c060980 tx=0x7f0a08015000).stop 2020-11-20T12:12:06.716 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.707+0000 7f0a21af8700 1 -- 172.21.15.189:0/390763797 shutdown_connections 2020-11-20T12:12:06.716 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.707+0000 7f0a21af8700 1 --2- 172.21.15.189:0/390763797 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f09f40557c0 0x7f09f4057c80 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:06.716 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.707+0000 7f0a21af8700 1 --2- 172.21.15.189:0/390763797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0a1c060310 0x7f0a1c09c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:06.716 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.707+0000 7f0a21af8700 1 --2- 172.21.15.189:0/390763797 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0a1c0a9cc0 0x7f0a1c13e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:06.716 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.707+0000 7f0a21af8700 1 --2- 172.21.15.189:0/390763797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a1c0a7d40 0x7f0a1c14b910 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:06.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.707+0000 7f0a21af8700 1 -- 172.21.15.189:0/390763797 >> 172.21.15.189:0/390763797 conn(0x7f0a1c063210 msgr2=0x7f0a1c060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:06.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.707+0000 7f0a21af8700 1 -- 172.21.15.189:0/390763797 shutdown_connections 2020-11-20T12:12:06.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:06.707+0000 7f0a21af8700 1 -- 172.21.15.189:0/390763797 wait complete. 2020-11-20T12:12:06.718 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 30 2020-11-20T12:12:06.735 INFO:tasks.mds_thrash.fs.[None]:mds.b has gained rank=0, replacing gid=5053 2020-11-20T12:12:06.735 INFO:tasks.mds_thrash.fs.[None]:waiting for 3 secs before reviving mds.a 2020-11-20T12:12:09.736 INFO:tasks.mds_thrash.fs.[None]:reviving mds.a 2020-11-20T12:12:09.736 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:12:09.737 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:12:09.740 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:12:09.741 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:09.819 INFO:tasks.ceph.mds.a.smithi189.stdout:starting mds.a at 2020-11-20T12:12:09.875 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.867+0000 7f896f59e700 1 Processor -- start 2020-11-20T12:12:09.875 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.867+0000 7f896f59e700 1 -- start start 2020-11-20T12:12:09.877 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.867+0000 7f896f59e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f89700623c0 0x7f89700677f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:09.877 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.867+0000 7f896f59e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f897013aac0 0x7f897013af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:09.877 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.867+0000 7f896f59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f897013b430 con 0x7f89701340e0 2020-11-20T12:12:09.877 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.867+0000 7f896f59e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89700a8e60 con 0x7f89700623c0 2020-11-20T12:12:09.877 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.867+0000 7f896f59e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89700a9c60 con 0x7f897013aac0 2020-11-20T12:12:09.878 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.867+0000 7f896ed9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f89701340e0 0x7f8970134570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51926/0 (socket says 172.21.15.189:51926) 2020-11-20T12:12:09.878 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.867+0000 7f896ed9d700 1 -- 172.21.15.189:0/3031938055 learned_addr learned my addr 172.21.15.189:0/3031938055 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:12:09.878 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.867+0000 7f896e59c700 1 --2- 172.21.15.189:0/3031938055 >> v2:172.21.15.189:3300/0 conn(0x7f89700623c0 0x7f89700677f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:09.878 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.867+0000 7f896dd9b700 1 --2- 172.21.15.189:0/3031938055 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f897013aac0 0x7f897013af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:09.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896e59c700 1 -- 172.21.15.189:0/3031938055 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f897013aac0 msgr2=0x7f897013af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:09.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896e59c700 1 --2- 172.21.15.189:0/3031938055 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f897013aac0 0x7f897013af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:09.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896e59c700 1 -- 172.21.15.189:0/3031938055 >> v1:172.21.15.189:6789/0 conn(0x7f89701340e0 legacy=0x7f8970134570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:09.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896d59a700 1 -- 172.21.15.189:0/3031938055 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001515649 0 0) 0x7f897013b430 con 0x7f89701340e0 2020-11-20T12:12:09.881 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896e59c700 1 -- 172.21.15.189:0/3031938055 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8970067e90 con 0x7f89700623c0 2020-11-20T12:12:09.881 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896e59c700 1 --2- 172.21.15.189:0/3031938055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89700623c0 0x7f89700677f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f896000b9a0 tx=0x7f8960006df0).ready entity=mon.1 client_cookie=fb5cce0f264952f4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:09.881 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896d59a700 1 -- 172.21.15.189:0/3031938055 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89600097b0 con 0x7f89700623c0 2020-11-20T12:12:09.881 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896d59a700 1 -- 172.21.15.189:0/3031938055 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8960009910 con 0x7f89700623c0 2020-11-20T12:12:09.882 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896d59a700 1 -- 172.21.15.189:0/3031938055 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8960009c90 con 0x7f89700623c0 2020-11-20T12:12:09.882 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896f59e700 1 -- 172.21.15.189:0/3031938055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89700623c0 msgr2=0x7f89700677f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:09.882 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896f59e700 1 --2- 172.21.15.189:0/3031938055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89700623c0 0x7f89700677f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f896000b9a0 tx=0x7f8960006df0).stop 2020-11-20T12:12:09.882 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896f59e700 1 -- 172.21.15.189:0/3031938055 shutdown_connections 2020-11-20T12:12:09.882 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896f59e700 1 --2- 172.21.15.189:0/3031938055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89700623c0 0x7f89700677f0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:09.882 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896f59e700 1 --2- 172.21.15.189:0/3031938055 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f897013aac0 0x7f897013af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:09.883 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896f59e700 1 -- 172.21.15.189:0/3031938055 >> 172.21.15.189:0/3031938055 conn(0x7f8970065fe0 msgr2=0x7f8970061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:09.883 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896f59e700 1 -- 172.21.15.189:0/3031938055 shutdown_connections 2020-11-20T12:12:09.883 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896f59e700 1 -- 172.21.15.189:0/3031938055 wait complete. 2020-11-20T12:12:09.883 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896f59e700 1 Processor -- start 2020-11-20T12:12:09.883 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896f59e700 1 -- start start 2020-11-20T12:12:09.886 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896f59e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f89701340e0 0x7f8970148300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:09.886 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896f59e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f897013aac0 0x7f89701360c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:09.886 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896f59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89700ae150 0x7f8970138a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:09.886 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896f59e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8970147730 con 0x7f897013aac0 2020-11-20T12:12:09.887 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896f59e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89700ae590 con 0x7f89700ae150 2020-11-20T12:12:09.887 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896f59e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8970147920 con 0x7f89701340e0 2020-11-20T12:12:09.887 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896ed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89700ae150 0x7f8970138a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:09.887 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896ed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89700ae150 0x7f8970138a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50312/0 (socket says 172.21.15.189:50312) 2020-11-20T12:12:09.887 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896ed9d700 1 -- 172.21.15.189:0/837573261 learned_addr learned my addr 172.21.15.189:0/837573261 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:09.888 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896e59c700 1 --2- 172.21.15.189:0/837573261 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f89701340e0 0x7f8970148300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:09.888 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.871+0000 7f896dd9b700 1 --2- 172.21.15.189:0/837573261 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f897013aac0 0x7f89701360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:09.888 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.875+0000 7f896ed9d700 1 -- 172.21.15.189:0/837573261 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f89701340e0 msgr2=0x7f8970148300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:09.888 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.875+0000 7f896ed9d700 1 --2- 172.21.15.189:0/837573261 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f89701340e0 0x7f8970148300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:09.888 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.875+0000 7f896ed9d700 1 -- 172.21.15.189:0/837573261 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f897013aac0 msgr2=0x7f89701360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:09.889 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.875+0000 7f896ed9d700 1 --2- 172.21.15.189:0/837573261 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f897013aac0 0x7f89701360c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:09.889 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.875+0000 7f896ed9d700 1 -- 172.21.15.189:0/837573261 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8970147bc0 con 0x7f89700ae150 2020-11-20T12:12:09.889 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.875+0000 7f896ed9d700 1 --2- 172.21.15.189:0/837573261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89700ae150 0x7f8970138a40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f896800cae0 tx=0x7f8968007de0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:09.889 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.875+0000 7f895e7fc700 1 -- 172.21.15.189:0/837573261 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8968005b80 con 0x7f89700ae150 2020-11-20T12:12:09.889 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.875+0000 7f896f59e700 1 -- 172.21.15.189:0/837573261 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8970147e60 con 0x7f89700ae150 2020-11-20T12:12:09.890 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.875+0000 7f895e7fc700 1 -- 172.21.15.189:0/837573261 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8968005ce0 con 0x7f89700ae150 2020-11-20T12:12:09.890 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.875+0000 7f895e7fc700 1 -- 172.21.15.189:0/837573261 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8968007340 con 0x7f89700ae150 2020-11-20T12:12:09.890 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.875+0000 7f896f59e700 1 -- 172.21.15.189:0/837573261 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8970138fb0 con 0x7f89700ae150 2020-11-20T12:12:09.890 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.875+0000 7f896f59e700 1 -- 172.21.15.189:0/837573261 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8970147ff0 con 0x7f89700ae150 2020-11-20T12:12:09.890 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.875+0000 7f895e7fc700 1 -- 172.21.15.189:0/837573261 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f8968010030 con 0x7f89700ae150 2020-11-20T12:12:09.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.875+0000 7f895e7fc700 1 --2- 172.21.15.189:0/837573261 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f89440557c0 0x7f8944057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:09.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.875+0000 7f895e7fc700 1 -- 172.21.15.189:0/837573261 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f896806e8d0 con 0x7f89700ae150 2020-11-20T12:12:09.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.875+0000 7f896e59c700 1 --2- 172.21.15.189:0/837573261 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f89440557c0 0x7f8944057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:09.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.875+0000 7f896e59c700 1 --2- 172.21.15.189:0/837573261 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f89440557c0 0x7f8944057c80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f8970065d10 tx=0x7f8960015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:09.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:09.879+0000 7f895e7fc700 1 -- 172.21.15.189:0/837573261 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f89680445c0 con 0x7f89700ae150 2020-11-20T12:12:10.097 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:10.087+0000 7f896f59e700 1 -- 172.21.15.189:0/837573261 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89700663c0 con 0x7f89700ae150 2020-11-20T12:12:10.098 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:10.091+0000 7f895e7fc700 1 -- 172.21.15.189:0/837573261 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+1785 (secure 0 0 0) 0x7f8968047be0 con 0x7f89700ae150 2020-11-20T12:12:10.098 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:12:10.098 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:05.684503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5122":{"gid":5122,"name":"b","rank":0,"incarnation":27,"state":"up:active","state_seq":14,"addr":"172.21.15.47:6833/4088050649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4088050649},{"type":"v1","addr":"172.21.15.47:6833","nonce":4088050649}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:12:10.099 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:10.091+0000 7f896f59e700 1 -- 172.21.15.189:0/837573261 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f89440557c0 msgr2=0x7f8944057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:10.100 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:10.091+0000 7f896f59e700 1 --2- 172.21.15.189:0/837573261 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f89440557c0 0x7f8944057c80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f8970065d10 tx=0x7f8960015000).stop 2020-11-20T12:12:10.100 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:10.091+0000 7f896f59e700 1 -- 172.21.15.189:0/837573261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89700ae150 msgr2=0x7f8970138a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:10.100 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:10.091+0000 7f896f59e700 1 --2- 172.21.15.189:0/837573261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89700ae150 0x7f8970138a40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f896800cae0 tx=0x7f8968007de0).stop 2020-11-20T12:12:10.100 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:10.091+0000 7f896f59e700 1 -- 172.21.15.189:0/837573261 shutdown_connections 2020-11-20T12:12:10.101 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:10.091+0000 7f896f59e700 1 --2- 172.21.15.189:0/837573261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89700ae150 0x7f8970138a40 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:10.101 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:10.091+0000 7f896f59e700 1 --2- 172.21.15.189:0/837573261 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f89701340e0 0x7f8970148300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:10.101 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:10.091+0000 7f896f59e700 1 --2- 172.21.15.189:0/837573261 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f89440557c0 0x7f8944057c80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:10.101 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:10.091+0000 7f896f59e700 1 --2- 172.21.15.189:0/837573261 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f897013aac0 0x7f89701360c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:10.102 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:10.091+0000 7f896f59e700 1 -- 172.21.15.189:0/837573261 >> 172.21.15.189:0/837573261 conn(0x7f8970065fe0 msgr2=0x7f8970130f90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:10.102 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:10.091+0000 7f896f59e700 1 -- 172.21.15.189:0/837573261 shutdown_connections 2020-11-20T12:12:10.102 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:10.091+0000 7f896f59e700 1 -- 172.21.15.189:0/837573261 wait complete. 2020-11-20T12:12:10.103 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 30 2020-11-20T12:12:10.113 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2020-11-20T12:12:12.114 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:12.254 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f95454700 1 Processor -- start 2020-11-20T12:12:12.255 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f95454700 1 -- start start 2020-11-20T12:12:12.257 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f95454700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1f900a9e20 0x7f1f900a9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:12.257 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f95454700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1f900a7bf0 0x7f1f900a0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:12.257 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f95454700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f900a5c70 con 0x7f1f90135400 2020-11-20T12:12:12.257 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f95454700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f900a5730 con 0x7f1f900a9e20 2020-11-20T12:12:12.257 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f95454700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f900a59d0 con 0x7f1f900a7bf0 2020-11-20T12:12:12.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f8e7fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1f900a7bf0 0x7f1f900a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:12.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f8e7fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1f900a7bf0 0x7f1f900a0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50318/0 (socket says 172.21.15.189:50318) 2020-11-20T12:12:12.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f8e7fc700 1 -- 172.21.15.189:0/1301770668 learned_addr learned my addr 172.21.15.189:0/1301770668 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:12.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f8effd700 1 --2- 172.21.15.189:0/1301770668 >> v2:172.21.15.47:3300/0 conn(0x7f1f900a9e20 0x7f1f900a9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:12.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f8e7fc700 1 -- 172.21.15.189:0/1301770668 >> v2:172.21.15.47:3300/0 conn(0x7f1f900a9e20 msgr2=0x7f1f900a9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:12.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f8e7fc700 1 --2- 172.21.15.189:0/1301770668 >> v2:172.21.15.47:3300/0 conn(0x7f1f900a9e20 0x7f1f900a9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:12.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f8e7fc700 1 -- 172.21.15.189:0/1301770668 >> v1:172.21.15.47:6789/0 conn(0x7f1f90135400 legacy=0x7f1f90135870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:12.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f8e7fc700 1 -- 172.21.15.189:0/1301770668 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f900aa200 con 0x7f1f900a7bf0 2020-11-20T12:12:12.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f8effd700 1 --2- 172.21.15.189:0/1301770668 >> v2:172.21.15.47:3300/0 conn(0x7f1f900a9e20 0x7f1f900a9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:12.260 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f8e7fc700 1 --2- 172.21.15.189:0/1301770668 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f900a7bf0 0x7f1f900a0770 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f1f900ae2c0 tx=0x7f1f84006e50).ready entity=mon.1 client_cookie=1e113c8582f42f95 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:12.260 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f8dffb700 1 -- 172.21.15.189:0/1301770668 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f84006030 con 0x7f1f900a7bf0 2020-11-20T12:12:12.260 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f8dffb700 1 -- 172.21.15.189:0/1301770668 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f84006190 con 0x7f1f900a7bf0 2020-11-20T12:12:12.260 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f8dffb700 1 -- 172.21.15.189:0/1301770668 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f84006510 con 0x7f1f900a7bf0 2020-11-20T12:12:12.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f95454700 1 -- 172.21.15.189:0/1301770668 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f900a7bf0 msgr2=0x7f1f900a0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:12.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f95454700 1 --2- 172.21.15.189:0/1301770668 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f900a7bf0 0x7f1f900a0770 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f1f900ae2c0 tx=0x7f1f84006e50).stop 2020-11-20T12:12:12.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f95454700 1 -- 172.21.15.189:0/1301770668 shutdown_connections 2020-11-20T12:12:12.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f95454700 1 --2- 172.21.15.189:0/1301770668 >> v2:172.21.15.47:3300/0 conn(0x7f1f900a9e20 0x7f1f900a9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:12.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f95454700 1 --2- 172.21.15.189:0/1301770668 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f900a7bf0 0x7f1f900a0770 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:12.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f95454700 1 -- 172.21.15.189:0/1301770668 >> 172.21.15.189:0/1301770668 conn(0x7f1f900627f0 msgr2=0x7f1f90065870 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:12.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f95454700 1 -- 172.21.15.189:0/1301770668 shutdown_connections 2020-11-20T12:12:12.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.247+0000 7f1f95454700 1 -- 172.21.15.189:0/1301770668 wait complete. 2020-11-20T12:12:12.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.251+0000 7f1f95454700 1 Processor -- start 2020-11-20T12:12:12.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.251+0000 7f1f95454700 1 -- start start 2020-11-20T12:12:12.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.251+0000 7f1f95454700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1f900a7bf0 0x7f1f90138680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:12.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.251+0000 7f1f95454700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f900a9e20 0x7f1f900a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:12.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.251+0000 7f1f95454700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f90135400 0x7f1f90134550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:12.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.251+0000 7f1f95454700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f900a2d20 con 0x7f1f900a9e20 2020-11-20T12:12:12.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.251+0000 7f1f95454700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f900a2f10 con 0x7f1f90135400 2020-11-20T12:12:12.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.251+0000 7f1f95454700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f900a31b0 con 0x7f1f900a7bf0 2020-11-20T12:12:12.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.251+0000 7f1f8f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f90135400 0x7f1f90134550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:12.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.251+0000 7f1f8f7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f90135400 0x7f1f90134550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50326/0 (socket says 172.21.15.189:50326) 2020-11-20T12:12:12.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.251+0000 7f1f8f7fe700 1 -- 172.21.15.189:0/1613549782 learned_addr learned my addr 172.21.15.189:0/1613549782 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:12.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.251+0000 7f1f8effd700 1 --2- 172.21.15.189:0/1613549782 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1f900a7bf0 0x7f1f90138680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:12.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.251+0000 7f1f8e7fc700 1 --2- 172.21.15.189:0/1613549782 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f900a9e20 0x7f1f900a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:12.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.251+0000 7f1f8f7fe700 1 -- 172.21.15.189:0/1613549782 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1f900a7bf0 msgr2=0x7f1f90138680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:12.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.251+0000 7f1f8f7fe700 1 --2- 172.21.15.189:0/1613549782 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1f900a7bf0 0x7f1f90138680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:12.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.251+0000 7f1f8f7fe700 1 -- 172.21.15.189:0/1613549782 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f900a9e20 msgr2=0x7f1f900a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:12.268 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.251+0000 7f1f8f7fe700 1 --2- 172.21.15.189:0/1613549782 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f900a9e20 0x7f1f900a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:12.269 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.251+0000 7f1f8f7fe700 1 -- 172.21.15.189:0/1613549782 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f900a3450 con 0x7f1f90135400 2020-11-20T12:12:12.269 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.251+0000 7f1f8f7fe700 1 --2- 172.21.15.189:0/1613549782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f90135400 0x7f1f90134550 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f1f8800bb10 tx=0x7f1f880090f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:12.269 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.251+0000 7f1f76ffd700 1 -- 172.21.15.189:0/1613549782 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8800c830 con 0x7f1f90135400 2020-11-20T12:12:12.269 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.251+0000 7f1f76ffd700 1 -- 172.21.15.189:0/1613549782 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8800c9b0 con 0x7f1f90135400 2020-11-20T12:12:12.269 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.251+0000 7f1f95454700 1 -- 172.21.15.189:0/1613549782 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f900a4050 con 0x7f1f90135400 2020-11-20T12:12:12.270 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.251+0000 7f1f76ffd700 1 -- 172.21.15.189:0/1613549782 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8800cb60 con 0x7f1f90135400 2020-11-20T12:12:12.270 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.251+0000 7f1f95454700 1 -- 172.21.15.189:0/1613549782 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f9013a9a0 con 0x7f1f90135400 2020-11-20T12:12:12.270 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.255+0000 7f1f95454700 1 -- 172.21.15.189:0/1613549782 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f90044f60 con 0x7f1f90135400 2020-11-20T12:12:12.270 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.255+0000 7f1f76ffd700 1 -- 172.21.15.189:0/1613549782 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f1f8801c070 con 0x7f1f90135400 2020-11-20T12:12:12.270 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.255+0000 7f1f76ffd700 1 --2- 172.21.15.189:0/1613549782 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1f6c0557c0 0x7f1f6c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:12.271 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.255+0000 7f1f76ffd700 1 -- 172.21.15.189:0/1613549782 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f1f8806dea0 con 0x7f1f90135400 2020-11-20T12:12:12.271 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.255+0000 7f1f8effd700 1 --2- 172.21.15.189:0/1613549782 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1f6c0557c0 0x7f1f6c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:12.272 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.255+0000 7f1f8effd700 1 --2- 172.21.15.189:0/1613549782 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1f6c0557c0 0x7f1f6c057c80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f1f8000a8c0 tx=0x7f1f80008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:12.272 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.263+0000 7f1f76ffd700 1 -- 172.21.15.189:0/1613549782 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1f88043b90 con 0x7f1f90135400 2020-11-20T12:12:12.516 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.507+0000 7f1f95454700 1 -- 172.21.15.189:0/1613549782 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f900a35e0 con 0x7f1f90135400 2020-11-20T12:12:12.517 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.507+0000 7f1f76ffd700 1 -- 172.21.15.189:0/1613549782 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+1785 (secure 0 0 0) 0x7f1f900a35e0 con 0x7f1f90135400 2020-11-20T12:12:12.517 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:12:12.518 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:05.684503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5122":{"gid":5122,"name":"b","rank":0,"incarnation":27,"state":"up:active","state_seq":14,"addr":"172.21.15.47:6833/4088050649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4088050649},{"type":"v1","addr":"172.21.15.47:6833","nonce":4088050649}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:12:12.519 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.511+0000 7f1f95454700 1 -- 172.21.15.189:0/1613549782 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1f6c0557c0 msgr2=0x7f1f6c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:12.519 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.511+0000 7f1f95454700 1 --2- 172.21.15.189:0/1613549782 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1f6c0557c0 0x7f1f6c057c80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f1f8000a8c0 tx=0x7f1f80008040).stop 2020-11-20T12:12:12.520 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.511+0000 7f1f95454700 1 -- 172.21.15.189:0/1613549782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f90135400 msgr2=0x7f1f90134550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:12.520 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.511+0000 7f1f95454700 1 --2- 172.21.15.189:0/1613549782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f90135400 0x7f1f90134550 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f1f8800bb10 tx=0x7f1f880090f0).stop 2020-11-20T12:12:12.520 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.511+0000 7f1f95454700 1 -- 172.21.15.189:0/1613549782 shutdown_connections 2020-11-20T12:12:12.520 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.511+0000 7f1f95454700 1 --2- 172.21.15.189:0/1613549782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f90135400 0x7f1f90134550 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:12.520 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.511+0000 7f1f95454700 1 --2- 172.21.15.189:0/1613549782 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1f900a7bf0 0x7f1f90138680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:12.521 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.511+0000 7f1f95454700 1 --2- 172.21.15.189:0/1613549782 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1f6c0557c0 0x7f1f6c057c80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:12.521 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.511+0000 7f1f95454700 1 --2- 172.21.15.189:0/1613549782 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f900a9e20 0x7f1f900a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:12.521 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.511+0000 7f1f95454700 1 -- 172.21.15.189:0/1613549782 >> 172.21.15.189:0/1613549782 conn(0x7f1f900627f0 msgr2=0x7f1f900acd00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:12.521 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.511+0000 7f1f95454700 1 -- 172.21.15.189:0/1613549782 shutdown_connections 2020-11-20T12:12:12.521 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:12.511+0000 7f1f95454700 1 -- 172.21.15.189:0/1613549782 wait complete. 2020-11-20T12:12:12.522 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 30 2020-11-20T12:12:12.532 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2020-11-20T12:12:14.533 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:14.652 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.643+0000 7f3faaf74700 1 Processor -- start 2020-11-20T12:12:14.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.643+0000 7f3faaf74700 1 -- start start 2020-11-20T12:12:14.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.647+0000 7f3faaf74700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f3f9c03af50 0x7f3f9c0c5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:14.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.647+0000 7f3faaf74700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3f9c0cc9a0 0x7f3f9c0c5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:14.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.647+0000 7f3faaf74700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f9c039480 con 0x7f3f9c0075c0 2020-11-20T12:12:14.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.647+0000 7f3faaf74700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f9c039fa0 con 0x7f3f9c03af50 2020-11-20T12:12:14.656 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.647+0000 7f3faaf74700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f9c02a4d0 con 0x7f3f9c0cc9a0 2020-11-20T12:12:14.656 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.647+0000 7f3faa773700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3f9c0cc9a0 0x7f3f9c0c5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:14.656 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.647+0000 7f3faa773700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3f9c0cc9a0 0x7f3f9c0c5e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50336/0 (socket says 172.21.15.189:50336) 2020-11-20T12:12:14.656 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.647+0000 7f3faa773700 1 -- 172.21.15.189:0/2937613866 learned_addr learned my addr 172.21.15.189:0/2937613866 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:14.656 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.647+0000 7f3fa9771700 1 --2- 172.21.15.189:0/2937613866 >> v2:172.21.15.47:3300/0 conn(0x7f3f9c03af50 0x7f3f9c0c5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:14.657 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.647+0000 7f3faa773700 1 -- 172.21.15.189:0/2937613866 >> v2:172.21.15.47:3300/0 conn(0x7f3f9c03af50 msgr2=0x7f3f9c0c5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:14.657 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.647+0000 7f3faa773700 1 --2- 172.21.15.189:0/2937613866 >> v2:172.21.15.47:3300/0 conn(0x7f3f9c03af50 0x7f3f9c0c5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:14.657 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.647+0000 7f3faa773700 1 -- 172.21.15.189:0/2937613866 >> v1:172.21.15.47:6789/0 conn(0x7f3f9c0075c0 legacy=0x7f3f9c0cc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:14.657 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.647+0000 7f3fa8f70700 1 -- 172.21.15.189:0/2937613866 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1838622705 0 0) 0x7f3f9c039480 con 0x7f3f9c0075c0 2020-11-20T12:12:14.658 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.647+0000 7f3faa773700 1 -- 172.21.15.189:0/2937613866 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f9c038f40 con 0x7f3f9c0cc9a0 2020-11-20T12:12:14.658 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.647+0000 7f3faa773700 1 --2- 172.21.15.189:0/2937613866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9c0cc9a0 0x7f3f9c0c5e40 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f3f9c00be50 tx=0x7f3fa405d940).ready entity=mon.1 client_cookie=a930cf6f6244bf78 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:14.658 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.647+0000 7f3fa8f70700 1 -- 172.21.15.189:0/2937613866 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa405cd20 con 0x7f3f9c0cc9a0 2020-11-20T12:12:14.658 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.647+0000 7f3fa8f70700 1 -- 172.21.15.189:0/2937613866 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa405ce80 con 0x7f3f9c0cc9a0 2020-11-20T12:12:14.658 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.647+0000 7f3fa8f70700 1 -- 172.21.15.189:0/2937613866 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa4068530 con 0x7f3f9c0cc9a0 2020-11-20T12:12:14.661 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.647+0000 7f3faaf74700 1 -- 172.21.15.189:0/2937613866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9c0cc9a0 msgr2=0x7f3f9c0c5e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:14.661 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.647+0000 7f3faaf74700 1 --2- 172.21.15.189:0/2937613866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9c0cc9a0 0x7f3f9c0c5e40 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f3f9c00be50 tx=0x7f3fa405d940).stop 2020-11-20T12:12:14.661 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.647+0000 7f3faaf74700 1 -- 172.21.15.189:0/2937613866 shutdown_connections 2020-11-20T12:12:14.662 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.647+0000 7f3faaf74700 1 --2- 172.21.15.189:0/2937613866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9c0cc9a0 0x7f3f9c0c5e40 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:14.662 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.647+0000 7f3faaf74700 1 --2- 172.21.15.189:0/2937613866 >> v2:172.21.15.47:3300/0 conn(0x7f3f9c03af50 0x7f3f9c0c5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:14.662 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.647+0000 7f3faaf74700 1 -- 172.21.15.189:0/2937613866 >> 172.21.15.189:0/2937613866 conn(0x7f3f9c03e3a0 msgr2=0x7f3f9c03a370 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:14.662 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.647+0000 7f3faaf74700 1 -- 172.21.15.189:0/2937613866 shutdown_connections 2020-11-20T12:12:14.662 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.647+0000 7f3faaf74700 1 -- 172.21.15.189:0/2937613866 wait complete. 2020-11-20T12:12:14.663 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3faaf74700 1 Processor -- start 2020-11-20T12:12:14.663 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3faaf74700 1 -- start start 2020-11-20T12:12:14.663 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3faaf74700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3f9c0075c0 0x7f3f9c0362d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:14.664 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3faaf74700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9c03af50 0x7f3f9c0c2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:14.664 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3faaf74700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3f9c0cc9a0 0x7f3f9c0c3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:14.664 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3faaf74700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c0cabc0 con 0x7f3f9c0cc9a0 2020-11-20T12:12:14.664 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3faaf74700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c0d9900 con 0x7f3f9c03af50 2020-11-20T12:12:14.664 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3faaf74700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9c0ca680 con 0x7f3f9c0075c0 2020-11-20T12:12:14.665 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3fa9771700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9c03af50 0x7f3f9c0c2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:14.665 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3fa9771700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9c03af50 0x7f3f9c0c2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50340/0 (socket says 172.21.15.189:50340) 2020-11-20T12:12:14.665 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3fa9771700 1 -- 172.21.15.189:0/937830170 learned_addr learned my addr 172.21.15.189:0/937830170 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:14.665 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3faa773700 1 --2- 172.21.15.189:0/937830170 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3f9c0cc9a0 0x7f3f9c0c3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:14.666 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3fa9f72700 1 --2- 172.21.15.189:0/937830170 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3f9c0075c0 0x7f3f9c0362d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:14.666 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3fa9771700 1 -- 172.21.15.189:0/937830170 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3f9c0075c0 msgr2=0x7f3f9c0362d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:14.666 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3fa9771700 1 --2- 172.21.15.189:0/937830170 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3f9c0075c0 0x7f3f9c0362d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:14.666 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3fa9771700 1 -- 172.21.15.189:0/937830170 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3f9c0cc9a0 msgr2=0x7f3f9c0c3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:14.666 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3fa9771700 1 --2- 172.21.15.189:0/937830170 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3f9c0cc9a0 0x7f3f9c0c3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:14.667 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3fa9771700 1 -- 172.21.15.189:0/937830170 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f9c0ca920 con 0x7f3f9c03af50 2020-11-20T12:12:14.667 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3faa773700 1 --2- 172.21.15.189:0/937830170 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3f9c0cc9a0 0x7f3f9c0c3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:12:14.667 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3fa9f72700 1 --2- 172.21.15.189:0/937830170 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3f9c0075c0 0x7f3f9c0362d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:12:14.667 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3fa9771700 1 --2- 172.21.15.189:0/937830170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9c03af50 0x7f3f9c0c2f80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f3f9400ebd0 tx=0x7f3f94006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:14.667 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3f91ffb700 1 -- 172.21.15.189:0/937830170 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f9400b9a0 con 0x7f3f9c03af50 2020-11-20T12:12:14.668 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3f91ffb700 1 -- 172.21.15.189:0/937830170 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f9400bb00 con 0x7f3f9c03af50 2020-11-20T12:12:14.668 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3faaf74700 1 -- 172.21.15.189:0/937830170 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f9c0cd580 con 0x7f3f9c03af50 2020-11-20T12:12:14.668 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3f91ffb700 1 -- 172.21.15.189:0/937830170 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f9400be80 con 0x7f3f9c03af50 2020-11-20T12:12:14.668 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3faaf74700 1 -- 172.21.15.189:0/937830170 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f9c0369e0 con 0x7f3f9c03af50 2020-11-20T12:12:14.668 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.651+0000 7f3faaf74700 1 -- 172.21.15.189:0/937830170 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f9c005330 con 0x7f3f9c03af50 2020-11-20T12:12:14.669 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.655+0000 7f3f91ffb700 1 -- 172.21.15.189:0/937830170 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f3f94012600 con 0x7f3f9c03af50 2020-11-20T12:12:14.670 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.655+0000 7f3f91ffb700 1 --2- 172.21.15.189:0/937830170 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3f7c0557c0 0x7f3f7c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:14.670 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.655+0000 7f3f91ffb700 1 -- 172.21.15.189:0/937830170 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f3f9406ebc0 con 0x7f3f9c03af50 2020-11-20T12:12:14.670 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.655+0000 7f3fa9f72700 1 --2- 172.21.15.189:0/937830170 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3f7c0557c0 0x7f3f7c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:14.670 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.655+0000 7f3fa9f72700 1 --2- 172.21.15.189:0/937830170 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3f7c0557c0 0x7f3f7c057c80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f3fa0001510 tx=0x7f3fa0009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:14.670 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.659+0000 7f3f91ffb700 1 -- 172.21.15.189:0/937830170 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3f94044850 con 0x7f3f9c03af50 2020-11-20T12:12:14.887 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.879+0000 7f3faaf74700 1 -- 172.21.15.189:0/937830170 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f9c008eb0 con 0x7f3f9c03af50 2020-11-20T12:12:14.888 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.879+0000 7f3f91ffb700 1 -- 172.21.15.189:0/937830170 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f3f94047e70 con 0x7f3f9c03af50 2020-11-20T12:12:14.888 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:12:14.889 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5203,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.189:6835/2651986971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2651986971},{"type":"v1","addr":"172.21.15.189:6835","nonce":2651986971}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:05.684503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5122":{"gid":5122,"name":"b","rank":0,"incarnation":27,"state":"up:active","state_seq":14,"addr":"172.21.15.47:6833/4088050649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4088050649},{"type":"v1","addr":"172.21.15.47:6833","nonce":4088050649}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:12:14.890 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.883+0000 7f3faaf74700 1 -- 172.21.15.189:0/937830170 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3f7c0557c0 msgr2=0x7f3f7c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:14.890 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.883+0000 7f3faaf74700 1 --2- 172.21.15.189:0/937830170 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3f7c0557c0 0x7f3f7c057c80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f3fa0001510 tx=0x7f3fa0009040).stop 2020-11-20T12:12:14.890 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.883+0000 7f3faaf74700 1 -- 172.21.15.189:0/937830170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9c03af50 msgr2=0x7f3f9c0c2f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:14.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.883+0000 7f3faaf74700 1 --2- 172.21.15.189:0/937830170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9c03af50 0x7f3f9c0c2f80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f3f9400ebd0 tx=0x7f3f94006df0).stop 2020-11-20T12:12:14.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.883+0000 7f3faaf74700 1 -- 172.21.15.189:0/937830170 shutdown_connections 2020-11-20T12:12:14.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.883+0000 7f3faaf74700 1 --2- 172.21.15.189:0/937830170 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3f7c0557c0 0x7f3f7c057c80 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:14.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.883+0000 7f3faaf74700 1 --2- 172.21.15.189:0/937830170 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3f9c0cc9a0 0x7f3f9c0c3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:14.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.883+0000 7f3faaf74700 1 --2- 172.21.15.189:0/937830170 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3f9c0075c0 0x7f3f9c0362d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:14.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.883+0000 7f3faaf74700 1 --2- 172.21.15.189:0/937830170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9c03af50 0x7f3f9c0c2f80 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:14.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.883+0000 7f3faaf74700 1 -- 172.21.15.189:0/937830170 >> 172.21.15.189:0/937830170 conn(0x7f3f9c03e3a0 msgr2=0x7f3f9c004b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:14.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.883+0000 7f3faaf74700 1 -- 172.21.15.189:0/937830170 shutdown_connections 2020-11-20T12:12:14.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:14.883+0000 7f3faaf74700 1 -- 172.21.15.189:0/937830170 wait complete. 2020-11-20T12:12:14.893 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 31 2020-11-20T12:12:14.902 INFO:tasks.mds_thrash.fs.[None]:mds.a reported in up:standby state 2020-11-20T12:12:14.903 INFO:tasks.mds_thrash.fs.[None]:waiting for 6 secs before thrashing 2020-11-20T12:12:20.904 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:21.040 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.027+0000 7f34625ac700 1 Processor -- start 2020-11-20T12:12:21.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.027+0000 7f34625ac700 1 -- start start 2020-11-20T12:12:21.042 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.027+0000 7f34625ac700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f345c0ad3e0 0x7f345c066d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:21.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.027+0000 7f34625ac700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f345c13a6e0 0x7f345c13ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:21.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.027+0000 7f34625ac700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f345c067730 con 0x7f345c0ad000 2020-11-20T12:12:21.044 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.027+0000 7f34625ac700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f345c066050 con 0x7f345c13a6e0 2020-11-20T12:12:21.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.027+0000 7f34625ac700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f345c138190 con 0x7f345c0ad3e0 2020-11-20T12:12:21.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.027+0000 7f34615aa700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f345c0ad000 0x7f345c13e480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:51968/0 (socket says 172.21.15.189:51968) 2020-11-20T12:12:21.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.027+0000 7f34615aa700 1 -- 172.21.15.189:0/164275993 learned_addr learned my addr 172.21.15.189:0/164275993 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:12:21.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.027+0000 7f3460da9700 1 --2- 172.21.15.189:0/164275993 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f345c0ad3e0 0x7f345c066d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:21.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.027+0000 7f3461dab700 1 --2- 172.21.15.189:0/164275993 >> v2:172.21.15.47:3300/0 conn(0x7f345c13a6e0 0x7f345c13ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:21.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.027+0000 7f344bfff700 1 -- 172.21.15.189:0/164275993 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4238416265 0 0) 0x7f345c067730 con 0x7f345c0ad000 2020-11-20T12:12:21.056 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f344bfff700 1 -- 172.21.15.189:0/164275993 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3444003240 con 0x7f345c0ad000 2020-11-20T12:12:21.056 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f344bfff700 1 -- 172.21.15.189:0/164275993 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1490741838 0 0) 0x7f3444003240 con 0x7f345c0ad000 2020-11-20T12:12:21.056 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f344bfff700 1 -- 172.21.15.189:0/164275993 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f345c0ad3e0 msgr2=0x7f345c066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:21.057 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f344bfff700 1 --2- 172.21.15.189:0/164275993 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f345c0ad3e0 0x7f345c066d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:21.057 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f344bfff700 1 -- 172.21.15.189:0/164275993 >> v2:172.21.15.47:3300/0 conn(0x7f345c13a6e0 msgr2=0x7f345c13ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:21.057 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f344bfff700 1 --2- 172.21.15.189:0/164275993 >> v2:172.21.15.47:3300/0 conn(0x7f345c13a6e0 0x7f345c13ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:21.057 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f344bfff700 1 -- 172.21.15.189:0/164275993 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f345c13dd40 con 0x7f345c0ad000 2020-11-20T12:12:21.057 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f344bfff700 1 -- 172.21.15.189:0/164275993 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f34580014e0 con 0x7f345c0ad000 2020-11-20T12:12:21.058 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f344bfff700 1 -- 172.21.15.189:0/164275993 >> v1:172.21.15.189:6789/0 conn(0x7f345c0ad000 legacy=0x7f345c13e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:21.058 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f344bfff700 1 --2- 172.21.15.189:0/164275993 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3444003730 0x7f3444003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:21.058 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f344bfff700 1 --2- 172.21.15.189:0/164275993 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f345c0ad3e0 0x7f34440044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:21.058 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f344bfff700 1 --2- 172.21.15.189:0/164275993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34440049d0 0x7f3444004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:21.058 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f344bfff700 1 -- 172.21.15.189:0/164275993 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345c138190 con 0x7f3444003730 2020-11-20T12:12:21.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f344bfff700 1 -- 172.21.15.189:0/164275993 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34440041d0 con 0x7f34440049d0 2020-11-20T12:12:21.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f344bfff700 1 -- 172.21.15.189:0/164275993 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f344400b730 con 0x7f345c0ad3e0 2020-11-20T12:12:21.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f3461dab700 1 --2- 172.21.15.189:0/164275993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34440049d0 0x7f3444004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:21.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f3460da9700 1 --2- 172.21.15.189:0/164275993 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f345c0ad3e0 0x7f34440044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:21.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f34615aa700 1 --2- 172.21.15.189:0/164275993 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3444003730 0x7f3444003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:21.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f3461dab700 1 -- 172.21.15.189:0/164275993 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f345c0ad3e0 msgr2=0x7f34440044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:21.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f3461dab700 1 --2- 172.21.15.189:0/164275993 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f345c0ad3e0 0x7f34440044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:21.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f3461dab700 1 -- 172.21.15.189:0/164275993 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3444003730 msgr2=0x7f3444003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:21.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f3461dab700 1 --2- 172.21.15.189:0/164275993 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3444003730 0x7f3444003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:21.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f3461dab700 1 -- 172.21.15.189:0/164275993 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3444003240 con 0x7f34440049d0 2020-11-20T12:12:21.061 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f3461dab700 1 --2- 172.21.15.189:0/164275993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34440049d0 0x7f3444004e50 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f3454009e70 tx=0x7f34540091d0).ready entity=mon.1 client_cookie=6d6cdd16d69c6623 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:21.061 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f344bfff700 1 -- 172.21.15.189:0/164275993 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34540095c0 con 0x7f34440049d0 2020-11-20T12:12:21.061 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f344bfff700 1 -- 172.21.15.189:0/164275993 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345400ecf0 con 0x7f34440049d0 2020-11-20T12:12:21.061 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f344bfff700 1 -- 172.21.15.189:0/164275993 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3454012600 con 0x7f34440049d0 2020-11-20T12:12:21.062 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f34625ac700 1 -- 172.21.15.189:0/164275993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34440049d0 msgr2=0x7f3444004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:21.062 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f34625ac700 1 --2- 172.21.15.189:0/164275993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34440049d0 0x7f3444004e50 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f3454009e70 tx=0x7f34540091d0).stop 2020-11-20T12:12:21.062 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f3461dab700 1 -- 172.21.15.189:0/164275993 reap_dead start 2020-11-20T12:12:21.062 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f34625ac700 1 -- 172.21.15.189:0/164275993 shutdown_connections 2020-11-20T12:12:21.062 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f34625ac700 1 -- 172.21.15.189:0/164275993 >> 172.21.15.189:0/164275993 conn(0x7f345c062080 msgr2=0x7f345c062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:21.063 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f34625ac700 1 -- 172.21.15.189:0/164275993 shutdown_connections 2020-11-20T12:12:21.063 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f34625ac700 1 -- 172.21.15.189:0/164275993 wait complete. 2020-11-20T12:12:21.063 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f34625ac700 1 Processor -- start 2020-11-20T12:12:21.063 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.031+0000 7f34625ac700 1 -- start start 2020-11-20T12:12:21.063 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.035+0000 7f34625ac700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f345c0a5e00 0x7f345c0b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:21.064 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.035+0000 7f34625ac700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f345c149ba0 0x7f345c14a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:21.064 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.035+0000 7f34625ac700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f345c0a9400 0x7f345c0a9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:21.064 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.035+0000 7f34625ac700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345c14a510 con 0x7f345c149ba0 2020-11-20T12:12:21.064 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.035+0000 7f34625ac700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345c0aaa10 con 0x7f345c0a9400 2020-11-20T12:12:21.064 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.035+0000 7f34625ac700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f345c0aacb0 con 0x7f345c0a5e00 2020-11-20T12:12:21.065 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.035+0000 7f3461dab700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f345c0a9400 0x7f345c0a9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:21.065 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.035+0000 7f3461dab700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f345c0a9400 0x7f345c0a9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50362/0 (socket says 172.21.15.189:50362) 2020-11-20T12:12:21.066 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.035+0000 7f3461dab700 1 -- 172.21.15.189:0/184773982 learned_addr learned my addr 172.21.15.189:0/184773982 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:21.066 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.035+0000 7f34615aa700 1 --2- 172.21.15.189:0/184773982 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f345c0a5e00 0x7f345c0b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:21.066 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.035+0000 7f3460da9700 1 --2- 172.21.15.189:0/184773982 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f345c149ba0 0x7f345c14a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:21.066 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.035+0000 7f3461dab700 1 -- 172.21.15.189:0/184773982 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f345c0a5e00 msgr2=0x7f345c0b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:21.066 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.035+0000 7f3461dab700 1 --2- 172.21.15.189:0/184773982 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f345c0a5e00 0x7f345c0b00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:21.067 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.035+0000 7f3461dab700 1 -- 172.21.15.189:0/184773982 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f345c149ba0 msgr2=0x7f345c14a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:21.067 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.035+0000 7f3461dab700 1 --2- 172.21.15.189:0/184773982 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f345c149ba0 0x7f345c14a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:21.067 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.035+0000 7f3461dab700 1 -- 172.21.15.189:0/184773982 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f345c13b7d0 con 0x7f345c0a9400 2020-11-20T12:12:21.067 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.035+0000 7f3461dab700 1 --2- 172.21.15.189:0/184773982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f345c0a9400 0x7f345c0a9860 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f3454014000 tx=0x7f345401a000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:21.067 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.035+0000 7f34497fa700 1 -- 172.21.15.189:0/184773982 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345400ee60 con 0x7f345c0a9400 2020-11-20T12:12:21.068 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.035+0000 7f34497fa700 1 -- 172.21.15.189:0/184773982 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345401b630 con 0x7f345c0a9400 2020-11-20T12:12:21.068 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.035+0000 7f34497fa700 1 -- 172.21.15.189:0/184773982 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345401b9d0 con 0x7f345c0a9400 2020-11-20T12:12:21.068 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.035+0000 7f34625ac700 1 -- 172.21.15.189:0/184773982 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f345c0aaf50 con 0x7f345c0a9400 2020-11-20T12:12:21.068 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.035+0000 7f34625ac700 1 -- 172.21.15.189:0/184773982 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f345c0aa5e0 con 0x7f345c0a9400 2020-11-20T12:12:21.068 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.035+0000 7f34625ac700 1 -- 172.21.15.189:0/184773982 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f345c0aa800 con 0x7f345c0a9400 2020-11-20T12:12:21.069 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.035+0000 7f34497fa700 1 -- 172.21.15.189:0/184773982 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f34540253e0 con 0x7f345c0a9400 2020-11-20T12:12:21.069 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.039+0000 7f34497fa700 1 --2- 172.21.15.189:0/184773982 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f34340557c0 0x7f3434057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:21.069 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.039+0000 7f34497fa700 1 -- 172.21.15.189:0/184773982 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f3454002030 con 0x7f345c0a9400 2020-11-20T12:12:21.069 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.039+0000 7f34615aa700 1 --2- 172.21.15.189:0/184773982 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f34340557c0 0x7f3434057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:21.070 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.039+0000 7f34615aa700 1 --2- 172.21.15.189:0/184773982 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f34340557c0 0x7f3434057c80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f3458000bc0 tx=0x7f3458007040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:21.070 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.043+0000 7f34497fa700 1 -- 172.21.15.189:0/184773982 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3454045c80 con 0x7f345c0a9400 2020-11-20T12:12:21.273 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.263+0000 7f34625ac700 1 -- 172.21.15.189:0/184773982 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f345c0b1370 con 0x7f345c0a9400 2020-11-20T12:12:21.273 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.263+0000 7f34497fa700 1 -- 172.21.15.189:0/184773982 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f3454016030 con 0x7f345c0a9400 2020-11-20T12:12:21.274 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:12:21.274 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5203,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.189:6835/2651986971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2651986971},{"type":"v1","addr":"172.21.15.189:6835","nonce":2651986971}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:05.684503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5122":{"gid":5122,"name":"b","rank":0,"incarnation":27,"state":"up:active","state_seq":14,"addr":"172.21.15.47:6833/4088050649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4088050649},{"type":"v1","addr":"172.21.15.47:6833","nonce":4088050649}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:12:21.276 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.267+0000 7f34625ac700 1 -- 172.21.15.189:0/184773982 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f34340557c0 msgr2=0x7f3434057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:21.277 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.267+0000 7f34625ac700 1 --2- 172.21.15.189:0/184773982 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f34340557c0 0x7f3434057c80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f3458000bc0 tx=0x7f3458007040).stop 2020-11-20T12:12:21.277 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.267+0000 7f34625ac700 1 -- 172.21.15.189:0/184773982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f345c0a9400 msgr2=0x7f345c0a9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:21.277 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.267+0000 7f34625ac700 1 --2- 172.21.15.189:0/184773982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f345c0a9400 0x7f345c0a9860 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f3454014000 tx=0x7f345401a000).stop 2020-11-20T12:12:21.277 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.267+0000 7f34625ac700 1 -- 172.21.15.189:0/184773982 shutdown_connections 2020-11-20T12:12:21.278 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.267+0000 7f34625ac700 1 --2- 172.21.15.189:0/184773982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f345c0a9400 0x7f345c0a9860 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:21.278 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.267+0000 7f34625ac700 1 --2- 172.21.15.189:0/184773982 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f345c0a5e00 0x7f345c0b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:21.278 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.267+0000 7f34625ac700 1 --2- 172.21.15.189:0/184773982 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f34340557c0 0x7f3434057c80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:21.278 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.267+0000 7f34625ac700 1 --2- 172.21.15.189:0/184773982 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f345c149ba0 0x7f345c14a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:21.278 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.267+0000 7f34625ac700 1 -- 172.21.15.189:0/184773982 >> 172.21.15.189:0/184773982 conn(0x7f345c062080 msgr2=0x7f345c0ac1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:21.279 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.267+0000 7f34625ac700 1 -- 172.21.15.189:0/184773982 shutdown_connections 2020-11-20T12:12:21.279 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.267+0000 7f34625ac700 1 -- 172.21.15.189:0/184773982 wait complete. 2020-11-20T12:12:21.279 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 31 2020-11-20T12:12:21.293 INFO:tasks.mds_thrash.fs.[None]:kill mds.b (rank=0) 2020-11-20T12:12:21.293 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:12:21.294 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:12:21.312 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:12:21.312 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:21.441 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.431+0000 7f1c1dcc5700 1 Processor -- start 2020-11-20T12:12:21.441 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.431+0000 7f1c1dcc5700 1 -- start start 2020-11-20T12:12:21.443 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c1dcc5700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1c180ad000 0x7f1c1813e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:21.443 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c1dcc5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1c180ad3e0 0x7f1c18066d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:21.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c1dcc5700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1c1813a6e0 0x7f1c1813ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:21.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c1dcc5700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c18066050 con 0x7f1c180ad000 2020-11-20T12:12:21.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c1dcc5700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c18138190 con 0x7f1c180ad3e0 2020-11-20T12:12:21.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c1dcc5700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c18067730 con 0x7f1c1813a6e0 2020-11-20T12:12:21.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c17fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1c180ad3e0 0x7f1c18066d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:21.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c17fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1c180ad3e0 0x7f1c18066d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50370/0 (socket says 172.21.15.189:50370) 2020-11-20T12:12:21.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c17fff700 1 -- 172.21.15.189:0/1303202300 learned_addr learned my addr 172.21.15.189:0/1303202300 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:21.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c1ccc3700 1 --2- 172.21.15.189:0/1303202300 >> v2:172.21.15.47:3300/0 conn(0x7f1c180ad000 0x7f1c1813e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:21.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c1d4c4700 1 --2- 172.21.15.189:0/1303202300 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1c1813a6e0 0x7f1c1813ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:21.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c17fff700 1 -- 172.21.15.189:0/1303202300 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1c1813a6e0 msgr2=0x7f1c1813ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:21.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c17fff700 1 --2- 172.21.15.189:0/1303202300 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1c1813a6e0 0x7f1c1813ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:21.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c17fff700 1 -- 172.21.15.189:0/1303202300 >> v2:172.21.15.47:3300/0 conn(0x7f1c180ad000 msgr2=0x7f1c1813e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:21.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c17fff700 1 --2- 172.21.15.189:0/1303202300 >> v2:172.21.15.47:3300/0 conn(0x7f1c180ad000 0x7f1c1813e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:21.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c17fff700 1 -- 172.21.15.189:0/1303202300 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c1813dd40 con 0x7f1c180ad3e0 2020-11-20T12:12:21.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c1d4c4700 1 --2- 172.21.15.189:0/1303202300 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1c1813a6e0 0x7f1c1813ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:12:21.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c17fff700 1 --2- 172.21.15.189:0/1303202300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c180ad3e0 0x7f1c18066d40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f1c180b18d0 tx=0x7f1c04006eb0).ready entity=mon.1 client_cookie=35dba82ff88fe86b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:21.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c177fe700 1 -- 172.21.15.189:0/1303202300 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c04009960 con 0x7f1c180ad3e0 2020-11-20T12:12:21.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c177fe700 1 -- 172.21.15.189:0/1303202300 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c04009ac0 con 0x7f1c180ad3e0 2020-11-20T12:12:21.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c177fe700 1 -- 172.21.15.189:0/1303202300 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c04009e40 con 0x7f1c180ad3e0 2020-11-20T12:12:21.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c1dcc5700 1 -- 172.21.15.189:0/1303202300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c180ad3e0 msgr2=0x7f1c18066d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:21.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c1dcc5700 1 --2- 172.21.15.189:0/1303202300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c180ad3e0 0x7f1c18066d40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f1c180b18d0 tx=0x7f1c04006eb0).stop 2020-11-20T12:12:21.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c1dcc5700 1 -- 172.21.15.189:0/1303202300 shutdown_connections 2020-11-20T12:12:21.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c1dcc5700 1 --2- 172.21.15.189:0/1303202300 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1c1813a6e0 0x7f1c1813ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:21.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c1dcc5700 1 --2- 172.21.15.189:0/1303202300 >> v2:172.21.15.47:3300/0 conn(0x7f1c180ad000 0x7f1c1813e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:21.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c1dcc5700 1 --2- 172.21.15.189:0/1303202300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c180ad3e0 0x7f1c18066d40 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:21.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c1dcc5700 1 -- 172.21.15.189:0/1303202300 >> 172.21.15.189:0/1303202300 conn(0x7f1c18062080 msgr2=0x7f1c18062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:21.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c1dcc5700 1 -- 172.21.15.189:0/1303202300 shutdown_connections 2020-11-20T12:12:21.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c1dcc5700 1 -- 172.21.15.189:0/1303202300 wait complete. 2020-11-20T12:12:21.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c1dcc5700 1 Processor -- start 2020-11-20T12:12:21.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.435+0000 7f1c1dcc5700 1 -- start start 2020-11-20T12:12:21.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c1dcc5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c180ad000 0x7f1c181408b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:21.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c1dcc5700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1c180ad3e0 0x7f1c1814cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:21.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c1dcc5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c180a62a0 0x7f1c18149a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:21.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c1dcc5700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1814d390 con 0x7f1c180ad000 2020-11-20T12:12:21.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c1dcc5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1814d620 con 0x7f1c180a62a0 2020-11-20T12:12:21.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c1dcc5700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1814d8c0 con 0x7f1c180ad3e0 2020-11-20T12:12:21.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c1d4c4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c180a62a0 0x7f1c18149a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:21.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c1d4c4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c180a62a0 0x7f1c18149a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50378/0 (socket says 172.21.15.189:50378) 2020-11-20T12:12:21.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c1d4c4700 1 -- 172.21.15.189:0/203260413 learned_addr learned my addr 172.21.15.189:0/203260413 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:21.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c17fff700 1 --2- 172.21.15.189:0/203260413 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1c180ad3e0 0x7f1c1814cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:21.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c1ccc3700 1 --2- 172.21.15.189:0/203260413 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c180ad000 0x7f1c181408b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:21.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c1d4c4700 1 -- 172.21.15.189:0/203260413 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1c180ad3e0 msgr2=0x7f1c1814cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:21.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c1d4c4700 1 --2- 172.21.15.189:0/203260413 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1c180ad3e0 0x7f1c1814cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:21.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c1d4c4700 1 -- 172.21.15.189:0/203260413 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c180ad000 msgr2=0x7f1c181408b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:21.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c1d4c4700 1 --2- 172.21.15.189:0/203260413 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c180ad000 0x7f1c181408b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:21.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c1d4c4700 1 -- 172.21.15.189:0/203260413 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c1814db60 con 0x7f1c180a62a0 2020-11-20T12:12:21.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c1ccc3700 1 --2- 172.21.15.189:0/203260413 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c180ad000 0x7f1c181408b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:12:21.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c17fff700 1 --2- 172.21.15.189:0/203260413 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1c180ad3e0 0x7f1c1814cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:12:21.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c1d4c4700 1 --2- 172.21.15.189:0/203260413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c180a62a0 0x7f1c18149a20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f1c1000db60 tx=0x7f1c10008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:21.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c14ff9700 1 -- 172.21.15.189:0/203260413 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1000b9d0 con 0x7f1c180a62a0 2020-11-20T12:12:21.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c14ff9700 1 -- 172.21.15.189:0/203260413 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c1000bb30 con 0x7f1c180a62a0 2020-11-20T12:12:21.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c1dcc5700 1 -- 172.21.15.189:0/203260413 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c18149f10 con 0x7f1c180a62a0 2020-11-20T12:12:21.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c14ff9700 1 -- 172.21.15.189:0/203260413 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c10012420 con 0x7f1c180a62a0 2020-11-20T12:12:21.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c1dcc5700 1 -- 172.21.15.189:0/203260413 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c180aa6c0 con 0x7f1c180a62a0 2020-11-20T12:12:21.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c1dcc5700 1 -- 172.21.15.189:0/203260413 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c180b13f0 con 0x7f1c180a62a0 2020-11-20T12:12:21.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c14ff9700 1 -- 172.21.15.189:0/203260413 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f1c10012790 con 0x7f1c180a62a0 2020-11-20T12:12:21.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c14ff9700 1 --2- 172.21.15.189:0/203260413 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1bf00557c0 0x7f1bf0057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:21.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.439+0000 7f1c1ccc3700 1 --2- 172.21.15.189:0/203260413 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1bf00557c0 0x7f1bf0057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:21.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.443+0000 7f1c14ff9700 1 -- 172.21.15.189:0/203260413 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f1c1006ebc0 con 0x7f1c180a62a0 2020-11-20T12:12:21.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.443+0000 7f1c1ccc3700 1 --2- 172.21.15.189:0/203260413 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1bf00557c0 0x7f1bf0057c80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f1c0c006fd0 tx=0x7f1c0c008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:21.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.447+0000 7f1c14ff9700 1 -- 172.21.15.189:0/203260413 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1c100448b0 con 0x7f1c180a62a0 2020-11-20T12:12:21.667 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.659+0000 7f1c1dcc5700 1 -- 172.21.15.189:0/203260413 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c180446d0 con 0x7f1c180a62a0 2020-11-20T12:12:21.668 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.659+0000 7f1c14ff9700 1 -- 172.21.15.189:0/203260413 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f1c10047ed0 con 0x7f1c180a62a0 2020-11-20T12:12:21.668 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:12:21.669 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5203,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.189:6835/2651986971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2651986971},{"type":"v1","addr":"172.21.15.189:6835","nonce":2651986971}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:05.684503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5122":{"gid":5122,"name":"b","rank":0,"incarnation":27,"state":"up:active","state_seq":14,"addr":"172.21.15.47:6833/4088050649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4088050649},{"type":"v1","addr":"172.21.15.47:6833","nonce":4088050649}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:12:21.670 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.663+0000 7f1c1dcc5700 1 -- 172.21.15.189:0/203260413 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1bf00557c0 msgr2=0x7f1bf0057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:21.670 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.663+0000 7f1c1dcc5700 1 --2- 172.21.15.189:0/203260413 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1bf00557c0 0x7f1bf0057c80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f1c0c006fd0 tx=0x7f1c0c008040).stop 2020-11-20T12:12:21.670 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.663+0000 7f1c1dcc5700 1 -- 172.21.15.189:0/203260413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c180a62a0 msgr2=0x7f1c18149a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:21.671 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.663+0000 7f1c1dcc5700 1 --2- 172.21.15.189:0/203260413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c180a62a0 0x7f1c18149a20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f1c1000db60 tx=0x7f1c10008040).stop 2020-11-20T12:12:21.671 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.663+0000 7f1c1dcc5700 1 -- 172.21.15.189:0/203260413 shutdown_connections 2020-11-20T12:12:21.671 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.663+0000 7f1c1dcc5700 1 --2- 172.21.15.189:0/203260413 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1bf00557c0 0x7f1bf0057c80 secure :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0x7f1c0c006fd0 tx=0x7f1c0c008040).stop 2020-11-20T12:12:21.671 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.663+0000 7f1c1dcc5700 1 --2- 172.21.15.189:0/203260413 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1c180ad000 0x7f1c181408b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:21.671 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.663+0000 7f1c1dcc5700 1 --2- 172.21.15.189:0/203260413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c180a62a0 0x7f1c18149a20 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:21.672 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.663+0000 7f1c1dcc5700 1 --2- 172.21.15.189:0/203260413 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1c180ad3e0 0x7f1c1814cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:21.672 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.663+0000 7f1c1dcc5700 1 -- 172.21.15.189:0/203260413 >> 172.21.15.189:0/203260413 conn(0x7f1c18062080 msgr2=0x7f1c1813aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:21.672 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.663+0000 7f1c1dcc5700 1 -- 172.21.15.189:0/203260413 shutdown_connections 2020-11-20T12:12:21.672 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:21.663+0000 7f1c1dcc5700 1 -- 172.21.15.189:0/203260413 wait complete. 2020-11-20T12:12:21.673 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 31 2020-11-20T12:12:21.689 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:12:23.550 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:12:23.558 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:12:23.690 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:23.820 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.810+0000 7fdd7f6bf700 1 Processor -- start 2020-11-20T12:12:23.821 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.810+0000 7fdd7f6bf700 1 -- start start 2020-11-20T12:12:23.824 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.810+0000 7fdd7f6bf700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdd78068020 0x7fdd781339f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:23.824 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.810+0000 7fdd7f6bf700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd780663e0 con 0x7fdd780a9e20 2020-11-20T12:12:23.824 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.810+0000 7fdd7f6bf700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd78067b80 con 0x7fdd780a9290 2020-11-20T12:12:23.824 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.810+0000 7fdd7f6bf700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd780a9970 con 0x7fdd78068020 2020-11-20T12:12:23.824 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.810+0000 7fdd7dc5c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdd780a9290 0x7fdd78133ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52014/0 (socket says 172.21.15.189:52014) 2020-11-20T12:12:23.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.810+0000 7fdd7dc5c700 1 -- 172.21.15.189:0/22165786 learned_addr learned my addr 172.21.15.189:0/22165786 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:12:23.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.810+0000 7fdd7cc5a700 1 --2- 172.21.15.189:0/22165786 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdd78068020 0x7fdd781339f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:23.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd67fff700 1 -- 172.21.15.189:0/22165786 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031591941 0 0) 0x7fdd78067b80 con 0x7fdd780a9290 2020-11-20T12:12:23.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd67fff700 1 -- 172.21.15.189:0/22165786 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd60004350 con 0x7fdd780a9290 2020-11-20T12:12:23.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd67fff700 1 -- 172.21.15.189:0/22165786 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999609929 0 0) 0x7fdd780663e0 con 0x7fdd780a9e20 2020-11-20T12:12:23.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd67fff700 1 -- 172.21.15.189:0/22165786 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd78067b80 con 0x7fdd780a9e20 2020-11-20T12:12:23.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd67fff700 1 -- 172.21.15.189:0/22165786 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2928470977 0 0) 0x7fdd60004350 con 0x7fdd780a9290 2020-11-20T12:12:23.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd67fff700 1 -- 172.21.15.189:0/22165786 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdd78068020 msgr2=0x7fdd781339f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:23.832 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd67fff700 1 --2- 172.21.15.189:0/22165786 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdd78068020 0x7fdd781339f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:23.832 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd67fff700 1 -- 172.21.15.189:0/22165786 >> v1:172.21.15.47:6789/0 conn(0x7fdd780a9e20 legacy=0x7fdd780a83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:23.832 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd67fff700 1 -- 172.21.15.189:0/22165786 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd78066680 con 0x7fdd780a9290 2020-11-20T12:12:23.833 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd67fff700 1 -- 172.21.15.189:0/22165786 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7fdd700018a0 con 0x7fdd780a9290 2020-11-20T12:12:23.833 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd67fff700 1 -- 172.21.15.189:0/22165786 >> v1:172.21.15.189:6789/0 conn(0x7fdd780a9290 legacy=0x7fdd78133ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:23.833 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd67fff700 1 --2- 172.21.15.189:0/22165786 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdd78068020 0x7fdd600049d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:23.836 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd67fff700 1 --2- 172.21.15.189:0/22165786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd60004ec0 0x7fdd600055f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:23.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd67fff700 1 --2- 172.21.15.189:0/22165786 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd60005ae0 0x7fdd60005f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:23.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd67fff700 1 -- 172.21.15.189:0/22165786 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd780a9970 con 0x7fdd60005ae0 2020-11-20T12:12:23.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd67fff700 1 -- 172.21.15.189:0/22165786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd60006450 con 0x7fdd60004ec0 2020-11-20T12:12:23.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd67fff700 1 -- 172.21.15.189:0/22165786 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd60003270 con 0x7fdd78068020 2020-11-20T12:12:23.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd7cc5a700 1 --2- 172.21.15.189:0/22165786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd60004ec0 0x7fdd600055f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:23.839 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd7cc5a700 1 -- 172.21.15.189:0/22165786 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdd78068020 msgr2=0x7fdd600049d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:23.839 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd7dc5c700 1 --2- 172.21.15.189:0/22165786 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd60005ae0 0x7fdd60005f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:23.839 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd7d45b700 1 --2- 172.21.15.189:0/22165786 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdd78068020 0x7fdd600049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:23.840 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd7cc5a700 1 --2- 172.21.15.189:0/22165786 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdd78068020 0x7fdd600049d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:23.840 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd7cc5a700 1 -- 172.21.15.189:0/22165786 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd60005ae0 msgr2=0x7fdd60005f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:23.840 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd7cc5a700 1 --2- 172.21.15.189:0/22165786 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd60005ae0 0x7fdd60005f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:23.841 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd7cc5a700 1 -- 172.21.15.189:0/22165786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd60004350 con 0x7fdd60004ec0 2020-11-20T12:12:23.841 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd7dc5c700 1 --2- 172.21.15.189:0/22165786 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd60005ae0 0x7fdd60005f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:23.842 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd7d45b700 1 --2- 172.21.15.189:0/22165786 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdd78068020 0x7fdd600049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:23.842 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd7cc5a700 1 --2- 172.21.15.189:0/22165786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd60004ec0 0x7fdd600055f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fdd6800bb90 tx=0x7fdd68008040).ready entity=mon.1 client_cookie=f1e4cff382918125 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:23.843 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd67fff700 1 -- 172.21.15.189:0/22165786 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6800b2c0 con 0x7fdd60004ec0 2020-11-20T12:12:23.843 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd67fff700 1 -- 172.21.15.189:0/22165786 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd6800ea90 con 0x7fdd60004ec0 2020-11-20T12:12:23.843 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd67fff700 1 -- 172.21.15.189:0/22165786 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6800ee10 con 0x7fdd60004ec0 2020-11-20T12:12:23.844 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd7f6bf700 1 -- 172.21.15.189:0/22165786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd60004ec0 msgr2=0x7fdd600055f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:23.844 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd7f6bf700 1 --2- 172.21.15.189:0/22165786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd60004ec0 0x7fdd600055f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fdd6800bb90 tx=0x7fdd68008040).stop 2020-11-20T12:12:23.844 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd7dc5c700 1 -- 172.21.15.189:0/22165786 reap_dead start 2020-11-20T12:12:23.844 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd7f6bf700 1 -- 172.21.15.189:0/22165786 shutdown_connections 2020-11-20T12:12:23.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd7f6bf700 1 -- 172.21.15.189:0/22165786 >> 172.21.15.189:0/22165786 conn(0x7fdd780627f0 msgr2=0x7fdd78065870 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:23.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd7f6bf700 1 -- 172.21.15.189:0/22165786 shutdown_connections 2020-11-20T12:12:23.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd7f6bf700 1 -- 172.21.15.189:0/22165786 wait complete. 2020-11-20T12:12:23.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd7f6bf700 1 Processor -- start 2020-11-20T12:12:23.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.814+0000 7fdd7f6bf700 1 -- start start 2020-11-20T12:12:23.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.818+0000 7fdd7f6bf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd60004ec0 0x7fdd78139ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:23.848 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.818+0000 7fdd7f6bf700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd780a1330 0x7fdd781387f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:23.848 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.818+0000 7fdd7f6bf700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdd78138ce0 0x7fdd78148e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:23.848 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.818+0000 7fdd7f6bf700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7812bc90 con 0x7fdd780a1330 2020-11-20T12:12:23.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.818+0000 7fdd7f6bf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd781392c0 con 0x7fdd60004ec0 2020-11-20T12:12:23.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.818+0000 7fdd7f6bf700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd78139560 con 0x7fdd78138ce0 2020-11-20T12:12:23.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.818+0000 7fdd7d45b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd60004ec0 0x7fdd78139ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:23.850 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.818+0000 7fdd7d45b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd60004ec0 0x7fdd78139ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50400/0 (socket says 172.21.15.189:50400) 2020-11-20T12:12:23.850 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.818+0000 7fdd7d45b700 1 -- 172.21.15.189:0/2397946460 learned_addr learned my addr 172.21.15.189:0/2397946460 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:23.850 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.818+0000 7fdd7cc5a700 1 --2- 172.21.15.189:0/2397946460 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd780a1330 0x7fdd781387f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:23.851 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.818+0000 7fdd7dc5c700 1 --2- 172.21.15.189:0/2397946460 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdd78138ce0 0x7fdd78148e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:23.851 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.818+0000 7fdd7d45b700 1 -- 172.21.15.189:0/2397946460 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdd78138ce0 msgr2=0x7fdd78148e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:23.851 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.818+0000 7fdd7d45b700 1 --2- 172.21.15.189:0/2397946460 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdd78138ce0 0x7fdd78148e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:23.852 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.818+0000 7fdd7d45b700 1 -- 172.21.15.189:0/2397946460 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd780a1330 msgr2=0x7fdd781387f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:23.853 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.818+0000 7fdd7d45b700 1 --2- 172.21.15.189:0/2397946460 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd780a1330 0x7fdd781387f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:23.853 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.818+0000 7fdd7d45b700 1 -- 172.21.15.189:0/2397946460 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd78139800 con 0x7fdd60004ec0 2020-11-20T12:12:23.853 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.818+0000 7fdd7d45b700 1 --2- 172.21.15.189:0/2397946460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd60004ec0 0x7fdd78139ca0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fdd7400db00 tx=0x7fdd74009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:23.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.818+0000 7fdd657fa700 1 -- 172.21.15.189:0/2397946460 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd74016070 con 0x7fdd60004ec0 2020-11-20T12:12:23.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.818+0000 7fdd7f6bf700 1 -- 172.21.15.189:0/2397946460 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd780a17b0 con 0x7fdd60004ec0 2020-11-20T12:12:23.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.818+0000 7fdd657fa700 1 -- 172.21.15.189:0/2397946460 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd7400de10 con 0x7fdd60004ec0 2020-11-20T12:12:23.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.818+0000 7fdd657fa700 1 -- 172.21.15.189:0/2397946460 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7400ba40 con 0x7fdd60004ec0 2020-11-20T12:12:23.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.818+0000 7fdd7f6bf700 1 -- 172.21.15.189:0/2397946460 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd78149520 con 0x7fdd60004ec0 2020-11-20T12:12:23.855 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.818+0000 7fdd7f6bf700 1 -- 172.21.15.189:0/2397946460 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd78044f60 con 0x7fdd60004ec0 2020-11-20T12:12:23.855 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.818+0000 7fdd657fa700 1 -- 172.21.15.189:0/2397946460 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fdd7401b410 con 0x7fdd60004ec0 2020-11-20T12:12:23.855 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.822+0000 7fdd657fa700 1 --2- 172.21.15.189:0/2397946460 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fdd500557c0 0x7fdd50057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:23.855 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.822+0000 7fdd657fa700 1 -- 172.21.15.189:0/2397946460 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6503+0+0 (secure 0 0 0) 0x7fdd7406e6b0 con 0x7fdd60004ec0 2020-11-20T12:12:23.855 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.822+0000 7fdd7cc5a700 1 --2- 172.21.15.189:0/2397946460 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fdd500557c0 0x7fdd50057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:23.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.822+0000 7fdd7cc5a700 1 --2- 172.21.15.189:0/2397946460 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fdd500557c0 0x7fdd50057c80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fdd780acec0 tx=0x7fdd68019000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:23.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:23.826+0000 7fdd657fa700 1 -- 172.21.15.189:0/2397946460 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fdd740443c0 con 0x7fdd60004ec0 2020-11-20T12:12:24.036 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:24.026+0000 7fdd7f6bf700 1 -- 172.21.15.189:0/2397946460 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd780a75f0 con 0x7fdd60004ec0 2020-11-20T12:12:24.038 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:24.026+0000 7fdd657fa700 1 -- 172.21.15.189:0/2397946460 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fdd740479e0 con 0x7fdd60004ec0 2020-11-20T12:12:24.038 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:12:24.039 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5203,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.189:6835/2651986971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2651986971},{"type":"v1","addr":"172.21.15.189:6835","nonce":2651986971}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:05.684503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5122":{"gid":5122,"name":"b","rank":0,"incarnation":27,"state":"up:active","state_seq":14,"addr":"172.21.15.47:6833/4088050649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4088050649},{"type":"v1","addr":"172.21.15.47:6833","nonce":4088050649}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:12:24.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:24.030+0000 7fdd7f6bf700 1 -- 172.21.15.189:0/2397946460 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fdd500557c0 msgr2=0x7fdd50057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:24.042 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:24.030+0000 7fdd7f6bf700 1 --2- 172.21.15.189:0/2397946460 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fdd500557c0 0x7fdd50057c80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fdd780acec0 tx=0x7fdd68019000).stop 2020-11-20T12:12:24.042 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:24.030+0000 7fdd7f6bf700 1 -- 172.21.15.189:0/2397946460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd60004ec0 msgr2=0x7fdd78139ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:24.042 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:24.030+0000 7fdd7f6bf700 1 --2- 172.21.15.189:0/2397946460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd60004ec0 0x7fdd78139ca0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fdd7400db00 tx=0x7fdd74009040).stop 2020-11-20T12:12:24.042 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:24.030+0000 7fdd7f6bf700 1 -- 172.21.15.189:0/2397946460 shutdown_connections 2020-11-20T12:12:24.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:24.030+0000 7fdd7f6bf700 1 --2- 172.21.15.189:0/2397946460 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdd78138ce0 0x7fdd78148e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:24.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:24.030+0000 7fdd7f6bf700 1 --2- 172.21.15.189:0/2397946460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd60004ec0 0x7fdd78139ca0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:24.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:24.030+0000 7fdd7f6bf700 1 --2- 172.21.15.189:0/2397946460 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fdd500557c0 0x7fdd50057c80 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:24.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:24.030+0000 7fdd7f6bf700 1 --2- 172.21.15.189:0/2397946460 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdd780a1330 0x7fdd781387f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:24.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:24.030+0000 7fdd7f6bf700 1 -- 172.21.15.189:0/2397946460 >> 172.21.15.189:0/2397946460 conn(0x7fdd780627f0 msgr2=0x7fdd780a07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:24.044 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:24.030+0000 7fdd7f6bf700 1 -- 172.21.15.189:0/2397946460 shutdown_connections 2020-11-20T12:12:24.044 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:24.030+0000 7fdd7f6bf700 1 -- 172.21.15.189:0/2397946460 wait complete. 2020-11-20T12:12:24.044 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 31 2020-11-20T12:12:24.056 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:12:26.057 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:26.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.182+0000 7fc20759e700 1 Processor -- start 2020-11-20T12:12:26.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.182+0000 7fc20759e700 1 -- start start 2020-11-20T12:12:26.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.182+0000 7fc20759e700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc208066ed0 0x7fc20813b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:26.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.182+0000 7fc20759e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc208066800 con 0x7fc2080a8840 2020-11-20T12:12:26.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.182+0000 7fc20759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc208066aa0 con 0x7fc20813ba60 2020-11-20T12:12:26.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.182+0000 7fc20759e700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc208067450 con 0x7fc208066ed0 2020-11-20T12:12:26.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.182+0000 7fc206d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc20813ba60 0x7fc2081495e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52036/0 (socket says 172.21.15.189:52036) 2020-11-20T12:12:26.194 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.182+0000 7fc206d9d700 1 -- 172.21.15.189:0/1090425537 learned_addr learned my addr 172.21.15.189:0/1090425537 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:12:26.194 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.182+0000 7fc205d9b700 1 --2- 172.21.15.189:0/1090425537 >> v2:172.21.15.47:3300/0 conn(0x7fc208066ed0 0x7fc20813b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:26.194 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.182+0000 7fc20559a700 1 -- 172.21.15.189:0/1090425537 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948503437 0 0) 0x7fc208066aa0 con 0x7fc20813ba60 2020-11-20T12:12:26.194 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.182+0000 7fc20559a700 1 -- 172.21.15.189:0/1090425537 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1e8003240 con 0x7fc20813ba60 2020-11-20T12:12:26.194 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.182+0000 7fc20559a700 1 -- 172.21.15.189:0/1090425537 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1666392373 0 0) 0x7fc208066800 con 0x7fc2080a8840 2020-11-20T12:12:26.195 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.182+0000 7fc20559a700 1 -- 172.21.15.189:0/1090425537 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc208066aa0 con 0x7fc2080a8840 2020-11-20T12:12:26.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.182+0000 7fc205d9b700 1 -- 172.21.15.189:0/1090425537 >> v1:172.21.15.189:6789/0 conn(0x7fc20813ba60 legacy=0x7fc2081495e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:26.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.182+0000 7fc205d9b700 1 -- 172.21.15.189:0/1090425537 >> v1:172.21.15.47:6789/0 conn(0x7fc2080a8840 legacy=0x7fc20813eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:26.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.182+0000 7fc205d9b700 1 -- 172.21.15.189:0/1090425537 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2080676f0 con 0x7fc208066ed0 2020-11-20T12:12:26.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.182+0000 7fc205d9b700 1 --2- 172.21.15.189:0/1090425537 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc208066ed0 0x7fc20813b570 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fc200009c90 tx=0x7fc2000091d0).ready entity=mon.0 client_cookie=2e91c3fa8d6c88fa server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:26.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc20559a700 1 -- 172.21.15.189:0/1090425537 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20000b9c0 con 0x7fc208066ed0 2020-11-20T12:12:26.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc20559a700 1 -- 172.21.15.189:0/1090425537 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc20000bb20 con 0x7fc208066ed0 2020-11-20T12:12:26.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc20559a700 1 -- 172.21.15.189:0/1090425537 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20000bea0 con 0x7fc208066ed0 2020-11-20T12:12:26.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc20759e700 1 -- 172.21.15.189:0/1090425537 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc208066ed0 msgr2=0x7fc20813b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:26.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc20759e700 1 --2- 172.21.15.189:0/1090425537 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc208066ed0 0x7fc20813b570 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fc200009c90 tx=0x7fc2000091d0).stop 2020-11-20T12:12:26.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc20759e700 1 -- 172.21.15.189:0/1090425537 shutdown_connections 2020-11-20T12:12:26.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc20759e700 1 --2- 172.21.15.189:0/1090425537 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc208066ed0 0x7fc20813b570 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:26.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc20759e700 1 -- 172.21.15.189:0/1090425537 >> 172.21.15.189:0/1090425537 conn(0x7fc208060170 msgr2=0x7fc208060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:26.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc20759e700 1 -- 172.21.15.189:0/1090425537 shutdown_connections 2020-11-20T12:12:26.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc20759e700 1 -- 172.21.15.189:0/1090425537 wait complete. 2020-11-20T12:12:26.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc20759e700 1 Processor -- start 2020-11-20T12:12:26.202 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc20759e700 1 -- start start 2020-11-20T12:12:26.202 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc20759e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc208066ed0 0x7fc208140330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:26.202 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc20759e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2080a8840 0x7fc20814e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:26.202 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc20759e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc20813ba60 0x7fc2080a1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:26.202 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc20759e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20814e8a0 con 0x7fc208066ed0 2020-11-20T12:12:26.203 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc20759e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20814eb00 con 0x7fc2080a8840 2020-11-20T12:12:26.203 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc20759e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc20814eda0 con 0x7fc20813ba60 2020-11-20T12:12:26.203 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc205d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2080a8840 0x7fc20814e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:26.203 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc205d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2080a8840 0x7fc20814e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50418/0 (socket says 172.21.15.189:50418) 2020-11-20T12:12:26.203 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc205d9b700 1 -- 172.21.15.189:0/985634453 learned_addr learned my addr 172.21.15.189:0/985634453 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:26.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc20659c700 1 --2- 172.21.15.189:0/985634453 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc208066ed0 0x7fc208140330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:26.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc206d9d700 1 --2- 172.21.15.189:0/985634453 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc20813ba60 0x7fc2080a1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:26.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc205d9b700 1 -- 172.21.15.189:0/985634453 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc20813ba60 msgr2=0x7fc2080a1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:26.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc205d9b700 1 --2- 172.21.15.189:0/985634453 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc20813ba60 0x7fc2080a1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:26.204 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc205d9b700 1 -- 172.21.15.189:0/985634453 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc208066ed0 msgr2=0x7fc208140330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:26.205 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc205d9b700 1 --2- 172.21.15.189:0/985634453 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc208066ed0 0x7fc208140330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:26.205 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc205d9b700 1 -- 172.21.15.189:0/985634453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2080a4fd0 con 0x7fc2080a8840 2020-11-20T12:12:26.205 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc206d9d700 1 --2- 172.21.15.189:0/985634453 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc20813ba60 0x7fc2080a1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:12:26.205 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.186+0000 7fc205d9b700 1 --2- 172.21.15.189:0/985634453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2080a8840 0x7fc20814e3b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7fc20000baf0 tx=0x7fc200016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:26.205 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.190+0000 7fc1fe7fc700 1 -- 172.21.15.189:0/985634453 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20001d480 con 0x7fc2080a8840 2020-11-20T12:12:26.206 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.190+0000 7fc1fe7fc700 1 -- 172.21.15.189:0/985634453 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc20001d5e0 con 0x7fc2080a8840 2020-11-20T12:12:26.206 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.190+0000 7fc20759e700 1 -- 172.21.15.189:0/985634453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc20813c880 con 0x7fc2080a8840 2020-11-20T12:12:26.206 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.190+0000 7fc1fe7fc700 1 -- 172.21.15.189:0/985634453 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20001d960 con 0x7fc2080a8840 2020-11-20T12:12:26.206 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.190+0000 7fc20759e700 1 -- 172.21.15.189:0/985634453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2080a2400 con 0x7fc2080a8840 2020-11-20T12:12:26.207 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.190+0000 7fc20759e700 1 -- 172.21.15.189:0/985634453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2080a5160 con 0x7fc2080a8840 2020-11-20T12:12:26.207 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.190+0000 7fc1fe7fc700 1 -- 172.21.15.189:0/985634453 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fc200017070 con 0x7fc2080a8840 2020-11-20T12:12:26.208 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.190+0000 7fc1fe7fc700 1 --2- 172.21.15.189:0/985634453 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc1e40557f0 0x7fc1e4057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:26.208 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.190+0000 7fc1fe7fc700 1 -- 172.21.15.189:0/985634453 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6503+0+0 (secure 0 0 0) 0x7fc200070890 con 0x7fc2080a8840 2020-11-20T12:12:26.208 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.190+0000 7fc20659c700 1 --2- 172.21.15.189:0/985634453 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc1e40557f0 0x7fc1e4057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:26.208 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.194+0000 7fc20659c700 1 --2- 172.21.15.189:0/985634453 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc1e40557f0 0x7fc1e4057cb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fc1f0000bc0 tx=0x7fc1f0007a00).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:26.209 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.198+0000 7fc1fe7fc700 1 -- 172.21.15.189:0/985634453 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc200046630 con 0x7fc2080a8840 2020-11-20T12:12:26.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.398+0000 7fc20759e700 1 -- 172.21.15.189:0/985634453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2080446d0 con 0x7fc2080a8840 2020-11-20T12:12:26.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.398+0000 7fc1fe7fc700 1 -- 172.21.15.189:0/985634453 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fc200015020 con 0x7fc2080a8840 2020-11-20T12:12:26.407 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:12:26.408 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5203,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.189:6835/2651986971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2651986971},{"type":"v1","addr":"172.21.15.189:6835","nonce":2651986971}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:05.684503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5122":{"gid":5122,"name":"b","rank":0,"incarnation":27,"state":"up:active","state_seq":14,"addr":"172.21.15.47:6833/4088050649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4088050649},{"type":"v1","addr":"172.21.15.47:6833","nonce":4088050649}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:12:26.409 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.398+0000 7fc20759e700 1 -- 172.21.15.189:0/985634453 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc1e40557f0 msgr2=0x7fc1e4057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:26.409 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.398+0000 7fc20759e700 1 --2- 172.21.15.189:0/985634453 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc1e40557f0 0x7fc1e4057cb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fc1f0000bc0 tx=0x7fc1f0007a00).stop 2020-11-20T12:12:26.409 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.398+0000 7fc20759e700 1 -- 172.21.15.189:0/985634453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2080a8840 msgr2=0x7fc20814e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:26.410 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.398+0000 7fc20759e700 1 --2- 172.21.15.189:0/985634453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2080a8840 0x7fc20814e3b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7fc20000baf0 tx=0x7fc200016000).stop 2020-11-20T12:12:26.410 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.398+0000 7fc20759e700 1 -- 172.21.15.189:0/985634453 shutdown_connections 2020-11-20T12:12:26.410 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.398+0000 7fc20759e700 1 --2- 172.21.15.189:0/985634453 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc1e40557f0 0x7fc1e4057cb0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:26.410 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.398+0000 7fc20759e700 1 --2- 172.21.15.189:0/985634453 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc208066ed0 0x7fc208140330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:26.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.398+0000 7fc20759e700 1 --2- 172.21.15.189:0/985634453 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc20813ba60 0x7fc2080a1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:26.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.398+0000 7fc20759e700 1 --2- 172.21.15.189:0/985634453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2080a8840 0x7fc20814e3b0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:26.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.398+0000 7fc20759e700 1 -- 172.21.15.189:0/985634453 >> 172.21.15.189:0/985634453 conn(0x7fc208060170 msgr2=0x7fc208144040 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:26.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.398+0000 7fc20759e700 1 -- 172.21.15.189:0/985634453 shutdown_connections 2020-11-20T12:12:26.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:26.398+0000 7fc20759e700 1 -- 172.21.15.189:0/985634453 wait complete. 2020-11-20T12:12:26.412 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 31 2020-11-20T12:12:26.422 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:12:28.423 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:28.559 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.550+0000 7f9c50c8e700 1 Processor -- start 2020-11-20T12:12:28.559 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.550+0000 7f9c50c8e700 1 -- start start 2020-11-20T12:12:28.561 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.550+0000 7f9c50c8e700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f9c4c0ad000 0x7f9c4c13e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:28.561 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.550+0000 7f9c50c8e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9c4c0ad3e0 0x7f9c4c066d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:28.562 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.550+0000 7f9c50c8e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c4c067730 con 0x7f9c4c13a6e0 2020-11-20T12:12:28.562 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.550+0000 7f9c50c8e700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c4c138190 con 0x7f9c4c0ad000 2020-11-20T12:12:28.562 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.550+0000 7f9c50c8e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c4c13dd40 con 0x7f9c4c0ad3e0 2020-11-20T12:12:28.562 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.550+0000 7f9c4affd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9c4c0ad3e0 0x7f9c4c066d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:28.563 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.550+0000 7f9c4b7fe700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f9c4c0ad000 0x7f9c4c13e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:28.563 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.550+0000 7f9c4affd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9c4c0ad3e0 0x7f9c4c066d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50428/0 (socket says 172.21.15.189:50428) 2020-11-20T12:12:28.563 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.550+0000 7f9c4affd700 1 -- 172.21.15.189:0/3712756505 learned_addr learned my addr 172.21.15.189:0/3712756505 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:28.563 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.550+0000 7f9c4affd700 1 -- 172.21.15.189:0/3712756505 >> v2:172.21.15.47:3300/0 conn(0x7f9c4c0ad000 msgr2=0x7f9c4c13e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:28.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.550+0000 7f9c4affd700 1 --2- 172.21.15.189:0/3712756505 >> v2:172.21.15.47:3300/0 conn(0x7f9c4c0ad000 0x7f9c4c13e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:28.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.550+0000 7f9c4affd700 1 -- 172.21.15.189:0/3712756505 >> v1:172.21.15.47:6789/0 conn(0x7f9c4c13a6e0 legacy=0x7f9c4c13ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:28.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.550+0000 7f9c4a7fc700 1 -- 172.21.15.189:0/3712756505 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2040150917 0 0) 0x7f9c4c067730 con 0x7f9c4c13a6e0 2020-11-20T12:12:28.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.550+0000 7f9c4affd700 1 -- 172.21.15.189:0/3712756505 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c4c13dfe0 con 0x7f9c4c0ad3e0 2020-11-20T12:12:28.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.550+0000 7f9c4b7fe700 1 --2- 172.21.15.189:0/3712756505 >> v2:172.21.15.47:3300/0 conn(0x7f9c4c0ad000 0x7f9c4c13e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:12:28.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.550+0000 7f9c4affd700 1 --2- 172.21.15.189:0/3712756505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c4c0ad3e0 0x7f9c4c066d40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f9c4c0b18d0 tx=0x7f9c34006eb0).ready entity=mon.1 client_cookie=76055aafdaf6082e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:28.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.550+0000 7f9c4a7fc700 1 -- 172.21.15.189:0/3712756505 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c34009810 con 0x7f9c4c0ad3e0 2020-11-20T12:12:28.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.550+0000 7f9c4a7fc700 1 -- 172.21.15.189:0/3712756505 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c34009970 con 0x7f9c4c0ad3e0 2020-11-20T12:12:28.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.550+0000 7f9c4a7fc700 1 -- 172.21.15.189:0/3712756505 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c34009cf0 con 0x7f9c4c0ad3e0 2020-11-20T12:12:28.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c50c8e700 1 -- 172.21.15.189:0/3712756505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c4c0ad3e0 msgr2=0x7f9c4c066d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:28.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c50c8e700 1 --2- 172.21.15.189:0/3712756505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c4c0ad3e0 0x7f9c4c066d40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f9c4c0b18d0 tx=0x7f9c34006eb0).stop 2020-11-20T12:12:28.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c50c8e700 1 -- 172.21.15.189:0/3712756505 shutdown_connections 2020-11-20T12:12:28.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c50c8e700 1 --2- 172.21.15.189:0/3712756505 >> v2:172.21.15.47:3300/0 conn(0x7f9c4c0ad000 0x7f9c4c13e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:28.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c50c8e700 1 --2- 172.21.15.189:0/3712756505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c4c0ad3e0 0x7f9c4c066d40 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:28.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c50c8e700 1 -- 172.21.15.189:0/3712756505 >> 172.21.15.189:0/3712756505 conn(0x7f9c4c062080 msgr2=0x7f9c4c062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:28.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c50c8e700 1 -- 172.21.15.189:0/3712756505 shutdown_connections 2020-11-20T12:12:28.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c50c8e700 1 -- 172.21.15.189:0/3712756505 wait complete. 2020-11-20T12:12:28.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c50c8e700 1 Processor -- start 2020-11-20T12:12:28.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c50c8e700 1 -- start start 2020-11-20T12:12:28.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c50c8e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c4c0ad000 0x7f9c4c0afdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:28.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c50c8e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9c4c0ad3e0 0x7f9c4c1499b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:28.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c50c8e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9c4c0a6450 0x7f9c4c0aa850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:28.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c50c8e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c14d300 con 0x7f9c4c0a6450 2020-11-20T12:12:28.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c50c8e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c14d5a0 con 0x7f9c4c0ad000 2020-11-20T12:12:28.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c50c8e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4c14d840 con 0x7f9c4c0ad3e0 2020-11-20T12:12:28.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c4b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c4c0ad000 0x7f9c4c0afdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:28.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c4b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c4c0ad000 0x7f9c4c0afdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50432/0 (socket says 172.21.15.189:50432) 2020-11-20T12:12:28.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c4b7fe700 1 -- 172.21.15.189:0/1366180288 learned_addr learned my addr 172.21.15.189:0/1366180288 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:28.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c4affd700 1 --2- 172.21.15.189:0/1366180288 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9c4c0ad3e0 0x7f9c4c1499b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:28.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c4bfff700 1 --2- 172.21.15.189:0/1366180288 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9c4c0a6450 0x7f9c4c0aa850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:28.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c4b7fe700 1 -- 172.21.15.189:0/1366180288 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9c4c0ad3e0 msgr2=0x7f9c4c1499b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:28.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c4b7fe700 1 --2- 172.21.15.189:0/1366180288 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9c4c0ad3e0 0x7f9c4c1499b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:28.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c4b7fe700 1 -- 172.21.15.189:0/1366180288 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9c4c0a6450 msgr2=0x7f9c4c0aa850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:28.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c4b7fe700 1 --2- 172.21.15.189:0/1366180288 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9c4c0a6450 0x7f9c4c0aa850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:28.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c4b7fe700 1 -- 172.21.15.189:0/1366180288 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c4c14cf80 con 0x7f9c4c0ad000 2020-11-20T12:12:28.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c4b7fe700 1 --2- 172.21.15.189:0/1366180288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c4c0ad000 0x7f9c4c0afdd0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f9c3c00c1d0 tx=0x7f9c3c0091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:28.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c437fe700 1 -- 172.21.15.189:0/1366180288 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c00cea0 con 0x7f9c4c0ad000 2020-11-20T12:12:28.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c437fe700 1 -- 172.21.15.189:0/1366180288 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c3c0123f0 con 0x7f9c4c0ad000 2020-11-20T12:12:28.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c50c8e700 1 -- 172.21.15.189:0/1366180288 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c4c14dce0 con 0x7f9c4c0ad000 2020-11-20T12:12:28.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c437fe700 1 -- 172.21.15.189:0/1366180288 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c012770 con 0x7f9c4c0ad000 2020-11-20T12:12:28.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.554+0000 7f9c50c8e700 1 -- 172.21.15.189:0/1366180288 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c4c0a7f90 con 0x7f9c4c0ad000 2020-11-20T12:12:28.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.558+0000 7f9c50c8e700 1 -- 172.21.15.189:0/1366180288 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c4c14da00 con 0x7f9c4c0ad000 2020-11-20T12:12:28.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.558+0000 7f9c437fe700 1 -- 172.21.15.189:0/1366180288 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f9c3c022070 con 0x7f9c4c0ad000 2020-11-20T12:12:28.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.558+0000 7f9c437fe700 1 --2- 172.21.15.189:0/1366180288 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9c280557c0 0x7f9c28057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:28.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.558+0000 7f9c4affd700 1 --2- 172.21.15.189:0/1366180288 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9c280557c0 0x7f9c28057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:28.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.558+0000 7f9c437fe700 1 -- 172.21.15.189:0/1366180288 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f9c3c06f620 con 0x7f9c4c0ad000 2020-11-20T12:12:28.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.558+0000 7f9c4affd700 1 --2- 172.21.15.189:0/1366180288 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9c280557c0 0x7f9c28057c80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f9c34006ef0 tx=0x7f9c34015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:28.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.562+0000 7f9c437fe700 1 -- 172.21.15.189:0/1366180288 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9c3c045390 con 0x7f9c4c0ad000 2020-11-20T12:12:28.771 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.762+0000 7f9c50c8e700 1 -- 172.21.15.189:0/1366180288 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c4c0a7f90 con 0x7f9c4c0ad000 2020-11-20T12:12:28.772 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.762+0000 7f9c437fe700 1 -- 172.21.15.189:0/1366180288 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f9c3c029870 con 0x7f9c4c0ad000 2020-11-20T12:12:28.773 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:12:28.773 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5203,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.189:6835/2651986971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2651986971},{"type":"v1","addr":"172.21.15.189:6835","nonce":2651986971}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:05.684503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5122":{"gid":5122,"name":"b","rank":0,"incarnation":27,"state":"up:active","state_seq":14,"addr":"172.21.15.47:6833/4088050649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4088050649},{"type":"v1","addr":"172.21.15.47:6833","nonce":4088050649}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:12:28.776 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.766+0000 7f9c50c8e700 1 -- 172.21.15.189:0/1366180288 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9c280557c0 msgr2=0x7f9c28057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:28.776 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.766+0000 7f9c50c8e700 1 --2- 172.21.15.189:0/1366180288 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9c280557c0 0x7f9c28057c80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f9c34006ef0 tx=0x7f9c34015000).stop 2020-11-20T12:12:28.776 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.766+0000 7f9c50c8e700 1 -- 172.21.15.189:0/1366180288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c4c0ad000 msgr2=0x7f9c4c0afdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:28.777 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.766+0000 7f9c50c8e700 1 --2- 172.21.15.189:0/1366180288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c4c0ad000 0x7f9c4c0afdd0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f9c3c00c1d0 tx=0x7f9c3c0091d0).stop 2020-11-20T12:12:28.777 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.766+0000 7f9c50c8e700 1 -- 172.21.15.189:0/1366180288 shutdown_connections 2020-11-20T12:12:28.778 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.766+0000 7f9c50c8e700 1 --2- 172.21.15.189:0/1366180288 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9c280557c0 0x7f9c28057c80 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:28.778 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.766+0000 7f9c50c8e700 1 --2- 172.21.15.189:0/1366180288 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9c4c0a6450 0x7f9c4c0aa850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:28.778 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.766+0000 7f9c50c8e700 1 --2- 172.21.15.189:0/1366180288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c4c0ad000 0x7f9c4c0afdd0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:28.779 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.766+0000 7f9c50c8e700 1 --2- 172.21.15.189:0/1366180288 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9c4c0ad3e0 0x7f9c4c1499b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:28.779 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.766+0000 7f9c50c8e700 1 -- 172.21.15.189:0/1366180288 >> 172.21.15.189:0/1366180288 conn(0x7f9c4c062080 msgr2=0x7f9c4c0ac590 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:28.780 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.766+0000 7f9c50c8e700 1 -- 172.21.15.189:0/1366180288 shutdown_connections 2020-11-20T12:12:28.780 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:28.766+0000 7f9c50c8e700 1 -- 172.21.15.189:0/1366180288 wait complete. 2020-11-20T12:12:28.780 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 31 2020-11-20T12:12:28.790 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:12:30.792 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:30.922 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a7b59e700 1 Processor -- start 2020-11-20T12:12:30.923 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a7b59e700 1 -- start start 2020-11-20T12:12:30.924 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a7b59e700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2a7c062220 0x7f2a7c0a94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:30.925 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a7b59e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2a7c062600 0x7f2a7c0a9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:30.925 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a7b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a7c13c390 con 0x7f2a7c141e60 2020-11-20T12:12:30.925 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a7b59e700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a7c0a99d0 con 0x7f2a7c062220 2020-11-20T12:12:30.925 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a7b59e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a7c142d50 con 0x7f2a7c062600 2020-11-20T12:12:30.926 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a7ad9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2a7c141e60 0x7f2a7c0679e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52066/0 (socket says 172.21.15.189:52066) 2020-11-20T12:12:30.926 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a79d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2a7c062600 0x7f2a7c0a9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:30.926 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a7ad9d700 1 -- 172.21.15.189:0/3775722031 learned_addr learned my addr 172.21.15.189:0/3775722031 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:12:30.926 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a7a59c700 1 --2- 172.21.15.189:0/3775722031 >> v2:172.21.15.47:3300/0 conn(0x7f2a7c062220 0x7f2a7c0a94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:30.926 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a79d9b700 1 -- 172.21.15.189:0/3775722031 >> v2:172.21.15.47:3300/0 conn(0x7f2a7c062220 msgr2=0x7f2a7c0a94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:30.927 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a79d9b700 1 --2- 172.21.15.189:0/3775722031 >> v2:172.21.15.47:3300/0 conn(0x7f2a7c062220 0x7f2a7c0a94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:30.927 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a79d9b700 1 -- 172.21.15.189:0/3775722031 >> v1:172.21.15.189:6789/0 conn(0x7f2a7c141e60 legacy=0x7f2a7c0679e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:30.927 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a7959a700 1 -- 172.21.15.189:0/3775722031 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381255866 0 0) 0x7f2a7c13c390 con 0x7f2a7c141e60 2020-11-20T12:12:30.927 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a79d9b700 1 -- 172.21.15.189:0/3775722031 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a7c142370 con 0x7f2a7c062600 2020-11-20T12:12:30.927 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a79d9b700 1 --2- 172.21.15.189:0/3775722031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a7c062600 0x7f2a7c0a9fb0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f2a7c0a9b40 tx=0x7f2a6c0091d0).ready entity=mon.1 client_cookie=3dc2b90cfefbe768 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:30.928 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a7959a700 1 -- 172.21.15.189:0/3775722031 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c00ba70 con 0x7f2a7c062600 2020-11-20T12:12:30.928 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a7959a700 1 -- 172.21.15.189:0/3775722031 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a6c00bbd0 con 0x7f2a7c062600 2020-11-20T12:12:30.928 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a7959a700 1 -- 172.21.15.189:0/3775722031 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c011400 con 0x7f2a7c062600 2020-11-20T12:12:30.931 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a7b59e700 1 -- 172.21.15.189:0/3775722031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a7c062600 msgr2=0x7f2a7c0a9fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:30.931 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a7b59e700 1 --2- 172.21.15.189:0/3775722031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a7c062600 0x7f2a7c0a9fb0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f2a7c0a9b40 tx=0x7f2a6c0091d0).stop 2020-11-20T12:12:30.932 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a7b59e700 1 -- 172.21.15.189:0/3775722031 shutdown_connections 2020-11-20T12:12:30.932 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a7b59e700 1 --2- 172.21.15.189:0/3775722031 >> v2:172.21.15.47:3300/0 conn(0x7f2a7c062220 0x7f2a7c0a94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:30.933 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a7b59e700 1 --2- 172.21.15.189:0/3775722031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a7c062600 0x7f2a7c0a9fb0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:30.933 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a7b59e700 1 -- 172.21.15.189:0/3775722031 >> 172.21.15.189:0/3775722031 conn(0x7f2a7c065fe0 msgr2=0x7f2a7c061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:30.933 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a7b59e700 1 -- 172.21.15.189:0/3775722031 shutdown_connections 2020-11-20T12:12:30.933 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.914+0000 7f2a7b59e700 1 -- 172.21.15.189:0/3775722031 wait complete. 2020-11-20T12:12:30.933 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.918+0000 7f2a7b59e700 1 Processor -- start 2020-11-20T12:12:30.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.918+0000 7f2a7b59e700 1 -- start start 2020-11-20T12:12:30.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.918+0000 7f2a7b59e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2a7c062220 0x7f2a7c0a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:30.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.918+0000 7f2a7b59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a7c062600 0x7f2a7c13eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:30.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.918+0000 7f2a7b59e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a7c141e60 0x7f2a7c09e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:30.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.918+0000 7f2a7b59e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7c0a1370 con 0x7f2a7c141e60 2020-11-20T12:12:30.935 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.918+0000 7f2a7b59e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7c0a10d0 con 0x7f2a7c062600 2020-11-20T12:12:30.935 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.918+0000 7f2a7b59e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7c09dbd0 con 0x7f2a7c062220 2020-11-20T12:12:30.935 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.918+0000 7f2a79d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a7c062600 0x7f2a7c13eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:30.935 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.918+0000 7f2a79d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a7c062600 0x7f2a7c13eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50448/0 (socket says 172.21.15.189:50448) 2020-11-20T12:12:30.935 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.918+0000 7f2a79d9b700 1 -- 172.21.15.189:0/3190783329 learned_addr learned my addr 172.21.15.189:0/3190783329 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:30.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.918+0000 7f2a7a59c700 1 --2- 172.21.15.189:0/3190783329 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2a7c062220 0x7f2a7c0a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:30.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.918+0000 7f2a7ad9d700 1 --2- 172.21.15.189:0/3190783329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a7c141e60 0x7f2a7c09e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:30.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.918+0000 7f2a79d9b700 1 -- 172.21.15.189:0/3190783329 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2a7c062220 msgr2=0x7f2a7c0a0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:30.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.918+0000 7f2a79d9b700 1 --2- 172.21.15.189:0/3190783329 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2a7c062220 0x7f2a7c0a0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:30.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.918+0000 7f2a79d9b700 1 -- 172.21.15.189:0/3190783329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a7c141e60 msgr2=0x7f2a7c09e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:30.937 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.918+0000 7f2a79d9b700 1 --2- 172.21.15.189:0/3190783329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a7c141e60 0x7f2a7c09e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:30.937 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.918+0000 7f2a79d9b700 1 -- 172.21.15.189:0/3190783329 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a7c13f420 con 0x7f2a7c062600 2020-11-20T12:12:30.937 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.918+0000 7f2a7ad9d700 1 --2- 172.21.15.189:0/3190783329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a7c141e60 0x7f2a7c09e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:12:30.937 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.918+0000 7f2a7a59c700 1 --2- 172.21.15.189:0/3190783329 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2a7c062220 0x7f2a7c0a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:12:30.937 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.918+0000 7f2a79d9b700 1 --2- 172.21.15.189:0/3190783329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a7c062600 0x7f2a7c13eb70 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f2a6c001150 tx=0x7f2a6c016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:30.938 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.918+0000 7f2a727fc700 1 -- 172.21.15.189:0/3190783329 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c01c5f0 con 0x7f2a7c062600 2020-11-20T12:12:30.938 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.918+0000 7f2a727fc700 1 -- 172.21.15.189:0/3190783329 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a6c01c750 con 0x7f2a7c062600 2020-11-20T12:12:30.938 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.918+0000 7f2a7b59e700 1 -- 172.21.15.189:0/3190783329 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a7c09ed70 con 0x7f2a7c062600 2020-11-20T12:12:30.938 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.918+0000 7f2a727fc700 1 -- 172.21.15.189:0/3190783329 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c01cad0 con 0x7f2a7c062600 2020-11-20T12:12:30.938 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.918+0000 7f2a7b59e700 1 -- 172.21.15.189:0/3190783329 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a7c09e780 con 0x7f2a7c062600 2020-11-20T12:12:30.939 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.922+0000 7f2a7b59e700 1 -- 172.21.15.189:0/3190783329 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a7c09e9a0 con 0x7f2a7c062600 2020-11-20T12:12:30.940 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.922+0000 7f2a727fc700 1 -- 172.21.15.189:0/3190783329 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f2a6c024030 con 0x7f2a7c062600 2020-11-20T12:12:30.940 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.922+0000 7f2a727fc700 1 --2- 172.21.15.189:0/3190783329 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2a580557f0 0x7f2a58057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:30.940 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.922+0000 7f2a727fc700 1 -- 172.21.15.189:0/3190783329 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f2a6c0701b0 con 0x7f2a7c062600 2020-11-20T12:12:30.940 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.922+0000 7f2a7a59c700 1 --2- 172.21.15.189:0/3190783329 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2a580557f0 0x7f2a58057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:30.940 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.922+0000 7f2a7a59c700 1 --2- 172.21.15.189:0/3190783329 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2a580557f0 0x7f2a58057cb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f2a64006fd0 tx=0x7f2a64008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:30.941 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:30.930+0000 7f2a727fc700 1 -- 172.21.15.189:0/3190783329 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2a6c045fd0 con 0x7f2a7c062600 2020-11-20T12:12:31.137 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:31.126+0000 7f2a7b59e700 1 -- 172.21.15.189:0/3190783329 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a7c09dd90 con 0x7f2a7c062600 2020-11-20T12:12:31.138 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:31.130+0000 7f2a727fc700 1 -- 172.21.15.189:0/3190783329 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f2a6c022020 con 0x7f2a7c062600 2020-11-20T12:12:31.138 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:12:31.139 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5203,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.189:6835/2651986971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2651986971},{"type":"v1","addr":"172.21.15.189:6835","nonce":2651986971}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:05.684503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5122":{"gid":5122,"name":"b","rank":0,"incarnation":27,"state":"up:active","state_seq":14,"addr":"172.21.15.47:6833/4088050649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4088050649},{"type":"v1","addr":"172.21.15.47:6833","nonce":4088050649}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:12:31.140 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:31.130+0000 7f2a7b59e700 1 -- 172.21.15.189:0/3190783329 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2a580557f0 msgr2=0x7f2a58057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:31.141 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:31.130+0000 7f2a7b59e700 1 --2- 172.21.15.189:0/3190783329 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2a580557f0 0x7f2a58057cb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f2a64006fd0 tx=0x7f2a64008040).stop 2020-11-20T12:12:31.141 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:31.130+0000 7f2a7b59e700 1 -- 172.21.15.189:0/3190783329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a7c062600 msgr2=0x7f2a7c13eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:31.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:31.130+0000 7f2a7b59e700 1 --2- 172.21.15.189:0/3190783329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a7c062600 0x7f2a7c13eb70 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f2a6c001150 tx=0x7f2a6c016000).stop 2020-11-20T12:12:31.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:31.130+0000 7f2a7b59e700 1 -- 172.21.15.189:0/3190783329 shutdown_connections 2020-11-20T12:12:31.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:31.130+0000 7f2a7b59e700 1 --2- 172.21.15.189:0/3190783329 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2a580557f0 0x7f2a58057cb0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:31.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:31.130+0000 7f2a7b59e700 1 --2- 172.21.15.189:0/3190783329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2a7c141e60 0x7f2a7c09e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:31.143 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:31.130+0000 7f2a7b59e700 1 --2- 172.21.15.189:0/3190783329 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2a7c062220 0x7f2a7c0a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:31.143 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:31.130+0000 7f2a7b59e700 1 --2- 172.21.15.189:0/3190783329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a7c062600 0x7f2a7c13eb70 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:31.143 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:31.130+0000 7f2a7b59e700 1 -- 172.21.15.189:0/3190783329 >> 172.21.15.189:0/3190783329 conn(0x7f2a7c065fe0 msgr2=0x7f2a7c0a8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:31.143 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:31.130+0000 7f2a7b59e700 1 -- 172.21.15.189:0/3190783329 shutdown_connections 2020-11-20T12:12:31.144 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:31.130+0000 7f2a7b59e700 1 -- 172.21.15.189:0/3190783329 wait complete. 2020-11-20T12:12:31.144 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 31 2020-11-20T12:12:31.153 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:12:33.155 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:33.291 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.282+0000 7fcc3515f700 1 Processor -- start 2020-11-20T12:12:33.291 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.282+0000 7fcc3515f700 1 -- start start 2020-11-20T12:12:33.292 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.282+0000 7fcc3515f700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc30067ca0 0x7fcc3013b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:33.293 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.282+0000 7fcc3515f700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fcc300ae390 0x7fcc30137b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:33.293 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.282+0000 7fcc3515f700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc300aeb40 con 0x7fcc300678c0 2020-11-20T12:12:33.293 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.282+0000 7fcc3515f700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc300ae8a0 con 0x7fcc300ae390 2020-11-20T12:12:33.293 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.282+0000 7fcc3515f700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc30137690 con 0x7fcc30067ca0 2020-11-20T12:12:33.295 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.282+0000 7fcc2e59c700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc30067ca0 0x7fcc3013b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:33.295 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.282+0000 7fcc2f59e700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fcc300ae390 0x7fcc30137b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:33.295 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.282+0000 7fcc2ed9d700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fcc300678c0 0x7fcc300a8c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.189:46740/0 (socket says 172.21.15.189:46740) 2020-11-20T12:12:33.295 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.282+0000 7fcc2ed9d700 1 -- 172.21.15.189:0/4187955124 learned_addr learned my addr 172.21.15.189:0/4187955124 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:12:33.296 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.282+0000 7fcc2dd9b700 1 -- 172.21.15.189:0/4187955124 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2006229873 0 0) 0x7fcc300aeb40 con 0x7fcc300678c0 2020-11-20T12:12:33.296 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.282+0000 7fcc2dd9b700 1 -- 172.21.15.189:0/4187955124 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc1c003240 con 0x7fcc300678c0 2020-11-20T12:12:33.296 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.282+0000 7fcc2e59c700 1 -- 172.21.15.189:0/4187955124 >> v2:172.21.15.47:3300/0 conn(0x7fcc300ae390 msgr2=0x7fcc30137b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:33.297 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.282+0000 7fcc2e59c700 1 --2- 172.21.15.189:0/4187955124 >> v2:172.21.15.47:3300/0 conn(0x7fcc300ae390 0x7fcc30137b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:33.297 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.282+0000 7fcc2e59c700 1 -- 172.21.15.189:0/4187955124 >> v1:172.21.15.47:6789/0 conn(0x7fcc300678c0 legacy=0x7fcc300a8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:33.297 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.282+0000 7fcc2e59c700 1 -- 172.21.15.189:0/4187955124 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3013b820 con 0x7fcc30067ca0 2020-11-20T12:12:33.297 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.282+0000 7fcc2f59e700 1 --2- 172.21.15.189:0/4187955124 >> v2:172.21.15.47:3300/0 conn(0x7fcc300ae390 0x7fcc30137b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:12:33.298 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.286+0000 7fcc2e59c700 1 --2- 172.21.15.189:0/4187955124 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc30067ca0 0x7fcc3013b330 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fcc2400b9a0 tx=0x7fcc24006df0).ready entity=mon.2 client_cookie=67d1ac61cdd36b6b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:33.298 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.286+0000 7fcc2dd9b700 1 -- 172.21.15.189:0/4187955124 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc24009680 con 0x7fcc30067ca0 2020-11-20T12:12:33.298 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.286+0000 7fcc2dd9b700 1 -- 172.21.15.189:0/4187955124 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc240097e0 con 0x7fcc30067ca0 2020-11-20T12:12:33.298 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.286+0000 7fcc2dd9b700 1 -- 172.21.15.189:0/4187955124 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc24009b60 con 0x7fcc30067ca0 2020-11-20T12:12:33.299 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.286+0000 7fcc3515f700 1 -- 172.21.15.189:0/4187955124 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc30067ca0 msgr2=0x7fcc3013b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:33.300 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.286+0000 7fcc3515f700 1 --2- 172.21.15.189:0/4187955124 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc30067ca0 0x7fcc3013b330 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fcc2400b9a0 tx=0x7fcc24006df0).stop 2020-11-20T12:12:33.300 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.286+0000 7fcc3515f700 1 -- 172.21.15.189:0/4187955124 shutdown_connections 2020-11-20T12:12:33.300 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.286+0000 7fcc3515f700 1 --2- 172.21.15.189:0/4187955124 >> v2:172.21.15.47:3300/0 conn(0x7fcc300ae390 0x7fcc30137b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:33.301 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.286+0000 7fcc3515f700 1 --2- 172.21.15.189:0/4187955124 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc30067ca0 0x7fcc3013b330 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:33.301 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.286+0000 7fcc3515f700 1 -- 172.21.15.189:0/4187955124 >> 172.21.15.189:0/4187955124 conn(0x7fcc300625a0 msgr2=0x7fcc300adc80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:33.301 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.286+0000 7fcc3515f700 1 -- 172.21.15.189:0/4187955124 shutdown_connections 2020-11-20T12:12:33.302 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.286+0000 7fcc3515f700 1 -- 172.21.15.189:0/4187955124 wait complete. 2020-11-20T12:12:33.302 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.290+0000 7fcc3515f700 1 Processor -- start 2020-11-20T12:12:33.302 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.290+0000 7fcc3515f700 1 -- start start 2020-11-20T12:12:33.305 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.290+0000 7fcc3515f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc300678c0 0x7fcc300a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:33.306 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.290+0000 7fcc3515f700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc30067ca0 0x7fcc3013a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:33.306 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.290+0000 7fcc3515f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc300ae390 0x7fcc300a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:33.306 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.290+0000 7fcc3515f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc300a6ce0 con 0x7fcc300678c0 2020-11-20T12:12:33.306 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.290+0000 7fcc3515f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc300a67a0 con 0x7fcc300ae390 2020-11-20T12:12:33.307 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.290+0000 7fcc3515f700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc300a6a40 con 0x7fcc30067ca0 2020-11-20T12:12:33.307 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.290+0000 7fcc2f59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc300ae390 0x7fcc300a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:33.307 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.290+0000 7fcc2ed9d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc300678c0 0x7fcc300a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:33.308 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.290+0000 7fcc2f59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc300ae390 0x7fcc300a5360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50464/0 (socket says 172.21.15.189:50464) 2020-11-20T12:12:33.308 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.290+0000 7fcc2f59e700 1 -- 172.21.15.189:0/3787125544 learned_addr learned my addr 172.21.15.189:0/3787125544 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:33.308 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.290+0000 7fcc2e59c700 1 --2- 172.21.15.189:0/3787125544 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc30067ca0 0x7fcc3013a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:33.308 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.290+0000 7fcc2f59e700 1 -- 172.21.15.189:0/3787125544 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc30067ca0 msgr2=0x7fcc3013a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:33.308 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.290+0000 7fcc2f59e700 1 --2- 172.21.15.189:0/3787125544 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc30067ca0 0x7fcc3013a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:33.309 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.290+0000 7fcc2f59e700 1 -- 172.21.15.189:0/3787125544 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc300678c0 msgr2=0x7fcc300a6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:33.309 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.290+0000 7fcc2f59e700 1 --2- 172.21.15.189:0/3787125544 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc300678c0 0x7fcc300a6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:33.309 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.290+0000 7fcc2f59e700 1 -- 172.21.15.189:0/3787125544 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc300a4110 con 0x7fcc300ae390 2020-11-20T12:12:33.310 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.290+0000 7fcc2f59e700 1 --2- 172.21.15.189:0/3787125544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc300ae390 0x7fcc300a5360 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7fcc2800ec80 tx=0x7fcc28006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:33.310 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.290+0000 7fcc16ffd700 1 -- 172.21.15.189:0/3787125544 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2800c9b0 con 0x7fcc300ae390 2020-11-20T12:12:33.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.290+0000 7fcc16ffd700 1 -- 172.21.15.189:0/3787125544 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc2800cb10 con 0x7fcc300ae390 2020-11-20T12:12:33.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.290+0000 7fcc3515f700 1 -- 172.21.15.189:0/3787125544 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc3009c9b0 con 0x7fcc300ae390 2020-11-20T12:12:33.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.290+0000 7fcc16ffd700 1 -- 172.21.15.189:0/3787125544 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2800ceb0 con 0x7fcc300ae390 2020-11-20T12:12:33.312 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.290+0000 7fcc3515f700 1 -- 172.21.15.189:0/3787125544 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc300a5a70 con 0x7fcc300ae390 2020-11-20T12:12:33.312 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.294+0000 7fcc3515f700 1 -- 172.21.15.189:0/3787125544 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc300a50a0 con 0x7fcc300ae390 2020-11-20T12:12:33.313 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.294+0000 7fcc16ffd700 1 -- 172.21.15.189:0/3787125544 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fcc28012600 con 0x7fcc300ae390 2020-11-20T12:12:33.313 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.294+0000 7fcc16ffd700 1 --2- 172.21.15.189:0/3787125544 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcc0c0557c0 0x7fcc0c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:33.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.294+0000 7fcc16ffd700 1 -- 172.21.15.189:0/3787125544 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6503+0+0 (secure 0 0 0) 0x7fcc2806ec50 con 0x7fcc300ae390 2020-11-20T12:12:33.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.294+0000 7fcc2ed9d700 1 --2- 172.21.15.189:0/3787125544 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcc0c0557c0 0x7fcc0c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:33.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.294+0000 7fcc2ed9d700 1 --2- 172.21.15.189:0/3787125544 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcc0c0557c0 0x7fcc0c057c80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fcc20005920 tx=0x7fcc20009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:33.315 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.298+0000 7fcc16ffd700 1 -- 172.21.15.189:0/3787125544 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcc28044940 con 0x7fcc300ae390 2020-11-20T12:12:33.508 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.498+0000 7fcc3515f700 1 -- 172.21.15.189:0/3787125544 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc300a5cc0 con 0x7fcc300ae390 2020-11-20T12:12:33.509 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.498+0000 7fcc16ffd700 1 -- 172.21.15.189:0/3787125544 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fcc28047f60 con 0x7fcc300ae390 2020-11-20T12:12:33.510 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:12:33.510 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5203,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.189:6835/2651986971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2651986971},{"type":"v1","addr":"172.21.15.189:6835","nonce":2651986971}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:05.684503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5122":{"gid":5122,"name":"b","rank":0,"incarnation":27,"state":"up:active","state_seq":14,"addr":"172.21.15.47:6833/4088050649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4088050649},{"type":"v1","addr":"172.21.15.47:6833","nonce":4088050649}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:12:33.511 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.502+0000 7fcc3515f700 1 -- 172.21.15.189:0/3787125544 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcc0c0557c0 msgr2=0x7fcc0c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:33.512 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.502+0000 7fcc3515f700 1 --2- 172.21.15.189:0/3787125544 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcc0c0557c0 0x7fcc0c057c80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fcc20005920 tx=0x7fcc20009040).stop 2020-11-20T12:12:33.512 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.502+0000 7fcc3515f700 1 -- 172.21.15.189:0/3787125544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc300ae390 msgr2=0x7fcc300a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:33.512 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.502+0000 7fcc3515f700 1 --2- 172.21.15.189:0/3787125544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc300ae390 0x7fcc300a5360 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7fcc2800ec80 tx=0x7fcc28006df0).stop 2020-11-20T12:12:33.512 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.502+0000 7fcc3515f700 1 -- 172.21.15.189:0/3787125544 shutdown_connections 2020-11-20T12:12:33.513 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.502+0000 7fcc3515f700 1 --2- 172.21.15.189:0/3787125544 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcc0c0557c0 0x7fcc0c057c80 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:33.513 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.502+0000 7fcc3515f700 1 --2- 172.21.15.189:0/3787125544 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc300678c0 0x7fcc300a6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:33.513 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.502+0000 7fcc3515f700 1 --2- 172.21.15.189:0/3787125544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc300ae390 0x7fcc300a5360 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:33.513 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.502+0000 7fcc3515f700 1 --2- 172.21.15.189:0/3787125544 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc30067ca0 0x7fcc3013a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:33.513 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.502+0000 7fcc3515f700 1 -- 172.21.15.189:0/3787125544 >> 172.21.15.189:0/3787125544 conn(0x7fcc300625a0 msgr2=0x7fcc300abb80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:33.514 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.502+0000 7fcc3515f700 1 -- 172.21.15.189:0/3787125544 shutdown_connections 2020-11-20T12:12:33.514 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:33.502+0000 7fcc3515f700 1 -- 172.21.15.189:0/3787125544 wait complete. 2020-11-20T12:12:33.514 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 31 2020-11-20T12:12:33.525 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:12:35.527 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:35.657 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.646+0000 7f3be2307700 1 Processor -- start 2020-11-20T12:12:35.657 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.646+0000 7f3be2307700 1 -- start start 2020-11-20T12:12:35.658 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.646+0000 7f3be2307700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3bdc0623c0 0x7f3bdc0677f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:35.659 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.646+0000 7f3be2307700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3bdc1347f0 0x7f3bdc13af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:35.659 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.650+0000 7f3be2307700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bdc13b430 con 0x7f3bdc13aac0 2020-11-20T12:12:35.659 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.650+0000 7f3be2307700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bdc0a8e60 con 0x7f3bdc1347f0 2020-11-20T12:12:35.659 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.650+0000 7f3be2307700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bdc0a9c60 con 0x7f3bdc0623c0 2020-11-20T12:12:35.660 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.650+0000 7f3be1b06700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3bdc1347f0 0x7f3bdc13af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:35.660 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.650+0000 7f3be1b06700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3bdc1347f0 0x7f3bdc13af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50474/0 (socket says 172.21.15.189:50474) 2020-11-20T12:12:35.660 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.650+0000 7f3be1305700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3bdc0623c0 0x7f3bdc0677f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:35.660 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.650+0000 7f3be1b06700 1 -- 172.21.15.189:0/1655964447 learned_addr learned my addr 172.21.15.189:0/1655964447 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:35.661 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.650+0000 7f3be1b06700 1 -- 172.21.15.189:0/1655964447 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3bdc0623c0 msgr2=0x7f3bdc0677f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:35.661 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.650+0000 7f3be1b06700 1 --2- 172.21.15.189:0/1655964447 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3bdc0623c0 0x7f3bdc0677f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:35.661 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.650+0000 7f3be1b06700 1 -- 172.21.15.189:0/1655964447 >> v1:172.21.15.47:6789/0 conn(0x7f3bdc13aac0 legacy=0x7f3bdc1340e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:35.661 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.650+0000 7f3be1b06700 1 -- 172.21.15.189:0/1655964447 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bdc067e90 con 0x7f3bdc1347f0 2020-11-20T12:12:35.663 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.650+0000 7f3be1b06700 1 --2- 172.21.15.189:0/1655964447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bdc1347f0 0x7f3bdc13af40 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f3bd4009d50 tx=0x7f3bd4009290).ready entity=mon.1 client_cookie=cb18fb9f7dfebb6c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:35.664 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.650+0000 7f3bcbfff700 1 -- 172.21.15.189:0/1655964447 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd400bbc0 con 0x7f3bdc1347f0 2020-11-20T12:12:35.664 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.650+0000 7f3bcbfff700 1 -- 172.21.15.189:0/1655964447 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd400bd20 con 0x7f3bdc1347f0 2020-11-20T12:12:35.664 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.650+0000 7f3bcbfff700 1 -- 172.21.15.189:0/1655964447 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd4011610 con 0x7f3bdc1347f0 2020-11-20T12:12:35.664 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.650+0000 7f3be2307700 1 -- 172.21.15.189:0/1655964447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bdc1347f0 msgr2=0x7f3bdc13af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:35.665 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.650+0000 7f3be2307700 1 --2- 172.21.15.189:0/1655964447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bdc1347f0 0x7f3bdc13af40 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f3bd4009d50 tx=0x7f3bd4009290).stop 2020-11-20T12:12:35.665 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.650+0000 7f3be2307700 1 -- 172.21.15.189:0/1655964447 shutdown_connections 2020-11-20T12:12:35.665 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.650+0000 7f3be2307700 1 --2- 172.21.15.189:0/1655964447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bdc1347f0 0x7f3bdc13af40 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:35.665 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.650+0000 7f3be2307700 1 --2- 172.21.15.189:0/1655964447 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3bdc0623c0 0x7f3bdc0677f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:35.665 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.650+0000 7f3be2307700 1 -- 172.21.15.189:0/1655964447 >> 172.21.15.189:0/1655964447 conn(0x7f3bdc065fe0 msgr2=0x7f3bdc061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:35.666 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.650+0000 7f3be2307700 1 -- 172.21.15.189:0/1655964447 shutdown_connections 2020-11-20T12:12:35.666 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.650+0000 7f3be2307700 1 -- 172.21.15.189:0/1655964447 wait complete. 2020-11-20T12:12:35.666 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.650+0000 7f3be2307700 1 Processor -- start 2020-11-20T12:12:35.666 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.650+0000 7f3be2307700 1 -- start start 2020-11-20T12:12:35.666 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.654+0000 7f3be2307700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3bdc0623c0 0x7f3bdc0ac550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:35.667 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.654+0000 7f3be2307700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bdc1347f0 0x7f3bdc0a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:35.667 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.654+0000 7f3be2307700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bdc13aac0 0x7f3bdc138c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:35.667 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.654+0000 7f3be2307700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bdc14d340 con 0x7f3bdc1347f0 2020-11-20T12:12:35.667 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.654+0000 7f3be2307700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bdc14d5e0 con 0x7f3bdc13aac0 2020-11-20T12:12:35.667 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.654+0000 7f3be2307700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bdc14d880 con 0x7f3bdc0623c0 2020-11-20T12:12:35.668 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.654+0000 7f3be1b06700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bdc13aac0 0x7f3bdc138c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:35.668 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.654+0000 7f3be1b06700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bdc13aac0 0x7f3bdc138c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50480/0 (socket says 172.21.15.189:50480) 2020-11-20T12:12:35.668 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.654+0000 7f3be1b06700 1 -- 172.21.15.189:0/3115026414 learned_addr learned my addr 172.21.15.189:0/3115026414 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:35.668 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.654+0000 7f3be1305700 1 --2- 172.21.15.189:0/3115026414 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3bdc0623c0 0x7f3bdc0ac550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:35.668 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.654+0000 7f3be0b04700 1 --2- 172.21.15.189:0/3115026414 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bdc1347f0 0x7f3bdc0a0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:35.669 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.654+0000 7f3be1b06700 1 -- 172.21.15.189:0/3115026414 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3bdc0623c0 msgr2=0x7f3bdc0ac550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:35.669 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.654+0000 7f3be1b06700 1 --2- 172.21.15.189:0/3115026414 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3bdc0623c0 0x7f3bdc0ac550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:35.669 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.654+0000 7f3be1b06700 1 -- 172.21.15.189:0/3115026414 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bdc1347f0 msgr2=0x7f3bdc0a0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:35.669 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.654+0000 7f3be1b06700 1 --2- 172.21.15.189:0/3115026414 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bdc1347f0 0x7f3bdc0a0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:35.669 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.654+0000 7f3be1b06700 1 -- 172.21.15.189:0/3115026414 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bdc0a13d0 con 0x7f3bdc13aac0 2020-11-20T12:12:35.670 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.654+0000 7f3be1305700 1 --2- 172.21.15.189:0/3115026414 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3bdc0623c0 0x7f3bdc0ac550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:12:35.670 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.654+0000 7f3be1b06700 1 --2- 172.21.15.189:0/3115026414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bdc13aac0 0x7f3bdc138c30 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f3bd4009fd0 tx=0x7f3bd4016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:35.671 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.658+0000 7f3bc97fa700 1 -- 172.21.15.189:0/3115026414 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd4011e70 con 0x7f3bdc13aac0 2020-11-20T12:12:35.671 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.658+0000 7f3be2307700 1 -- 172.21.15.189:0/3115026414 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bdc14ce00 con 0x7f3bdc13aac0 2020-11-20T12:12:35.672 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.658+0000 7f3bc97fa700 1 -- 172.21.15.189:0/3115026414 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd400be90 con 0x7f3bdc13aac0 2020-11-20T12:12:35.672 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.658+0000 7f3bc97fa700 1 -- 172.21.15.189:0/3115026414 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd4010e30 con 0x7f3bdc13aac0 2020-11-20T12:12:35.672 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.658+0000 7f3be2307700 1 -- 172.21.15.189:0/3115026414 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bdc139120 con 0x7f3bdc13aac0 2020-11-20T12:12:35.672 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.658+0000 7f3be2307700 1 -- 172.21.15.189:0/3115026414 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bdc14da40 con 0x7f3bdc13aac0 2020-11-20T12:12:35.672 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.658+0000 7f3bc97fa700 1 -- 172.21.15.189:0/3115026414 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f3bd4023030 con 0x7f3bdc13aac0 2020-11-20T12:12:35.673 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.658+0000 7f3bc97fa700 1 --2- 172.21.15.189:0/3115026414 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3bb40557c0 0x7f3bb4057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:35.673 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.658+0000 7f3bc97fa700 1 -- 172.21.15.189:0/3115026414 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6503+0+0 (secure 0 0 0) 0x7f3bd4070130 con 0x7f3bdc13aac0 2020-11-20T12:12:35.673 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.658+0000 7f3be1305700 1 --2- 172.21.15.189:0/3115026414 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3bb40557c0 0x7f3bb4057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:35.673 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.658+0000 7f3be1305700 1 --2- 172.21.15.189:0/3115026414 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3bb40557c0 0x7f3bb4057c80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f3bd8006fd0 tx=0x7f3bd8008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:35.676 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.666+0000 7f3bc97fa700 1 -- 172.21.15.189:0/3115026414 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3bd4045ed0 con 0x7f3bdc13aac0 2020-11-20T12:12:35.873 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.862+0000 7f3be2307700 1 -- 172.21.15.189:0/3115026414 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bdc0663c0 con 0x7f3bdc13aac0 2020-11-20T12:12:35.874 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.866+0000 7f3bc97fa700 1 -- 172.21.15.189:0/3115026414 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f3bd4021020 con 0x7f3bdc13aac0 2020-11-20T12:12:35.874 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:12:35.874 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5203,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.189:6835/2651986971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2651986971},{"type":"v1","addr":"172.21.15.189:6835","nonce":2651986971}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:05.684503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5122":{"gid":5122,"name":"b","rank":0,"incarnation":27,"state":"up:active","state_seq":14,"addr":"172.21.15.47:6833/4088050649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":4088050649},{"type":"v1","addr":"172.21.15.47:6833","nonce":4088050649}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:12:35.876 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.866+0000 7f3be2307700 1 -- 172.21.15.189:0/3115026414 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3bb40557c0 msgr2=0x7f3bb4057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:35.876 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.866+0000 7f3be2307700 1 --2- 172.21.15.189:0/3115026414 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3bb40557c0 0x7f3bb4057c80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f3bd8006fd0 tx=0x7f3bd8008040).stop 2020-11-20T12:12:35.877 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.866+0000 7f3be2307700 1 -- 172.21.15.189:0/3115026414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bdc13aac0 msgr2=0x7f3bdc138c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:35.877 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.866+0000 7f3be2307700 1 --2- 172.21.15.189:0/3115026414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bdc13aac0 0x7f3bdc138c30 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f3bd4009fd0 tx=0x7f3bd4016000).stop 2020-11-20T12:12:35.877 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.866+0000 7f3be2307700 1 -- 172.21.15.189:0/3115026414 shutdown_connections 2020-11-20T12:12:35.877 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.866+0000 7f3be2307700 1 --2- 172.21.15.189:0/3115026414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bdc13aac0 0x7f3bdc138c30 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:35.877 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.866+0000 7f3be2307700 1 --2- 172.21.15.189:0/3115026414 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3bdc0623c0 0x7f3bdc0ac550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:35.878 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.866+0000 7f3be2307700 1 --2- 172.21.15.189:0/3115026414 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3bb40557c0 0x7f3bb4057c80 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:35.878 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.866+0000 7f3be2307700 1 --2- 172.21.15.189:0/3115026414 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3bdc1347f0 0x7f3bdc0a0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:35.878 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.866+0000 7f3be2307700 1 -- 172.21.15.189:0/3115026414 >> 172.21.15.189:0/3115026414 conn(0x7f3bdc065fe0 msgr2=0x7f3bdc0a6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:35.878 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.866+0000 7f3be2307700 1 -- 172.21.15.189:0/3115026414 shutdown_connections 2020-11-20T12:12:35.878 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:35.866+0000 7f3be2307700 1 -- 172.21.15.189:0/3115026414 wait complete. 2020-11-20T12:12:35.879 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 31 2020-11-20T12:12:35.889 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:12:37.891 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:38.009 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:37.998+0000 7fb1d888f700 1 Processor -- start 2020-11-20T12:12:38.009 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:37.998+0000 7fb1d888f700 1 -- start start 2020-11-20T12:12:38.010 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d888f700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb1d40623c0 0x7fb1d4138290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:38.010 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d888f700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb1d4060bc0 0x7fb1d4061040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d888f700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1d4061530 con 0x7fb1d413b860 2020-11-20T12:12:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d888f700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1d40687e0 con 0x7fb1d4060bc0 2020-11-20T12:12:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d888f700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d4065ca0 con 0x7fb1d40623c0 2020-11-20T12:12:38.011 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d37fe700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb1d40623c0 0x7fb1d4138290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d2ffd700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb1d4060bc0 0x7fb1d4061040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d37fe700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb1d40623c0 0x7fb1d4138290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.189:43526/0 (socket says 172.21.15.189:43526) 2020-11-20T12:12:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d2ffd700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb1d4060bc0 0x7fb1d4061040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.189:56718/0 (socket says 172.21.15.189:56718) 2020-11-20T12:12:38.012 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d37fe700 1 -- 172.21.15.189:0/1217825435 learned_addr learned my addr 172.21.15.189:0/1217825435 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:38.014 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d37fe700 1 -- 172.21.15.189:0/1217825435 >> v2:172.21.15.47:3300/0 conn(0x7fb1d4060bc0 msgr2=0x7fb1d4061040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:38.014 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d37fe700 1 --2- 172.21.15.189:0/1217825435 >> v2:172.21.15.47:3300/0 conn(0x7fb1d4060bc0 0x7fb1d4061040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:38.014 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d37fe700 1 -- 172.21.15.189:0/1217825435 >> v1:172.21.15.47:6789/0 conn(0x7fb1d413b860 legacy=0x7fb1d413bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:38.015 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d37fe700 1 -- 172.21.15.189:0/1217825435 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d4138ba0 con 0x7fb1d40623c0 2020-11-20T12:12:38.015 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d2ffd700 1 --2- 172.21.15.189:0/1217825435 >> v2:172.21.15.47:3300/0 conn(0x7fb1d4060bc0 0x7fb1d4061040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:12:38.015 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d37fe700 1 --2- 172.21.15.189:0/1217825435 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb1d40623c0 0x7fb1d4138290 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fb1cc009d50 tx=0x7fb1cc009290).ready entity=mon.2 client_cookie=a53f185d8db79a38 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:38.015 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d27fc700 1 -- 172.21.15.189:0/1217825435 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc00bc50 con 0x7fb1d40623c0 2020-11-20T12:12:38.015 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d27fc700 1 -- 172.21.15.189:0/1217825435 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1cc00bdb0 con 0x7fb1d40623c0 2020-11-20T12:12:38.016 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d27fc700 1 -- 172.21.15.189:0/1217825435 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc011610 con 0x7fb1d40623c0 2020-11-20T12:12:38.016 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d888f700 1 -- 172.21.15.189:0/1217825435 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb1d40623c0 msgr2=0x7fb1d4138290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:38.016 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d888f700 1 --2- 172.21.15.189:0/1217825435 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb1d40623c0 0x7fb1d4138290 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fb1cc009d50 tx=0x7fb1cc009290).stop 2020-11-20T12:12:38.016 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d888f700 1 -- 172.21.15.189:0/1217825435 shutdown_connections 2020-11-20T12:12:38.016 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d888f700 1 --2- 172.21.15.189:0/1217825435 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb1d40623c0 0x7fb1d4138290 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:38.017 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d888f700 1 --2- 172.21.15.189:0/1217825435 >> v2:172.21.15.47:3300/0 conn(0x7fb1d4060bc0 0x7fb1d4061040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:38.017 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d888f700 1 -- 172.21.15.189:0/1217825435 >> 172.21.15.189:0/1217825435 conn(0x7fb1d4065fe0 msgr2=0x7fb1d4061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:38.017 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d888f700 1 -- 172.21.15.189:0/1217825435 shutdown_connections 2020-11-20T12:12:38.017 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.002+0000 7fb1d888f700 1 -- 172.21.15.189:0/1217825435 wait complete. 2020-11-20T12:12:38.019 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.006+0000 7fb1d888f700 1 Processor -- start 2020-11-20T12:12:38.019 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.006+0000 7fb1d888f700 1 -- start start 2020-11-20T12:12:38.019 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.006+0000 7fb1d888f700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb1d4060bc0 0x7fb1d40ac860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:38.020 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.006+0000 7fb1d888f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1d40623c0 0x7fb1d40a8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:38.020 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.006+0000 7fb1d888f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1d413b860 0x7fb1d41374c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:38.020 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.006+0000 7fb1d888f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d4147ca0 con 0x7fb1d40623c0 2020-11-20T12:12:38.020 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.006+0000 7fb1d888f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d4147f40 con 0x7fb1d413b860 2020-11-20T12:12:38.021 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.006+0000 7fb1d888f700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d41481e0 con 0x7fb1d4060bc0 2020-11-20T12:12:38.021 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.006+0000 7fb1d37fe700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb1d4060bc0 0x7fb1d40ac860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:38.021 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.006+0000 7fb1d3fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1d413b860 0x7fb1d41374c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:38.021 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.006+0000 7fb1d37fe700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb1d4060bc0 0x7fb1d40ac860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.189:43532/0 (socket says 172.21.15.189:43532) 2020-11-20T12:12:38.021 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.006+0000 7fb1d37fe700 1 -- 172.21.15.189:0/4082890916 learned_addr learned my addr 172.21.15.189:0/4082890916 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:38.022 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.006+0000 7fb1d2ffd700 1 --2- 172.21.15.189:0/4082890916 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1d40623c0 0x7fb1d40a8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:38.022 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.006+0000 7fb1d37fe700 1 -- 172.21.15.189:0/4082890916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1d413b860 msgr2=0x7fb1d41374c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:38.022 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.006+0000 7fb1d37fe700 1 --2- 172.21.15.189:0/4082890916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1d413b860 0x7fb1d41374c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:38.022 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.006+0000 7fb1d37fe700 1 -- 172.21.15.189:0/4082890916 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1d40623c0 msgr2=0x7fb1d40a8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:38.022 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.006+0000 7fb1d37fe700 1 --2- 172.21.15.189:0/4082890916 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1d40623c0 0x7fb1d40a8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:38.023 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.006+0000 7fb1d37fe700 1 -- 172.21.15.189:0/4082890916 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d40a8a70 con 0x7fb1d4060bc0 2020-11-20T12:12:38.023 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.006+0000 7fb1d3fff700 1 --2- 172.21.15.189:0/4082890916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1d413b860 0x7fb1d41374c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:12:38.023 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.006+0000 7fb1d37fe700 1 --2- 172.21.15.189:0/4082890916 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb1d4060bc0 0x7fb1d40ac860 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fb1d4065de0 tx=0x7fb1cc016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:38.023 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.006+0000 7fb1cb7fe700 1 -- 172.21.15.189:0/4082890916 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc01b5e0 con 0x7fb1d4060bc0 2020-11-20T12:12:38.023 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.006+0000 7fb1cb7fe700 1 -- 172.21.15.189:0/4082890916 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1cc01b740 con 0x7fb1d4060bc0 2020-11-20T12:12:38.024 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.006+0000 7fb1d888f700 1 -- 172.21.15.189:0/4082890916 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1d4147720 con 0x7fb1d4060bc0 2020-11-20T12:12:38.024 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.006+0000 7fb1cb7fe700 1 -- 172.21.15.189:0/4082890916 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc01bac0 con 0x7fb1d4060bc0 2020-11-20T12:12:38.024 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.006+0000 7fb1d888f700 1 -- 172.21.15.189:0/4082890916 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1d41379b0 con 0x7fb1d4060bc0 2020-11-20T12:12:38.024 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.010+0000 7fb1d888f700 1 -- 172.21.15.189:0/4082890916 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1d40663c0 con 0x7fb1d4060bc0 2020-11-20T12:12:38.025 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.010+0000 7fb1cb7fe700 1 -- 172.21.15.189:0/4082890916 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fb1cc017070 con 0x7fb1d4060bc0 2020-11-20T12:12:38.026 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.010+0000 7fb1cb7fe700 1 --2- 172.21.15.189:0/4082890916 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb1b00557c0 0x7fb1b0057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:38.026 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.010+0000 7fb1cb7fe700 1 -- 172.21.15.189:0/4082890916 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6589+0+0 (secure 0 0 0) 0x7fb1cc0707b0 con 0x7fb1d4060bc0 2020-11-20T12:12:38.026 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.010+0000 7fb1d2ffd700 1 --2- 172.21.15.189:0/4082890916 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb1b00557c0 0x7fb1b0057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:38.027 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.010+0000 7fb1d2ffd700 1 --2- 172.21.15.189:0/4082890916 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb1b00557c0 0x7fb1b0057c80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fb1d40ad7e0 tx=0x7fb1bc006df0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:38.031 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.022+0000 7fb1cb7fe700 1 -- 172.21.15.189:0/4082890916 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb1cc046340 con 0x7fb1d4060bc0 2020-11-20T12:12:38.233 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.222+0000 7fb1d888f700 1 -- 172.21.15.189:0/4082890916 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1d40446d0 con 0x7fb1d4060bc0 2020-11-20T12:12:38.234 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.226+0000 7fb1cb7fe700 1 -- 172.21.15.189:0/4082890916 <== mon.2 v2:172.21.15.47:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+1787 (secure 0 0 0) 0x7fb1cc015070 con 0x7fb1d4060bc0 2020-11-20T12:12:38.234 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:12:38.235 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:37.618599+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5203":{"gid":5203,"name":"a","rank":0,"incarnation":32,"state":"up:replay","state_seq":2,"addr":"172.21.15.189:6835/2651986971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2651986971},{"type":"v1","addr":"172.21.15.189:6835","nonce":2651986971}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:12:38.236 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.226+0000 7fb1d888f700 1 -- 172.21.15.189:0/4082890916 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb1b00557c0 msgr2=0x7fb1b0057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:38.237 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.226+0000 7fb1d888f700 1 --2- 172.21.15.189:0/4082890916 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb1b00557c0 0x7fb1b0057c80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fb1d40ad7e0 tx=0x7fb1bc006df0).stop 2020-11-20T12:12:38.237 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.226+0000 7fb1d888f700 1 -- 172.21.15.189:0/4082890916 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb1d4060bc0 msgr2=0x7fb1d40ac860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:38.237 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.226+0000 7fb1d888f700 1 --2- 172.21.15.189:0/4082890916 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb1d4060bc0 0x7fb1d40ac860 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fb1d4065de0 tx=0x7fb1cc016000).stop 2020-11-20T12:12:38.237 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.226+0000 7fb1d888f700 1 -- 172.21.15.189:0/4082890916 shutdown_connections 2020-11-20T12:12:38.238 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.226+0000 7fb1d888f700 1 --2- 172.21.15.189:0/4082890916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1d413b860 0x7fb1d41374c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:38.238 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.226+0000 7fb1d888f700 1 --2- 172.21.15.189:0/4082890916 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb1d4060bc0 0x7fb1d40ac860 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:38.238 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.226+0000 7fb1d888f700 1 --2- 172.21.15.189:0/4082890916 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb1b00557c0 0x7fb1b0057c80 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:38.238 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.226+0000 7fb1d888f700 1 --2- 172.21.15.189:0/4082890916 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb1d40623c0 0x7fb1d40a8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:38.238 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.226+0000 7fb1d888f700 1 -- 172.21.15.189:0/4082890916 >> 172.21.15.189:0/4082890916 conn(0x7fb1d4065fe0 msgr2=0x7fb1d4132e30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:38.239 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.226+0000 7fb1d888f700 1 -- 172.21.15.189:0/4082890916 shutdown_connections 2020-11-20T12:12:38.239 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.226+0000 7fb1d888f700 1 -- 172.21.15.189:0/4082890916 wait complete. 2020-11-20T12:12:38.239 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 32 2020-11-20T12:12:38.250 INFO:tasks.mds_thrash.fs.[None]:mds.b down, removed from mdsmap 2020-11-20T12:12:38.250 INFO:tasks.mds_thrash.fs.[None]:waiting for mds cluster to stabilize... 2020-11-20T12:12:38.250 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:38.369 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.358+0000 7f1bb670d700 1 Processor -- start 2020-11-20T12:12:38.369 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.358+0000 7f1bb670d700 1 -- start start 2020-11-20T12:12:38.372 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1bb670d700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1bb00a9800 0x7f1bb013ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:38.372 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1bb670d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1bb013f260 0x7f1bb013b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:38.373 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1bb670d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bb00676a0 con 0x7f1bb013be50 2020-11-20T12:12:38.373 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1bb670d700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bb0065ec0 con 0x7f1bb00a9800 2020-11-20T12:12:38.373 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1bb670d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bb0067c50 con 0x7f1bb013f260 2020-11-20T12:12:38.373 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1bb4caa700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1bb013be50 0x7f1bb0060170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52130/0 (socket says 172.21.15.189:52130) 2020-11-20T12:12:38.374 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1bb4caa700 1 -- 172.21.15.189:0/4291652096 learned_addr learned my addr 172.21.15.189:0/4291652096 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:12:38.374 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baf7fe700 1 --2- 172.21.15.189:0/4291652096 >> v2:172.21.15.189:3300/0 conn(0x7f1bb013f260 0x7f1bb013b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:38.374 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baffff700 1 --2- 172.21.15.189:0/4291652096 >> v2:172.21.15.47:3300/0 conn(0x7f1bb00a9800 0x7f1bb013ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:38.374 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baeffd700 1 -- 172.21.15.189:0/4291652096 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723184472 0 0) 0x7f1bb00676a0 con 0x7f1bb013be50 2020-11-20T12:12:38.374 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baeffd700 1 -- 172.21.15.189:0/4291652096 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b9c003240 con 0x7f1bb013be50 2020-11-20T12:12:38.377 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baeffd700 1 -- 172.21.15.189:0/4291652096 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1710474316 0 0) 0x7f1b9c003240 con 0x7f1bb013be50 2020-11-20T12:12:38.377 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baeffd700 1 -- 172.21.15.189:0/4291652096 >> v2:172.21.15.189:3300/0 conn(0x7f1bb013f260 msgr2=0x7f1bb013b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:38.377 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baeffd700 1 --2- 172.21.15.189:0/4291652096 >> v2:172.21.15.189:3300/0 conn(0x7f1bb013f260 0x7f1bb013b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:38.378 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baeffd700 1 -- 172.21.15.189:0/4291652096 >> v2:172.21.15.47:3300/0 conn(0x7f1bb00a9800 msgr2=0x7f1bb013ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:38.378 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baeffd700 1 --2- 172.21.15.189:0/4291652096 >> v2:172.21.15.47:3300/0 conn(0x7f1bb00a9800 0x7f1bb013ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:38.378 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baeffd700 1 -- 172.21.15.189:0/4291652096 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb0067160 con 0x7f1bb013be50 2020-11-20T12:12:38.378 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baeffd700 1 -- 172.21.15.189:0/4291652096 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f1ba40014e0 con 0x7f1bb013be50 2020-11-20T12:12:38.378 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baeffd700 1 -- 172.21.15.189:0/4291652096 >> v1:172.21.15.189:6789/0 conn(0x7f1bb013be50 legacy=0x7f1bb0060170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:38.379 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baeffd700 1 --2- 172.21.15.189:0/4291652096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b9c003730 0x7f1b9c003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:38.379 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baeffd700 1 --2- 172.21.15.189:0/4291652096 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b9c0044e0 0x7f1b9c004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:38.379 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baeffd700 1 --2- 172.21.15.189:0/4291652096 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1b9c004e50 0x7f1b9c00b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:38.379 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baeffd700 1 -- 172.21.15.189:0/4291652096 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb0067c50 con 0x7f1b9c0044e0 2020-11-20T12:12:38.379 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baeffd700 1 -- 172.21.15.189:0/4291652096 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c004260 con 0x7f1b9c003730 2020-11-20T12:12:38.380 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baeffd700 1 -- 172.21.15.189:0/4291652096 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c00bc20 con 0x7f1b9c004e50 2020-11-20T12:12:38.380 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baffff700 1 --2- 172.21.15.189:0/4291652096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b9c003730 0x7f1b9c003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:38.380 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baf7fe700 1 --2- 172.21.15.189:0/4291652096 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b9c0044e0 0x7f1b9c004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:38.380 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baffff700 1 -- 172.21.15.189:0/4291652096 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1b9c004e50 msgr2=0x7f1b9c00b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:38.380 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baffff700 1 --2- 172.21.15.189:0/4291652096 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1b9c004e50 0x7f1b9c00b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:12:38.381 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baffff700 1 -- 172.21.15.189:0/4291652096 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b9c0044e0 msgr2=0x7f1b9c004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:38.381 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baffff700 1 --2- 172.21.15.189:0/4291652096 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b9c0044e0 0x7f1b9c004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:38.381 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baffff700 1 -- 172.21.15.189:0/4291652096 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b9c003240 con 0x7f1b9c003730 2020-11-20T12:12:38.381 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baf7fe700 1 --2- 172.21.15.189:0/4291652096 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b9c0044e0 0x7f1b9c004960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:12:38.381 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1bb4caa700 1 -- 172.21.15.189:0/4291652096 reap_dead start 2020-11-20T12:12:38.382 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baffff700 1 --2- 172.21.15.189:0/4291652096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b9c003730 0x7f1b9c003b50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f1ba000bc00 tx=0x7f1ba0006df0).ready entity=mon.1 client_cookie=944816dcc08c1687 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:38.382 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baeffd700 1 -- 172.21.15.189:0/4291652096 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba000b310 con 0x7f1b9c003730 2020-11-20T12:12:38.382 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baeffd700 1 -- 172.21.15.189:0/4291652096 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba000eb40 con 0x7f1b9c003730 2020-11-20T12:12:38.382 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.362+0000 7f1baeffd700 1 -- 172.21.15.189:0/4291652096 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba000eec0 con 0x7f1b9c003730 2020-11-20T12:12:38.382 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.366+0000 7f1bb670d700 1 -- 172.21.15.189:0/4291652096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b9c003730 msgr2=0x7f1b9c003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:38.383 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.366+0000 7f1bb670d700 1 --2- 172.21.15.189:0/4291652096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b9c003730 0x7f1b9c003b50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f1ba000bc00 tx=0x7f1ba0006df0).stop 2020-11-20T12:12:38.383 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.366+0000 7f1bb670d700 1 -- 172.21.15.189:0/4291652096 shutdown_connections 2020-11-20T12:12:38.383 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.366+0000 7f1bb670d700 1 --2- 172.21.15.189:0/4291652096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b9c003730 0x7f1b9c003b50 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:38.383 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.366+0000 7f1bb670d700 1 -- 172.21.15.189:0/4291652096 >> 172.21.15.189:0/4291652096 conn(0x7f1bb0061f60 msgr2=0x7f1bb0062370 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:38.383 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.366+0000 7f1bb670d700 1 -- 172.21.15.189:0/4291652096 shutdown_connections 2020-11-20T12:12:38.384 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.366+0000 7f1bb670d700 1 -- 172.21.15.189:0/4291652096 wait complete. 2020-11-20T12:12:38.386 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.366+0000 7f1bb670d700 1 Processor -- start 2020-11-20T12:12:38.387 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.366+0000 7f1bb670d700 1 -- start start 2020-11-20T12:12:38.387 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.370+0000 7f1bb670d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b9c003730 0x7f1bb00ab270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:38.387 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.370+0000 7f1bb670d700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1bb00a4550 0x7f1bb00ac260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:38.387 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.370+0000 7f1bb670d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb00ac750 0x7f1bb0151840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:38.387 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.370+0000 7f1bb670d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb0151480 con 0x7f1b9c003730 2020-11-20T12:12:38.388 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.370+0000 7f1bb670d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb00abad0 con 0x7f1bb00ac750 2020-11-20T12:12:38.388 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.370+0000 7f1bb670d700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb00ab830 con 0x7f1bb00a4550 2020-11-20T12:12:38.388 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.370+0000 7f1bb4caa700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb00ac750 0x7f1bb0151840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:38.388 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.370+0000 7f1bb4caa700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb00ac750 0x7f1bb0151840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50520/0 (socket says 172.21.15.189:50520) 2020-11-20T12:12:38.388 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.370+0000 7f1bb4caa700 1 -- 172.21.15.189:0/983187633 learned_addr learned my addr 172.21.15.189:0/983187633 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:38.389 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.370+0000 7f1baffff700 1 --2- 172.21.15.189:0/983187633 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b9c003730 0x7f1bb00ab270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:38.389 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.370+0000 7f1baf7fe700 1 --2- 172.21.15.189:0/983187633 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1bb00a4550 0x7f1bb00ac260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:38.389 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.370+0000 7f1bb4caa700 1 -- 172.21.15.189:0/983187633 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1bb00a4550 msgr2=0x7f1bb00ac260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:38.389 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.370+0000 7f1bb4caa700 1 --2- 172.21.15.189:0/983187633 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1bb00a4550 0x7f1bb00ac260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:38.389 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.370+0000 7f1bb4caa700 1 -- 172.21.15.189:0/983187633 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b9c003730 msgr2=0x7f1bb00ab270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:38.390 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.370+0000 7f1bb4caa700 1 --2- 172.21.15.189:0/983187633 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b9c003730 0x7f1bb00ab270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:38.390 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.370+0000 7f1bb4caa700 1 -- 172.21.15.189:0/983187633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb00abdc0 con 0x7f1bb00ac750 2020-11-20T12:12:38.390 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.370+0000 7f1baf7fe700 1 --2- 172.21.15.189:0/983187633 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1bb00a4550 0x7f1bb00ac260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:12:38.390 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.370+0000 7f1bb4caa700 1 --2- 172.21.15.189:0/983187633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb00ac750 0x7f1bb0151840 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f1bb00666b0 tx=0x7f1bb00672f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:38.390 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.370+0000 7f1b8ffff700 1 -- 172.21.15.189:0/983187633 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba400ceb0 con 0x7f1bb00ac750 2020-11-20T12:12:38.391 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.370+0000 7f1b8ffff700 1 -- 172.21.15.189:0/983187633 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba40070d0 con 0x7f1bb00ac750 2020-11-20T12:12:38.391 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.370+0000 7f1bb670d700 1 -- 172.21.15.189:0/983187633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bb0142830 con 0x7f1bb00ac750 2020-11-20T12:12:38.391 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.370+0000 7f1b8ffff700 1 -- 172.21.15.189:0/983187633 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4004710 con 0x7f1bb00ac750 2020-11-20T12:12:38.391 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.370+0000 7f1bb670d700 1 -- 172.21.15.189:0/983187633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb00a4b70 con 0x7f1bb00ac750 2020-11-20T12:12:38.391 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.374+0000 7f1bb670d700 1 -- 172.21.15.189:0/983187633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bb00a9ed0 con 0x7f1bb00ac750 2020-11-20T12:12:38.391 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.374+0000 7f1b8ffff700 1 -- 172.21.15.189:0/983187633 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f1ba401b070 con 0x7f1bb00ac750 2020-11-20T12:12:38.392 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.374+0000 7f1b8ffff700 1 --2- 172.21.15.189:0/983187633 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1b880557f0 0x7f1b88057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:38.392 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.374+0000 7f1b8ffff700 1 -- 172.21.15.189:0/983187633 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6589+0+0 (secure 0 0 0) 0x7f1ba406dad0 con 0x7f1bb00ac750 2020-11-20T12:12:38.392 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.374+0000 7f1baffff700 1 --2- 172.21.15.189:0/983187633 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1b880557f0 0x7f1b88057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:38.392 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.374+0000 7f1baffff700 1 --2- 172.21.15.189:0/983187633 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1b880557f0 0x7f1b88057cb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f1ba000b4d0 tx=0x7f1ba0019000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:38.393 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.378+0000 7f1b8ffff700 1 -- 172.21.15.189:0/983187633 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1bb00a9ed0 con 0x7f1bb00ac750 2020-11-20T12:12:38.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.578+0000 7f1bb670d700 1 -- 172.21.15.189:0/983187633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1bb00a9ed0 con 0x7f1bb00ac750 2020-11-20T12:12:38.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.578+0000 7f1b8ffff700 1 -- 172.21.15.189:0/983187633 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+1787 (secure 0 0 0) 0x7f1bb00a9ed0 con 0x7f1bb00ac750 2020-11-20T12:12:38.588 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:12:38.588 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:37.618599+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5203":{"gid":5203,"name":"a","rank":0,"incarnation":32,"state":"up:replay","state_seq":2,"addr":"172.21.15.189:6835/2651986971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2651986971},{"type":"v1","addr":"172.21.15.189:6835","nonce":2651986971}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:12:38.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.578+0000 7f1bb670d700 1 -- 172.21.15.189:0/983187633 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1b880557f0 msgr2=0x7f1b88057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:38.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.578+0000 7f1bb670d700 1 --2- 172.21.15.189:0/983187633 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1b880557f0 0x7f1b88057cb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f1ba000b4d0 tx=0x7f1ba0019000).stop 2020-11-20T12:12:38.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.578+0000 7f1bb670d700 1 -- 172.21.15.189:0/983187633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb00ac750 msgr2=0x7f1bb0151840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:38.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.578+0000 7f1bb670d700 1 --2- 172.21.15.189:0/983187633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb00ac750 0x7f1bb0151840 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f1bb00666b0 tx=0x7f1bb00672f0).stop 2020-11-20T12:12:38.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.578+0000 7f1bb670d700 1 -- 172.21.15.189:0/983187633 shutdown_connections 2020-11-20T12:12:38.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.578+0000 7f1bb670d700 1 --2- 172.21.15.189:0/983187633 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1b880557f0 0x7f1b88057cb0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:38.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.578+0000 7f1bb670d700 1 --2- 172.21.15.189:0/983187633 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1b9c003730 0x7f1bb00ab270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:38.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.578+0000 7f1bb670d700 1 --2- 172.21.15.189:0/983187633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bb00ac750 0x7f1bb0151840 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:38.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.578+0000 7f1bb670d700 1 --2- 172.21.15.189:0/983187633 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1bb00a4550 0x7f1bb00ac260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:38.592 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.578+0000 7f1bb670d700 1 -- 172.21.15.189:0/983187633 >> 172.21.15.189:0/983187633 conn(0x7f1bb0061f60 msgr2=0x7f1bb00608c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:38.592 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.578+0000 7f1bb670d700 1 -- 172.21.15.189:0/983187633 shutdown_connections 2020-11-20T12:12:38.592 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:38.578+0000 7f1bb670d700 1 -- 172.21.15.189:0/983187633 wait complete. 2020-11-20T12:12:38.593 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 32 2020-11-20T12:12:38.602 INFO:tasks.mds_thrash.fs.[None]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 32, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2020-11-20T12:04:54.012110+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 32, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5203": { "addr": "172.21.15.189:6835/2651986971", "addrs": { "addrvec": [ { "addr": "172.21.15.189:6834", "nonce": 2651986971, "type": "v2" }, { "addr": "172.21.15.189:6835", "nonce": 2651986971, "type": "v1" } ] }, "export_targets": [], "features": 4540138297136906239, "flags": 0, "gid": 5203, "incarnation": 32, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 28, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2020-11-20T12:12:37.618599+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5203 } } } ], "standbys": [] } 2020-11-20T12:12:40.603 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:40.736 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.726+0000 7f3dfd41c700 1 Processor -- start 2020-11-20T12:12:40.736 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.726+0000 7f3dfd41c700 1 -- start start 2020-11-20T12:12:40.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.726+0000 7f3dfd41c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3df80623c0 0x7f3df80677f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:40.740 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.726+0000 7f3dfd41c700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3df81347f0 0x7f3df813af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:40.741 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.726+0000 7f3dfd41c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3df813b460 con 0x7f3df813aac0 2020-11-20T12:12:40.742 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.726+0000 7f3dfd41c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3df80a8e60 con 0x7f3df80623c0 2020-11-20T12:12:40.742 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.726+0000 7f3dfd41c700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3df80a9c60 con 0x7f3df81347f0 2020-11-20T12:12:40.742 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.726+0000 7f3df7fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3df80623c0 0x7f3df80677f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:40.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.726+0000 7f3df77fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3df813aac0 0x7f3df81340e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52156/0 (socket says 172.21.15.189:52156) 2020-11-20T12:12:40.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.726+0000 7f3df77fe700 1 -- 172.21.15.189:0/4012409876 learned_addr learned my addr 172.21.15.189:0/4012409876 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:12:40.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.726+0000 7f3dfcc1b700 1 --2- 172.21.15.189:0/4012409876 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3df81347f0 0x7f3df813af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:40.744 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.726+0000 7f3df6ffd700 1 -- 172.21.15.189:0/4012409876 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2145524219 0 0) 0x7f3df813b460 con 0x7f3df813aac0 2020-11-20T12:12:40.744 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.726+0000 7f3df6ffd700 1 -- 172.21.15.189:0/4012409876 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3de4003240 con 0x7f3df813aac0 2020-11-20T12:12:40.745 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.726+0000 7f3df7fff700 1 -- 172.21.15.189:0/4012409876 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3df81347f0 msgr2=0x7f3df813af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:40.745 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.726+0000 7f3df7fff700 1 --2- 172.21.15.189:0/4012409876 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3df81347f0 0x7f3df813af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:40.745 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.726+0000 7f3df7fff700 1 -- 172.21.15.189:0/4012409876 >> v1:172.21.15.189:6789/0 conn(0x7f3df813aac0 legacy=0x7f3df81340e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:40.745 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.726+0000 7f3df7fff700 1 -- 172.21.15.189:0/4012409876 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df8067e90 con 0x7f3df80623c0 2020-11-20T12:12:40.746 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.726+0000 7f3df7fff700 1 --2- 172.21.15.189:0/4012409876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3df80623c0 0x7f3df80677f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f3df813b5f0 tx=0x7f3df00091d0).ready entity=mon.1 client_cookie=baf850e31d3b2b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:40.746 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.726+0000 7f3df6ffd700 1 -- 172.21.15.189:0/4012409876 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df000b9c0 con 0x7f3df80623c0 2020-11-20T12:12:40.746 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.726+0000 7f3df6ffd700 1 -- 172.21.15.189:0/4012409876 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df000bb20 con 0x7f3df80623c0 2020-11-20T12:12:40.747 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.726+0000 7f3df6ffd700 1 -- 172.21.15.189:0/4012409876 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df000bea0 con 0x7f3df80623c0 2020-11-20T12:12:40.747 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.726+0000 7f3dfd41c700 1 -- 172.21.15.189:0/4012409876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3df80623c0 msgr2=0x7f3df80677f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:40.747 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.726+0000 7f3dfd41c700 1 --2- 172.21.15.189:0/4012409876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3df80623c0 0x7f3df80677f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f3df813b5f0 tx=0x7f3df00091d0).stop 2020-11-20T12:12:40.747 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3dfd41c700 1 -- 172.21.15.189:0/4012409876 shutdown_connections 2020-11-20T12:12:40.747 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3dfd41c700 1 --2- 172.21.15.189:0/4012409876 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3df81347f0 0x7f3df813af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:40.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3dfd41c700 1 --2- 172.21.15.189:0/4012409876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3df80623c0 0x7f3df80677f0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:40.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3dfd41c700 1 -- 172.21.15.189:0/4012409876 >> 172.21.15.189:0/4012409876 conn(0x7f3df8065fe0 msgr2=0x7f3df8061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:40.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3dfd41c700 1 -- 172.21.15.189:0/4012409876 shutdown_connections 2020-11-20T12:12:40.749 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3dfd41c700 1 -- 172.21.15.189:0/4012409876 wait complete. 2020-11-20T12:12:40.749 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3dfd41c700 1 Processor -- start 2020-11-20T12:12:40.749 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3dfd41c700 1 -- start start 2020-11-20T12:12:40.752 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3dfd41c700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3df80623c0 0x7f3df80ab940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:40.752 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3dfd41c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3df81347f0 0x7f3df8135a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:40.753 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3dfd41c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3df813aac0 0x7f3df81376f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:40.753 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3dfd41c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df81350b0 con 0x7f3df813aac0 2020-11-20T12:12:40.753 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3dfd41c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df8135310 con 0x7f3df81347f0 2020-11-20T12:12:40.753 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3dfd41c700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3df81355b0 con 0x7f3df80623c0 2020-11-20T12:12:40.754 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3df77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3df81347f0 0x7f3df8135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:40.754 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3df77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3df81347f0 0x7f3df8135a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50538/0 (socket says 172.21.15.189:50538) 2020-11-20T12:12:40.754 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3df77fe700 1 -- 172.21.15.189:0/4008702045 learned_addr learned my addr 172.21.15.189:0/4008702045 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:40.754 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3df7fff700 1 --2- 172.21.15.189:0/4008702045 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3df80623c0 0x7f3df80ab940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:40.754 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3dfcc1b700 1 --2- 172.21.15.189:0/4008702045 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3df813aac0 0x7f3df81376f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:40.755 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3df77fe700 1 -- 172.21.15.189:0/4008702045 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3df80623c0 msgr2=0x7f3df80ab940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:40.755 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3df77fe700 1 --2- 172.21.15.189:0/4008702045 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3df80623c0 0x7f3df80ab940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:40.755 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3df77fe700 1 -- 172.21.15.189:0/4008702045 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3df813aac0 msgr2=0x7f3df81376f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:40.755 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3df77fe700 1 --2- 172.21.15.189:0/4008702045 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3df813aac0 0x7f3df81376f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:40.756 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3df77fe700 1 -- 172.21.15.189:0/4008702045 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df8135f40 con 0x7f3df81347f0 2020-11-20T12:12:40.756 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3dfcc1b700 1 --2- 172.21.15.189:0/4008702045 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3df813aac0 0x7f3df81376f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:12:40.756 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3df77fe700 1 --2- 172.21.15.189:0/4008702045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3df81347f0 0x7f3df8135a50 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f3de800c710 tx=0x7f3de8007ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:40.756 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3dd7fff700 1 -- 172.21.15.189:0/4008702045 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de80072e0 con 0x7f3df81347f0 2020-11-20T12:12:40.757 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3dd7fff700 1 -- 172.21.15.189:0/4008702045 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3de8007440 con 0x7f3df81347f0 2020-11-20T12:12:40.757 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3dfd41c700 1 -- 172.21.15.189:0/4008702045 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3df80a5cc0 con 0x7f3df81347f0 2020-11-20T12:12:40.757 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3dd7fff700 1 -- 172.21.15.189:0/4008702045 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de8005800 con 0x7f3df81347f0 2020-11-20T12:12:40.757 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.730+0000 7f3dfd41c700 1 -- 172.21.15.189:0/4008702045 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3df8137e00 con 0x7f3df81347f0 2020-11-20T12:12:40.758 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.734+0000 7f3dfd41c700 1 -- 172.21.15.189:0/4008702045 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3df80a5ee0 con 0x7f3df81347f0 2020-11-20T12:12:40.758 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.734+0000 7f3dd7fff700 1 -- 172.21.15.189:0/4008702045 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f3de8010070 con 0x7f3df81347f0 2020-11-20T12:12:40.758 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.734+0000 7f3dd7fff700 1 --2- 172.21.15.189:0/4008702045 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3dd00557c0 0x7f3dd0057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:40.758 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.734+0000 7f3dd7fff700 1 -- 172.21.15.189:0/4008702045 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6589+0+0 (secure 0 0 0) 0x7f3de806de40 con 0x7f3df81347f0 2020-11-20T12:12:40.759 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.734+0000 7f3df7fff700 1 --2- 172.21.15.189:0/4008702045 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3dd00557c0 0x7f3dd0057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:40.759 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.734+0000 7f3df7fff700 1 --2- 172.21.15.189:0/4008702045 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3dd00557c0 0x7f3dd0057c80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f3df0009440 tx=0x7f3df0016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:40.759 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.738+0000 7f3dd7fff700 1 -- 172.21.15.189:0/4008702045 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3de8043ae0 con 0x7f3df81347f0 2020-11-20T12:12:40.961 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.950+0000 7f3dfd41c700 1 -- 172.21.15.189:0/4008702045 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3df80663c0 con 0x7f3df81347f0 2020-11-20T12:12:40.962 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.950+0000 7f3dd7fff700 1 -- 172.21.15.189:0/4008702045 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+1788 (secure 0 0 0) 0x7f3de8047100 con 0x7f3df81347f0 2020-11-20T12:12:40.962 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:12:40.962 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:40.691744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5203":{"gid":5203,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/2651986971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2651986971},{"type":"v1","addr":"172.21.15.189:6835","nonce":2651986971}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:12:40.963 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.954+0000 7f3dfd41c700 1 -- 172.21.15.189:0/4008702045 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3dd00557c0 msgr2=0x7f3dd0057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:40.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.954+0000 7f3dfd41c700 1 --2- 172.21.15.189:0/4008702045 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3dd00557c0 0x7f3dd0057c80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f3df0009440 tx=0x7f3df0016000).stop 2020-11-20T12:12:40.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.954+0000 7f3dfd41c700 1 -- 172.21.15.189:0/4008702045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3df81347f0 msgr2=0x7f3df8135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:40.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.954+0000 7f3dfd41c700 1 --2- 172.21.15.189:0/4008702045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3df81347f0 0x7f3df8135a50 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f3de800c710 tx=0x7f3de8007ad0).stop 2020-11-20T12:12:40.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.954+0000 7f3dfd41c700 1 -- 172.21.15.189:0/4008702045 shutdown_connections 2020-11-20T12:12:40.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.954+0000 7f3dfd41c700 1 --2- 172.21.15.189:0/4008702045 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f3dd00557c0 0x7f3dd0057c80 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:40.965 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.954+0000 7f3dfd41c700 1 --2- 172.21.15.189:0/4008702045 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3df813aac0 0x7f3df81376f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:40.965 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.954+0000 7f3dfd41c700 1 --2- 172.21.15.189:0/4008702045 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f3df80623c0 0x7f3df80ab940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:40.965 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.954+0000 7f3dfd41c700 1 --2- 172.21.15.189:0/4008702045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3df81347f0 0x7f3df8135a50 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:40.965 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.954+0000 7f3dfd41c700 1 -- 172.21.15.189:0/4008702045 >> 172.21.15.189:0/4008702045 conn(0x7f3df8065fe0 msgr2=0x7f3df80aa850 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:40.966 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.954+0000 7f3dfd41c700 1 -- 172.21.15.189:0/4008702045 shutdown_connections 2020-11-20T12:12:40.966 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:40.954+0000 7f3dfd41c700 1 -- 172.21.15.189:0/4008702045 wait complete. 2020-11-20T12:12:40.966 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 35 2020-11-20T12:12:40.983 INFO:tasks.mds_thrash.fs.[None]:mds.a has gained rank=0, replacing gid=5122 2020-11-20T12:12:40.983 INFO:tasks.mds_thrash.fs.[None]:waiting for 2 secs before reviving mds.b 2020-11-20T12:12:42.984 INFO:tasks.mds_thrash.fs.[None]:reviving mds.b 2020-11-20T12:12:42.984 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:12:42.985 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:12:42.988 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:12:42.988 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:43.067 INFO:tasks.ceph.mds.b.smithi047.stdout:starting mds.b at 2020-11-20T12:12:43.129 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb8c27d700 1 Processor -- start 2020-11-20T12:12:43.130 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb8c27d700 1 -- start start 2020-11-20T12:12:43.130 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb8c27d700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7feb840a9800 0x7feb84065ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:43.130 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb8c27d700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7feb84137e90 0x7feb84141b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:43.131 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb8c27d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb84065aa0 con 0x7feb840a8a80 2020-11-20T12:12:43.131 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb8c27d700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb840a3e80 con 0x7feb84137e90 2020-11-20T12:12:43.132 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb8c27d700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb84137220 con 0x7feb840a9800 2020-11-20T12:12:43.132 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb89818700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7feb840a8a80 0x7feb84137780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52168/0 (socket says 172.21.15.189:52168) 2020-11-20T12:12:43.132 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb89818700 1 -- 172.21.15.189:0/3172427938 learned_addr learned my addr 172.21.15.189:0/3172427938 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:12:43.133 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb8a019700 1 --2- 172.21.15.189:0/3172427938 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7feb840a9800 0x7feb84065ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:43.133 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb8a81a700 1 --2- 172.21.15.189:0/3172427938 >> v2:172.21.15.47:3300/0 conn(0x7feb84137e90 0x7feb84141b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:43.133 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb89017700 1 -- 172.21.15.189:0/3172427938 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2856186463 0 0) 0x7feb84065aa0 con 0x7feb840a8a80 2020-11-20T12:12:43.134 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb89017700 1 -- 172.21.15.189:0/3172427938 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb6c003240 con 0x7feb840a8a80 2020-11-20T12:12:43.134 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb89017700 1 -- 172.21.15.189:0/3172427938 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1106799561 0 0) 0x7feb6c003240 con 0x7feb840a8a80 2020-11-20T12:12:43.135 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb89017700 1 -- 172.21.15.189:0/3172427938 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7feb840a9800 msgr2=0x7feb84065ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:43.135 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb89017700 1 --2- 172.21.15.189:0/3172427938 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7feb840a9800 0x7feb84065ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:43.135 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb89017700 1 -- 172.21.15.189:0/3172427938 >> v2:172.21.15.47:3300/0 conn(0x7feb84137e90 msgr2=0x7feb84141b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:43.136 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb89017700 1 --2- 172.21.15.189:0/3172427938 >> v2:172.21.15.47:3300/0 conn(0x7feb84137e90 0x7feb84141b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:43.136 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb89017700 1 -- 172.21.15.189:0/3172427938 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb840a40e0 con 0x7feb840a8a80 2020-11-20T12:12:43.136 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb89017700 1 -- 172.21.15.189:0/3172427938 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7feb740014e0 con 0x7feb840a8a80 2020-11-20T12:12:43.137 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb89017700 1 -- 172.21.15.189:0/3172427938 >> v1:172.21.15.189:6789/0 conn(0x7feb840a8a80 legacy=0x7feb84137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:43.137 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb89017700 1 --2- 172.21.15.189:0/3172427938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb6c003730 0x7feb6c003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:43.138 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb89017700 1 --2- 172.21.15.189:0/3172427938 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7feb840a9800 0x7feb6c0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:43.138 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb89017700 1 --2- 172.21.15.189:0/3172427938 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb6c0049d0 0x7feb6c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:43.138 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb89017700 1 -- 172.21.15.189:0/3172427938 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb840a3e80 con 0x7feb6c0049d0 2020-11-20T12:12:43.139 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb89017700 1 -- 172.21.15.189:0/3172427938 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb84137220 con 0x7feb6c003730 2020-11-20T12:12:43.139 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb89017700 1 -- 172.21.15.189:0/3172427938 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb6c004250 con 0x7feb840a9800 2020-11-20T12:12:43.139 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb8a019700 1 --2- 172.21.15.189:0/3172427938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb6c003730 0x7feb6c003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:43.140 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb8a81a700 1 --2- 172.21.15.189:0/3172427938 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb6c0049d0 0x7feb6c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:43.140 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb89818700 1 --2- 172.21.15.189:0/3172427938 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7feb840a9800 0x7feb6c0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:43.140 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb8a019700 1 -- 172.21.15.189:0/3172427938 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7feb840a9800 msgr2=0x7feb6c0044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:43.141 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb8a019700 1 --2- 172.21.15.189:0/3172427938 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7feb840a9800 0x7feb6c0044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:43.141 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb8a019700 1 -- 172.21.15.189:0/3172427938 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb6c0049d0 msgr2=0x7feb6c004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:43.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb8a019700 1 --2- 172.21.15.189:0/3172427938 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb6c0049d0 0x7feb6c004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:43.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb8a019700 1 -- 172.21.15.189:0/3172427938 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb6c003240 con 0x7feb6c003730 2020-11-20T12:12:43.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.114+0000 7feb8a019700 1 --2- 172.21.15.189:0/3172427938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb6c003730 0x7feb6c003b50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7feb8000bc00 tx=0x7feb80006df0).ready entity=mon.1 client_cookie=363f1cfe7a155802 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:43.143 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.118+0000 7feb89017700 1 -- 172.21.15.189:0/3172427938 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8000b310 con 0x7feb6c003730 2020-11-20T12:12:43.143 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.118+0000 7feb89017700 1 -- 172.21.15.189:0/3172427938 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb8000ea90 con 0x7feb6c003730 2020-11-20T12:12:43.143 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.118+0000 7feb89017700 1 -- 172.21.15.189:0/3172427938 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8000ee10 con 0x7feb6c003730 2020-11-20T12:12:43.144 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.118+0000 7feb8c27d700 1 -- 172.21.15.189:0/3172427938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb6c003730 msgr2=0x7feb6c003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:43.144 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.118+0000 7feb8c27d700 1 --2- 172.21.15.189:0/3172427938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb6c003730 0x7feb6c003b50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7feb8000bc00 tx=0x7feb80006df0).stop 2020-11-20T12:12:43.145 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.118+0000 7feb8a81a700 1 -- 172.21.15.189:0/3172427938 reap_dead start 2020-11-20T12:12:43.145 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.118+0000 7feb8c27d700 1 -- 172.21.15.189:0/3172427938 shutdown_connections 2020-11-20T12:12:43.145 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.118+0000 7feb8c27d700 1 -- 172.21.15.189:0/3172427938 >> 172.21.15.189:0/3172427938 conn(0x7feb84061f60 msgr2=0x7feb84062370 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:43.146 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.118+0000 7feb8c27d700 1 -- 172.21.15.189:0/3172427938 shutdown_connections 2020-11-20T12:12:43.146 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.118+0000 7feb8c27d700 1 -- 172.21.15.189:0/3172427938 wait complete. 2020-11-20T12:12:43.146 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.118+0000 7feb8c27d700 1 Processor -- start 2020-11-20T12:12:43.147 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.118+0000 7feb8c27d700 1 -- start start 2020-11-20T12:12:43.151 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.118+0000 7feb8c27d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb6c003730 0x7feb8409c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:43.152 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.118+0000 7feb8c27d700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7feb840a6320 0x7feb8413cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:43.152 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.118+0000 7feb8c27d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb8414d610 0x7feb8414da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:43.153 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.118+0000 7feb8c27d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb84149a20 con 0x7feb6c003730 2020-11-20T12:12:43.153 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.118+0000 7feb8c27d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8409cca0 con 0x7feb8414d610 2020-11-20T12:12:43.153 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.118+0000 7feb8c27d700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb8414d170 con 0x7feb840a6320 2020-11-20T12:12:43.154 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.122+0000 7feb8a81a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb8414d610 0x7feb8414da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:43.154 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.122+0000 7feb8a81a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb8414d610 0x7feb8414da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50560/0 (socket says 172.21.15.189:50560) 2020-11-20T12:12:43.155 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.122+0000 7feb8a81a700 1 -- 172.21.15.189:0/3553098426 learned_addr learned my addr 172.21.15.189:0/3553098426 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:43.155 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.122+0000 7feb8a019700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb6c003730 0x7feb8409c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:43.156 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.122+0000 7feb89818700 1 --2- 172.21.15.189:0/3553098426 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7feb840a6320 0x7feb8413cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:43.156 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.122+0000 7feb8a81a700 1 -- 172.21.15.189:0/3553098426 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7feb840a6320 msgr2=0x7feb8413cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:43.156 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.122+0000 7feb8a81a700 1 --2- 172.21.15.189:0/3553098426 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7feb840a6320 0x7feb8413cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:43.157 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.122+0000 7feb8a81a700 1 -- 172.21.15.189:0/3553098426 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb6c003730 msgr2=0x7feb8409c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:43.157 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.122+0000 7feb8a81a700 1 --2- 172.21.15.189:0/3553098426 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb6c003730 0x7feb8409c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:43.158 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.122+0000 7feb8a81a700 1 -- 172.21.15.189:0/3553098426 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb84149da0 con 0x7feb8414d610 2020-11-20T12:12:43.158 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.122+0000 7feb8a81a700 1 --2- 172.21.15.189:0/3553098426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb8414d610 0x7feb8414da90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7feb7c0090e0 tx=0x7feb7c0091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:43.158 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.122+0000 7feb727fc700 1 -- 172.21.15.189:0/3553098426 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7c00faa0 con 0x7feb8414d610 2020-11-20T12:12:43.159 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.122+0000 7feb727fc700 1 -- 172.21.15.189:0/3553098426 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb7c00fc00 con 0x7feb8414d610 2020-11-20T12:12:43.159 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.122+0000 7feb727fc700 1 -- 172.21.15.189:0/3553098426 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7c014440 con 0x7feb8414d610 2020-11-20T12:12:43.159 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.122+0000 7feb8c27d700 1 -- 172.21.15.189:0/3553098426 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb84149f90 con 0x7feb8414d610 2020-11-20T12:12:43.160 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.122+0000 7feb8c27d700 1 -- 172.21.15.189:0/3553098426 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb8414a410 con 0x7feb8414d610 2020-11-20T12:12:43.160 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.122+0000 7feb8c27d700 1 -- 172.21.15.189:0/3553098426 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb8409caf0 con 0x7feb8414d610 2020-11-20T12:12:43.161 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.122+0000 7feb727fc700 1 -- 172.21.15.189:0/3553098426 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7feb7c014830 con 0x7feb8414d610 2020-11-20T12:12:43.161 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.122+0000 7feb727fc700 1 --2- 172.21.15.189:0/3553098426 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7feb5c0557f0 0x7feb5c057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:43.161 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.122+0000 7feb727fc700 1 -- 172.21.15.189:0/3553098426 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6589+0+0 (secure 0 0 0) 0x7feb7c06ef50 con 0x7feb8414d610 2020-11-20T12:12:43.161 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.122+0000 7feb8a019700 1 --2- 172.21.15.189:0/3553098426 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7feb5c0557f0 0x7feb5c057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:43.162 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.126+0000 7feb8a019700 1 --2- 172.21.15.189:0/3553098426 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7feb5c0557f0 0x7feb5c057cb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7feb80006e30 tx=0x7feb80019000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:43.162 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.130+0000 7feb727fc700 1 -- 172.21.15.189:0/3553098426 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7feb7c044bf0 con 0x7feb8414d610 2020-11-20T12:12:43.346 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.334+0000 7feb8c27d700 1 -- 172.21.15.189:0/3553098426 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb8414d330 con 0x7feb8414d610 2020-11-20T12:12:43.347 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.338+0000 7feb727fc700 1 -- 172.21.15.189:0/3553098426 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+1788 (secure 0 0 0) 0x7feb7c018020 con 0x7feb8414d610 2020-11-20T12:12:43.347 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:12:43.347 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:40.691744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5203":{"gid":5203,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/2651986971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2651986971},{"type":"v1","addr":"172.21.15.189:6835","nonce":2651986971}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:12:43.349 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.338+0000 7feb8c27d700 1 -- 172.21.15.189:0/3553098426 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7feb5c0557f0 msgr2=0x7feb5c057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:43.349 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.338+0000 7feb8c27d700 1 --2- 172.21.15.189:0/3553098426 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7feb5c0557f0 0x7feb5c057cb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7feb80006e30 tx=0x7feb80019000).stop 2020-11-20T12:12:43.349 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.338+0000 7feb8c27d700 1 -- 172.21.15.189:0/3553098426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb8414d610 msgr2=0x7feb8414da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:43.350 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.338+0000 7feb8c27d700 1 --2- 172.21.15.189:0/3553098426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb8414d610 0x7feb8414da90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7feb7c0090e0 tx=0x7feb7c0091d0).stop 2020-11-20T12:12:43.350 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.338+0000 7feb8c27d700 1 -- 172.21.15.189:0/3553098426 shutdown_connections 2020-11-20T12:12:43.350 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.338+0000 7feb8c27d700 1 --2- 172.21.15.189:0/3553098426 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7feb5c0557f0 0x7feb5c057cb0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:43.350 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.338+0000 7feb8c27d700 1 --2- 172.21.15.189:0/3553098426 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7feb6c003730 0x7feb8409c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:43.350 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.338+0000 7feb8c27d700 1 --2- 172.21.15.189:0/3553098426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb8414d610 0x7feb8414da90 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:43.351 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.338+0000 7feb8c27d700 1 --2- 172.21.15.189:0/3553098426 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7feb840a6320 0x7feb8413cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:43.351 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.338+0000 7feb8c27d700 1 -- 172.21.15.189:0/3553098426 >> 172.21.15.189:0/3553098426 conn(0x7feb84061f60 msgr2=0x7feb840659a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:43.351 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.338+0000 7feb8c27d700 1 -- 172.21.15.189:0/3553098426 shutdown_connections 2020-11-20T12:12:43.351 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:43.338+0000 7feb8c27d700 1 -- 172.21.15.189:0/3553098426 wait complete. 2020-11-20T12:12:43.352 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 35 2020-11-20T12:12:43.366 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2020-11-20T12:12:44.634 INFO:tasks.ceph.mds.a.smithi189.stderr:2020-11-20T12:12:44.622+0000 7f2cb0475700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:12:45.367 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:12:45.509 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.498+0000 7f38ab003700 1 Processor -- start 2020-11-20T12:12:45.510 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.498+0000 7f38ab003700 1 -- start start 2020-11-20T12:12:45.511 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.502+0000 7f38ab003700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f38a4060310 0x7f38a40a1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:45.511 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.502+0000 7f38ab003700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f38a40a8450 0x7f38a40a9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:45.511 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.502+0000 7f38ab003700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a40a8fa0 con 0x7f38a40a2440 2020-11-20T12:12:45.511 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.502+0000 7f38ab003700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38a40618b0 con 0x7f38a4060310 2020-11-20T12:12:45.512 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.502+0000 7f38ab003700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38a40a9240 con 0x7f38a40a8450 2020-11-20T12:12:45.512 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.502+0000 7f38aa001700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f38a4060310 0x7f38a40a1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:45.512 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.502+0000 7f38aa001700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f38a4060310 0x7f38a40a1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50564/0 (socket says 172.21.15.189:50564) 2020-11-20T12:12:45.512 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.502+0000 7f38aa001700 1 -- 172.21.15.189:0/2341855874 learned_addr learned my addr 172.21.15.189:0/2341855874 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:45.512 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.502+0000 7f38aa802700 1 --2- 172.21.15.189:0/2341855874 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f38a40a8450 0x7f38a40a9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:45.513 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.502+0000 7f38aa001700 1 -- 172.21.15.189:0/2341855874 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f38a40a8450 msgr2=0x7f38a40a9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:12:45.513 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.502+0000 7f38aa001700 1 --2- 172.21.15.189:0/2341855874 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f38a40a8450 0x7f38a40a9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:45.519 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.502+0000 7f38aa001700 1 -- 172.21.15.189:0/2341855874 >> v1:172.21.15.47:6789/0 conn(0x7f38a40a2440 legacy=0x7f38a40a7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:45.520 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.502+0000 7f38aa001700 1 -- 172.21.15.189:0/2341855874 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a40a94e0 con 0x7f38a4060310 2020-11-20T12:12:45.520 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.502+0000 7f38aa001700 1 --2- 172.21.15.189:0/2341855874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a4060310 0x7f38a40a1f50 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f38a0006fd0 tx=0x7f38a0006eb0).ready entity=mon.1 client_cookie=435c134c167ec3a9 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:45.520 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.502+0000 7f38a8fff700 1 -- 172.21.15.189:0/2341855874 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a00098a0 con 0x7f38a4060310 2020-11-20T12:12:45.520 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.502+0000 7f38a8fff700 1 -- 172.21.15.189:0/2341855874 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a0009a00 con 0x7f38a4060310 2020-11-20T12:12:45.521 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.502+0000 7f38a8fff700 1 -- 172.21.15.189:0/2341855874 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a0009d80 con 0x7f38a4060310 2020-11-20T12:12:45.521 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.502+0000 7f38ab003700 1 -- 172.21.15.189:0/2341855874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a4060310 msgr2=0x7f38a40a1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:45.521 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.502+0000 7f38ab003700 1 --2- 172.21.15.189:0/2341855874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a4060310 0x7f38a40a1f50 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f38a0006fd0 tx=0x7f38a0006eb0).stop 2020-11-20T12:12:45.521 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.502+0000 7f38ab003700 1 -- 172.21.15.189:0/2341855874 shutdown_connections 2020-11-20T12:12:45.521 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.502+0000 7f38ab003700 1 --2- 172.21.15.189:0/2341855874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a4060310 0x7f38a40a1f50 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:45.522 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.502+0000 7f38ab003700 1 --2- 172.21.15.189:0/2341855874 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f38a40a8450 0x7f38a40a9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:45.522 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.502+0000 7f38ab003700 1 -- 172.21.15.189:0/2341855874 >> 172.21.15.189:0/2341855874 conn(0x7f38a4063210 msgr2=0x7f38a4060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:45.522 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.502+0000 7f38ab003700 1 -- 172.21.15.189:0/2341855874 shutdown_connections 2020-11-20T12:12:45.522 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.502+0000 7f38ab003700 1 -- 172.21.15.189:0/2341855874 wait complete. 2020-11-20T12:12:45.522 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.506+0000 7f38ab003700 1 Processor -- start 2020-11-20T12:12:45.523 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.506+0000 7f38ab003700 1 -- start start 2020-11-20T12:12:45.523 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.506+0000 7f38ab003700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38a4060310 0x7f38a409c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:45.523 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.506+0000 7f38ab003700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f38a40a2440 0x7f38a4147c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:45.523 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.506+0000 7f38ab003700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a40a8450 0x7f38a4136920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:45.523 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.506+0000 7f38ab003700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a41485c0 con 0x7f38a4060310 2020-11-20T12:12:45.524 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.506+0000 7f38ab003700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a4148860 con 0x7f38a40a8450 2020-11-20T12:12:45.524 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.506+0000 7f38ab003700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38a4148b00 con 0x7f38a40a2440 2020-11-20T12:12:45.524 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.506+0000 7f38aa802700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a40a8450 0x7f38a4136920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:45.524 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.506+0000 7f38aa802700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a40a8450 0x7f38a4136920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50574/0 (socket says 172.21.15.189:50574) 2020-11-20T12:12:45.524 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.506+0000 7f38aa802700 1 -- 172.21.15.189:0/1973245168 learned_addr learned my addr 172.21.15.189:0/1973245168 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:12:45.525 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.506+0000 7f38aa001700 1 --2- 172.21.15.189:0/1973245168 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38a4060310 0x7f38a409c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:45.525 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.506+0000 7f38a9800700 1 --2- 172.21.15.189:0/1973245168 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f38a40a2440 0x7f38a4147c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:45.525 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.506+0000 7f38aa802700 1 -- 172.21.15.189:0/1973245168 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f38a40a2440 msgr2=0x7f38a4147c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:45.525 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.506+0000 7f38aa802700 1 --2- 172.21.15.189:0/1973245168 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f38a40a2440 0x7f38a4147c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:45.525 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.506+0000 7f38aa802700 1 -- 172.21.15.189:0/1973245168 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38a4060310 msgr2=0x7f38a409c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:45.526 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.506+0000 7f38aa802700 1 --2- 172.21.15.189:0/1973245168 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38a4060310 0x7f38a409c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:45.526 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.506+0000 7f38aa802700 1 -- 172.21.15.189:0/1973245168 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a4138020 con 0x7f38a40a8450 2020-11-20T12:12:45.526 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.506+0000 7f38a9800700 1 --2- 172.21.15.189:0/1973245168 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f38a40a2440 0x7f38a4147c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:12:45.526 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.506+0000 7f38aa802700 1 --2- 172.21.15.189:0/1973245168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a40a8450 0x7f38a4136920 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f389c00ec30 tx=0x7f389c0091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:45.526 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.506+0000 7f3891ffb700 1 -- 172.21.15.189:0/1973245168 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389c00caa0 con 0x7f38a40a8450 2020-11-20T12:12:45.527 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.506+0000 7f3891ffb700 1 -- 172.21.15.189:0/1973245168 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f389c00cc00 con 0x7f38a40a8450 2020-11-20T12:12:45.527 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.506+0000 7f3891ffb700 1 -- 172.21.15.189:0/1973245168 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389c0123f0 con 0x7f38a40a8450 2020-11-20T12:12:45.527 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.506+0000 7f38ab003700 1 -- 172.21.15.189:0/1973245168 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a4148130 con 0x7f38a40a8450 2020-11-20T12:12:45.527 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.506+0000 7f38ab003700 1 -- 172.21.15.189:0/1973245168 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a4136fb0 con 0x7f38a40a8450 2020-11-20T12:12:45.527 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.510+0000 7f38ab003700 1 -- 172.21.15.189:0/1973245168 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38a413a8f0 con 0x7f38a40a8450 2020-11-20T12:12:45.528 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.510+0000 7f3891ffb700 1 -- 172.21.15.189:0/1973245168 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f389c022070 con 0x7f38a40a8450 2020-11-20T12:12:45.529 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.510+0000 7f3891ffb700 1 --2- 172.21.15.189:0/1973245168 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f387c0557c0 0x7f387c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:12:45.529 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.510+0000 7f3891ffb700 1 -- 172.21.15.189:0/1973245168 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6589+0+0 (secure 0 0 0) 0x7f389c06efb0 con 0x7f38a40a8450 2020-11-20T12:12:45.529 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.510+0000 7f38aa001700 1 --2- 172.21.15.189:0/1973245168 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f387c0557c0 0x7f387c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:12:45.529 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.510+0000 7f38aa001700 1 --2- 172.21.15.189:0/1973245168 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f387c0557c0 0x7f387c057c80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f38a0009fd0 tx=0x7f38a0015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:12:45.530 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.518+0000 7f3891ffb700 1 -- 172.21.15.189:0/1973245168 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f38a413a8f0 con 0x7f38a40a8450 2020-11-20T12:12:45.742 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.730+0000 7f38ab003700 1 -- 172.21.15.189:0/1973245168 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38a413a8f0 con 0x7f38a40a8450 2020-11-20T12:12:45.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.734+0000 7f3891ffb700 1 -- 172.21.15.189:0/1973245168 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f38a413a8f0 con 0x7f38a40a8450 2020-11-20T12:12:45.743 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:12:45.743 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/1811549340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1811549340},{"type":"v1","addr":"172.21.15.47:6833","nonce":1811549340}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:40.691744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5203":{"gid":5203,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/2651986971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2651986971},{"type":"v1","addr":"172.21.15.189:6835","nonce":2651986971}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:12:45.747 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.734+0000 7f38ab003700 1 -- 172.21.15.189:0/1973245168 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f387c0557c0 msgr2=0x7f387c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:45.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.734+0000 7f38ab003700 1 --2- 172.21.15.189:0/1973245168 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f387c0557c0 0x7f387c057c80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f38a0009fd0 tx=0x7f38a0015000).stop 2020-11-20T12:12:45.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.734+0000 7f38ab003700 1 -- 172.21.15.189:0/1973245168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a40a8450 msgr2=0x7f38a4136920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:12:45.749 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.734+0000 7f38ab003700 1 --2- 172.21.15.189:0/1973245168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a40a8450 0x7f38a4136920 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f389c00ec30 tx=0x7f389c0091d0).stop 2020-11-20T12:12:45.750 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.734+0000 7f38ab003700 1 -- 172.21.15.189:0/1973245168 shutdown_connections 2020-11-20T12:12:45.750 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.734+0000 7f38ab003700 1 --2- 172.21.15.189:0/1973245168 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f387c0557c0 0x7f387c057c80 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:45.750 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.734+0000 7f38ab003700 1 --2- 172.21.15.189:0/1973245168 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38a4060310 0x7f38a409c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:45.750 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.734+0000 7f38ab003700 1 --2- 172.21.15.189:0/1973245168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a40a8450 0x7f38a4136920 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:45.751 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.734+0000 7f38ab003700 1 --2- 172.21.15.189:0/1973245168 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f38a40a2440 0x7f38a4147c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:12:45.751 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.734+0000 7f38ab003700 1 -- 172.21.15.189:0/1973245168 >> 172.21.15.189:0/1973245168 conn(0x7f38a4063210 msgr2=0x7f38a40acab0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:12:45.751 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.734+0000 7f38ab003700 1 -- 172.21.15.189:0/1973245168 shutdown_connections 2020-11-20T12:12:45.751 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:12:45.734+0000 7f38ab003700 1 -- 172.21.15.189:0/1973245168 wait complete. 2020-11-20T12:12:45.751 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 36 2020-11-20T12:12:45.758 INFO:tasks.mds_thrash.fs.[None]:mds.b reported in up:standby state 2020-11-20T12:12:45.758 INFO:tasks.mds_thrash.fs.[None]:waiting for 15 secs before thrashing 2020-11-20T12:12:53.936 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:12:53.940 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:13:00.759 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:00.899 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.890+0000 7f255b59e700 1 Processor -- start 2020-11-20T12:13:00.939 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.890+0000 7f255b59e700 1 -- start start 2020-11-20T12:13:00.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.890+0000 7f255b59e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f255c0ad3e0 0x7f255c066d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:00.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.890+0000 7f255b59e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f255c13a6e0 0x7f255c13ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:00.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.890+0000 7f255b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f255c067730 con 0x7f255c0ad000 2020-11-20T12:13:00.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.890+0000 7f255b59e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f255c066050 con 0x7f255c13a6e0 2020-11-20T12:13:00.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.890+0000 7f255b59e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f255c138190 con 0x7f255c0ad3e0 2020-11-20T12:13:00.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.890+0000 7f255a59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f255c0ad000 0x7f255c13e480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52200/0 (socket says 172.21.15.189:52200) 2020-11-20T12:13:00.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.890+0000 7f255a59c700 1 -- 172.21.15.189:0/350992250 learned_addr learned my addr 172.21.15.189:0/350992250 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:13:00.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.890+0000 7f255ad9d700 1 --2- 172.21.15.189:0/350992250 >> v2:172.21.15.189:3300/0 conn(0x7f255c13a6e0 0x7f255c13ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:00.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.890+0000 7f2551d9b700 1 --2- 172.21.15.189:0/350992250 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f255c0ad3e0 0x7f255c066d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:00.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.890+0000 7f255ad9d700 1 -- 172.21.15.189:0/350992250 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f255c0ad3e0 msgr2=0x7f255c066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:00.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.890+0000 7f255ad9d700 1 --2- 172.21.15.189:0/350992250 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f255c0ad3e0 0x7f255c066d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:00.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.890+0000 7f255ad9d700 1 -- 172.21.15.189:0/350992250 >> v1:172.21.15.189:6789/0 conn(0x7f255c0ad000 legacy=0x7f255c13e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:00.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.890+0000 7f255ad9d700 1 -- 172.21.15.189:0/350992250 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f255c13dd40 con 0x7f255c13a6e0 2020-11-20T12:13:00.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.890+0000 7f255ad9d700 1 --2- 172.21.15.189:0/350992250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f255c13a6e0 0x7f255c13ad00 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f2554009cc0 tx=0x7f25540091d0).ready entity=mon.1 client_cookie=cff5f5d2f9e76dcc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:00.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.890+0000 7f2559d9b700 1 -- 172.21.15.189:0/350992250 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255400bac0 con 0x7f255c13a6e0 2020-11-20T12:13:00.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.890+0000 7f2559d9b700 1 -- 172.21.15.189:0/350992250 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f255400bc20 con 0x7f255c13a6e0 2020-11-20T12:13:00.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.890+0000 7f2559d9b700 1 -- 172.21.15.189:0/350992250 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2554011400 con 0x7f255c13a6e0 2020-11-20T12:13:00.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.890+0000 7f255b59e700 1 -- 172.21.15.189:0/350992250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f255c13a6e0 msgr2=0x7f255c13ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:00.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.890+0000 7f255b59e700 1 --2- 172.21.15.189:0/350992250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f255c13a6e0 0x7f255c13ad00 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f2554009cc0 tx=0x7f25540091d0).stop 2020-11-20T12:13:00.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.890+0000 7f255b59e700 1 -- 172.21.15.189:0/350992250 shutdown_connections 2020-11-20T12:13:00.954 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.890+0000 7f255b59e700 1 --2- 172.21.15.189:0/350992250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f255c13a6e0 0x7f255c13ad00 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:00.954 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f255b59e700 1 --2- 172.21.15.189:0/350992250 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f255c0ad3e0 0x7f255c066d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:00.954 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f255b59e700 1 -- 172.21.15.189:0/350992250 >> 172.21.15.189:0/350992250 conn(0x7f255c062080 msgr2=0x7f255c062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:00.955 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f255b59e700 1 -- 172.21.15.189:0/350992250 shutdown_connections 2020-11-20T12:13:00.955 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f255b59e700 1 -- 172.21.15.189:0/350992250 wait complete. 2020-11-20T12:13:00.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f255b59e700 1 Processor -- start 2020-11-20T12:13:00.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f255b59e700 1 -- start start 2020-11-20T12:13:00.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f255b59e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f255c0ad000 0x7f255c0af6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:00.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f255b59e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f255c0ad3e0 0x7f255c14cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:00.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f255b59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f255c0aa930 0x7f255c1370e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:00.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f255b59e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255c0a9400 con 0x7f255c0ad3e0 2020-11-20T12:13:00.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f255b59e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255c0a96a0 con 0x7f255c0aa930 2020-11-20T12:13:00.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f255b59e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f255c0a9940 con 0x7f255c0ad000 2020-11-20T12:13:00.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f255ad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f255c0aa930 0x7f255c1370e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:00.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f255ad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f255c0aa930 0x7f255c1370e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50588/0 (socket says 172.21.15.189:50588) 2020-11-20T12:13:00.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f255ad9d700 1 -- 172.21.15.189:0/1798627309 learned_addr learned my addr 172.21.15.189:0/1798627309 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:13:00.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f2551d9b700 1 --2- 172.21.15.189:0/1798627309 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f255c0ad3e0 0x7f255c14cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:00.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f255a59c700 1 --2- 172.21.15.189:0/1798627309 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f255c0ad000 0x7f255c0af6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:00.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f255ad9d700 1 -- 172.21.15.189:0/1798627309 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f255c0ad000 msgr2=0x7f255c0af6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:00.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f255ad9d700 1 --2- 172.21.15.189:0/1798627309 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f255c0ad000 0x7f255c0af6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:00.960 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f255ad9d700 1 -- 172.21.15.189:0/1798627309 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f255c0ad3e0 msgr2=0x7f255c14cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:00.960 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f255ad9d700 1 --2- 172.21.15.189:0/1798627309 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f255c0ad3e0 0x7f255c14cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:00.960 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f255ad9d700 1 -- 172.21.15.189:0/1798627309 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f255c14d190 con 0x7f255c0aa930 2020-11-20T12:13:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f255ad9d700 1 --2- 172.21.15.189:0/1798627309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f255c0aa930 0x7f255c1370e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f2554009c90 tx=0x7f2554016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f2552ffd700 1 -- 172.21.15.189:0/1798627309 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2554010800 con 0x7f255c0aa930 2020-11-20T12:13:00.961 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f2552ffd700 1 -- 172.21.15.189:0/1798627309 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2554010960 con 0x7f255c0aa930 2020-11-20T12:13:00.962 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f255b59e700 1 -- 172.21.15.189:0/1798627309 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f255c0b0310 con 0x7f255c0aa930 2020-11-20T12:13:00.962 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f2552ffd700 1 -- 172.21.15.189:0/1798627309 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2554010ce0 con 0x7f255c0aa930 2020-11-20T12:13:00.962 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.894+0000 7f255b59e700 1 -- 172.21.15.189:0/1798627309 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f255c0a9f80 con 0x7f255c0aa930 2020-11-20T12:13:00.963 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.898+0000 7f255b59e700 1 -- 172.21.15.189:0/1798627309 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f255c0a9b00 con 0x7f255c0aa930 2020-11-20T12:13:00.963 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.898+0000 7f2552ffd700 1 -- 172.21.15.189:0/1798627309 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f2554017070 con 0x7f255c0aa930 2020-11-20T12:13:00.963 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.898+0000 7f2552ffd700 1 --2- 172.21.15.189:0/1798627309 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f25380557c0 0x7f2538057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:00.963 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.898+0000 7f2552ffd700 1 -- 172.21.15.189:0/1798627309 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6589+0+0 (secure 0 0 0) 0x7f2554070e30 con 0x7f255c0aa930 2020-11-20T12:13:00.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.898+0000 7f255a59c700 1 --2- 172.21.15.189:0/1798627309 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f25380557c0 0x7f2538057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:00.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.898+0000 7f255a59c700 1 --2- 172.21.15.189:0/1798627309 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f25380557c0 0x7f2538057c80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f254c0055a0 tx=0x7f254c009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:00.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:00.902+0000 7f2552ffd700 1 -- 172.21.15.189:0/1798627309 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2554046ad0 con 0x7f255c0aa930 2020-11-20T12:13:01.137 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.126+0000 7f255b59e700 1 -- 172.21.15.189:0/1798627309 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f255c0b13b0 con 0x7f255c0aa930 2020-11-20T12:13:01.138 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.130+0000 7f2552ffd700 1 -- 172.21.15.189:0/1798627309 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f2554015030 con 0x7f255c0aa930 2020-11-20T12:13:01.138 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:13:01.139 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/1811549340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1811549340},{"type":"v1","addr":"172.21.15.47:6833","nonce":1811549340}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:40.691744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5203":{"gid":5203,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/2651986971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2651986971},{"type":"v1","addr":"172.21.15.189:6835","nonce":2651986971}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:13:01.141 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.130+0000 7f255b59e700 1 -- 172.21.15.189:0/1798627309 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f25380557c0 msgr2=0x7f2538057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:01.141 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.130+0000 7f255b59e700 1 --2- 172.21.15.189:0/1798627309 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f25380557c0 0x7f2538057c80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f254c0055a0 tx=0x7f254c009040).stop 2020-11-20T12:13:01.141 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.130+0000 7f255b59e700 1 -- 172.21.15.189:0/1798627309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f255c0aa930 msgr2=0x7f255c1370e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:01.141 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.130+0000 7f255b59e700 1 --2- 172.21.15.189:0/1798627309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f255c0aa930 0x7f255c1370e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f2554009c90 tx=0x7f2554016000).stop 2020-11-20T12:13:01.141 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.130+0000 7f255b59e700 1 -- 172.21.15.189:0/1798627309 shutdown_connections 2020-11-20T12:13:01.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.130+0000 7f255b59e700 1 --2- 172.21.15.189:0/1798627309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f255c0aa930 0x7f255c1370e0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:01.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.130+0000 7f255b59e700 1 --2- 172.21.15.189:0/1798627309 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f255c0ad000 0x7f255c0af6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:01.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.130+0000 7f255b59e700 1 --2- 172.21.15.189:0/1798627309 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f25380557c0 0x7f2538057c80 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:01.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.130+0000 7f255b59e700 1 --2- 172.21.15.189:0/1798627309 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f255c0ad3e0 0x7f255c14cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:01.143 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.130+0000 7f255b59e700 1 -- 172.21.15.189:0/1798627309 >> 172.21.15.189:0/1798627309 conn(0x7f255c062080 msgr2=0x7f255c05ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:01.143 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.130+0000 7f255b59e700 1 -- 172.21.15.189:0/1798627309 shutdown_connections 2020-11-20T12:13:01.143 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.130+0000 7f255b59e700 1 -- 172.21.15.189:0/1798627309 wait complete. 2020-11-20T12:13:01.143 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 36 2020-11-20T12:13:01.155 INFO:tasks.mds_thrash.fs.[None]:kill mds.a (rank=0) 2020-11-20T12:13:01.156 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T12:13:01.156 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:13:01.170 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T12:13:01.171 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:01.348 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.338+0000 7f75dcc7c700 1 Processor -- start 2020-11-20T12:13:01.348 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.338+0000 7f75dcc7c700 1 -- start start 2020-11-20T12:13:01.351 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.338+0000 7f75dcc7c700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75d80623c0 0x7f75d8138290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:01.351 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.338+0000 7f75dcc7c700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d813c680 con 0x7f75d8060bc0 2020-11-20T12:13:01.352 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.338+0000 7f75dcc7c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d8065ca0 con 0x7f75d8061040 2020-11-20T12:13:01.352 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.338+0000 7f75dcc7c700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d80687e0 con 0x7f75d80623c0 2020-11-20T12:13:01.352 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.338+0000 7f75d7fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f75d8061040 0x7f75d813bf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52220/0 (socket says 172.21.15.189:52220) 2020-11-20T12:13:01.352 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.338+0000 7f75d7fff700 1 -- 172.21.15.189:0/2804007618 learned_addr learned my addr 172.21.15.189:0/2804007618 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:13:01.353 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.338+0000 7f75d77fe700 1 --2- 172.21.15.189:0/2804007618 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75d80623c0 0x7f75d8138290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:01.353 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.338+0000 7f75d6ffd700 1 -- 172.21.15.189:0/2804007618 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1884944599 0 0) 0x7f75d8065ca0 con 0x7f75d8061040 2020-11-20T12:13:01.353 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.338+0000 7f75d6ffd700 1 -- 172.21.15.189:0/2804007618 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75b8003240 con 0x7f75d8061040 2020-11-20T12:13:01.353 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.338+0000 7f75d6ffd700 1 -- 172.21.15.189:0/2804007618 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506056868 0 0) 0x7f75d813c680 con 0x7f75d8060bc0 2020-11-20T12:13:01.353 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.338+0000 7f75d6ffd700 1 -- 172.21.15.189:0/2804007618 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75d8065ca0 con 0x7f75d8060bc0 2020-11-20T12:13:01.354 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.338+0000 7f75d6ffd700 1 -- 172.21.15.189:0/2804007618 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2038854088 0 0) 0x7f75b8003240 con 0x7f75d8061040 2020-11-20T12:13:01.354 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.338+0000 7f75d6ffd700 1 -- 172.21.15.189:0/2804007618 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75d80623c0 msgr2=0x7f75d8138290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:01.354 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.338+0000 7f75d6ffd700 1 --2- 172.21.15.189:0/2804007618 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75d80623c0 0x7f75d8138290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:01.354 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.338+0000 7f75d6ffd700 1 -- 172.21.15.189:0/2804007618 >> v1:172.21.15.47:6789/0 conn(0x7f75d8060bc0 legacy=0x7f75d813b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:01.354 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.338+0000 7f75d6ffd700 1 -- 172.21.15.189:0/2804007618 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d8138ba0 con 0x7f75d8061040 2020-11-20T12:13:01.355 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75d6ffd700 1 -- 172.21.15.189:0/2804007618 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f75d0001860 con 0x7f75d8061040 2020-11-20T12:13:01.355 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75d6ffd700 1 -- 172.21.15.189:0/2804007618 >> v1:172.21.15.189:6789/0 conn(0x7f75d8061040 legacy=0x7f75d813bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:01.355 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75d6ffd700 1 --2- 172.21.15.189:0/2804007618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75b80038c0 0x7f75b8003cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:01.355 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75d6ffd700 1 --2- 172.21.15.189:0/2804007618 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75b80044e0 0x7f75b8004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:01.355 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75d6ffd700 1 --2- 172.21.15.189:0/2804007618 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75d80623c0 0x7f75b8004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:01.356 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75d6ffd700 1 -- 172.21.15.189:0/2804007618 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b8005340 con 0x7f75b80044e0 2020-11-20T12:13:01.356 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75d6ffd700 1 -- 172.21.15.189:0/2804007618 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b800b730 con 0x7f75b80038c0 2020-11-20T12:13:01.356 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75d6ffd700 1 -- 172.21.15.189:0/2804007618 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75b800b870 con 0x7f75d80623c0 2020-11-20T12:13:01.356 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75d77fe700 1 --2- 172.21.15.189:0/2804007618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75b80038c0 0x7f75b8003cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:01.356 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75cffff700 1 --2- 172.21.15.189:0/2804007618 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75b80044e0 0x7f75b8004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:01.357 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75d7fff700 1 --2- 172.21.15.189:0/2804007618 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75d80623c0 0x7f75b8004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:01.357 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75d77fe700 1 -- 172.21.15.189:0/2804007618 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75d80623c0 msgr2=0x7f75b8004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:01.357 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75d77fe700 1 --2- 172.21.15.189:0/2804007618 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75d80623c0 0x7f75b8004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:01.357 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75d77fe700 1 -- 172.21.15.189:0/2804007618 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75b80044e0 msgr2=0x7f75b8004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:01.357 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75d77fe700 1 --2- 172.21.15.189:0/2804007618 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75b80044e0 0x7f75b8004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:01.358 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75d77fe700 1 -- 172.21.15.189:0/2804007618 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b8003240 con 0x7f75b80038c0 2020-11-20T12:13:01.358 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75d7fff700 1 --2- 172.21.15.189:0/2804007618 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75d80623c0 0x7f75b8004e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:13:01.358 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75d77fe700 1 --2- 172.21.15.189:0/2804007618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75b80038c0 0x7f75b8003cc0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f75c800bc30 tx=0x7f75c8006df0).ready entity=mon.1 client_cookie=ac11b7034dc82d8e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:01.358 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75d6ffd700 1 -- 172.21.15.189:0/2804007618 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c800b310 con 0x7f75b80038c0 2020-11-20T12:13:01.358 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75d6ffd700 1 -- 172.21.15.189:0/2804007618 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c800ea90 con 0x7f75b80038c0 2020-11-20T12:13:01.359 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75d6ffd700 1 -- 172.21.15.189:0/2804007618 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c800ee10 con 0x7f75b80038c0 2020-11-20T12:13:01.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75dcc7c700 1 -- 172.21.15.189:0/2804007618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75b80038c0 msgr2=0x7f75b8003cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:01.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75dcc7c700 1 --2- 172.21.15.189:0/2804007618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75b80038c0 0x7f75b8003cc0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f75c800bc30 tx=0x7f75c8006df0).stop 2020-11-20T12:13:01.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75d7fff700 1 -- 172.21.15.189:0/2804007618 reap_dead start 2020-11-20T12:13:01.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75dcc7c700 1 -- 172.21.15.189:0/2804007618 shutdown_connections 2020-11-20T12:13:01.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75dcc7c700 1 -- 172.21.15.189:0/2804007618 >> 172.21.15.189:0/2804007618 conn(0x7f75d8065fe0 msgr2=0x7f75d8061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:01.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75dcc7c700 1 -- 172.21.15.189:0/2804007618 shutdown_connections 2020-11-20T12:13:01.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75dcc7c700 1 -- 172.21.15.189:0/2804007618 wait complete. 2020-11-20T12:13:01.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75dcc7c700 1 Processor -- start 2020-11-20T12:13:01.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75dcc7c700 1 -- start start 2020-11-20T12:13:01.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.342+0000 7f75dcc7c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75d813b390 0x7f75d80acec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:01.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75dcc7c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75d80ae3a0 0x7f75d80ad3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:01.364 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75dcc7c700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75d814abb0 0x7f75d814b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:01.364 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75dcc7c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d80ad8a0 con 0x7f75d80ae3a0 2020-11-20T12:13:01.364 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75dcc7c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d80a9510 con 0x7f75d813b390 2020-11-20T12:13:01.364 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75dcc7c700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d80a97b0 con 0x7f75d814abb0 2020-11-20T12:13:01.364 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75d77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75d813b390 0x7f75d80acec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:01.365 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75d77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75d813b390 0x7f75d80acec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50606/0 (socket says 172.21.15.189:50606) 2020-11-20T12:13:01.365 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75d77fe700 1 -- 172.21.15.189:0/2271486111 learned_addr learned my addr 172.21.15.189:0/2271486111 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:13:01.365 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75cffff700 1 --2- 172.21.15.189:0/2271486111 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75d80ae3a0 0x7f75d80ad3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:01.365 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75d7fff700 1 --2- 172.21.15.189:0/2271486111 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75d814abb0 0x7f75d814b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:01.366 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75d77fe700 1 -- 172.21.15.189:0/2271486111 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75d814abb0 msgr2=0x7f75d814b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:01.366 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75d77fe700 1 --2- 172.21.15.189:0/2271486111 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75d814abb0 0x7f75d814b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:01.366 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75d77fe700 1 -- 172.21.15.189:0/2271486111 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75d80ae3a0 msgr2=0x7f75d80ad3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:01.366 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75d77fe700 1 --2- 172.21.15.189:0/2271486111 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75d80ae3a0 0x7f75d80ad3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:01.366 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75d77fe700 1 -- 172.21.15.189:0/2271486111 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d80a9a50 con 0x7f75d813b390 2020-11-20T12:13:01.367 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75cffff700 1 --2- 172.21.15.189:0/2271486111 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75d80ae3a0 0x7f75d80ad3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:13:01.367 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75d77fe700 1 --2- 172.21.15.189:0/2271486111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75d813b390 0x7f75d80acec0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f75c8013000 tx=0x7f75c8019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:01.367 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75cf7fe700 1 -- 172.21.15.189:0/2271486111 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c800b950 con 0x7f75d813b390 2020-11-20T12:13:01.367 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75cf7fe700 1 -- 172.21.15.189:0/2271486111 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c8022070 con 0x7f75d813b390 2020-11-20T12:13:01.367 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75dcc7c700 1 -- 172.21.15.189:0/2271486111 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75d80a9cf0 con 0x7f75d813b390 2020-11-20T12:13:01.368 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75cf7fe700 1 -- 172.21.15.189:0/2271486111 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c800eed0 con 0x7f75d813b390 2020-11-20T12:13:01.368 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75dcc7c700 1 -- 172.21.15.189:0/2271486111 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d80ae0b0 con 0x7f75d813b390 2020-11-20T12:13:01.368 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75cf7fe700 1 -- 172.21.15.189:0/2271486111 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f75c802e070 con 0x7f75d813b390 2020-11-20T12:13:01.368 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75dcc7c700 1 -- 172.21.15.189:0/2271486111 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75d80663c0 con 0x7f75d813b390 2020-11-20T12:13:01.368 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75cf7fe700 1 --2- 172.21.15.189:0/2271486111 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f75ac0557c0 0x7f75ac057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:01.368 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75cf7fe700 1 -- 172.21.15.189:0/2271486111 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6589+0+0 (secure 0 0 0) 0x7f75c8070dc0 con 0x7f75d813b390 2020-11-20T12:13:01.369 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75cffff700 1 --2- 172.21.15.189:0/2271486111 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f75ac0557c0 0x7f75ac057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:01.369 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.346+0000 7f75cffff700 1 --2- 172.21.15.189:0/2271486111 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f75ac0557c0 0x7f75ac057c80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f75d80a92b0 tx=0x7f75c0009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:01.369 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.350+0000 7f75cf7fe700 1 -- 172.21.15.189:0/2271486111 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f75c8046be0 con 0x7f75d813b390 2020-11-20T12:13:01.562 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.554+0000 7f75dcc7c700 1 -- 172.21.15.189:0/2271486111 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75d80446d0 con 0x7f75d813b390 2020-11-20T12:13:01.563 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.554+0000 7f75cf7fe700 1 -- 172.21.15.189:0/2271486111 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f75c804a100 con 0x7f75d813b390 2020-11-20T12:13:01.563 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:13:01.563 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/1811549340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1811549340},{"type":"v1","addr":"172.21.15.47:6833","nonce":1811549340}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:40.691744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5203":{"gid":5203,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/2651986971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2651986971},{"type":"v1","addr":"172.21.15.189:6835","nonce":2651986971}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:13:01.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.554+0000 7f75dcc7c700 1 -- 172.21.15.189:0/2271486111 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f75ac0557c0 msgr2=0x7f75ac057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:01.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.554+0000 7f75dcc7c700 1 --2- 172.21.15.189:0/2271486111 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f75ac0557c0 0x7f75ac057c80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f75d80a92b0 tx=0x7f75c0009040).stop 2020-11-20T12:13:01.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.554+0000 7f75dcc7c700 1 -- 172.21.15.189:0/2271486111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75d813b390 msgr2=0x7f75d80acec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:01.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.554+0000 7f75dcc7c700 1 --2- 172.21.15.189:0/2271486111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75d813b390 0x7f75d80acec0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f75c8013000 tx=0x7f75c8019000).stop 2020-11-20T12:13:01.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.554+0000 7f75dcc7c700 1 -- 172.21.15.189:0/2271486111 shutdown_connections 2020-11-20T12:13:01.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.554+0000 7f75dcc7c700 1 --2- 172.21.15.189:0/2271486111 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75d814abb0 0x7f75d814b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:01.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.554+0000 7f75dcc7c700 1 --2- 172.21.15.189:0/2271486111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75d813b390 0x7f75d80acec0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:01.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.554+0000 7f75dcc7c700 1 --2- 172.21.15.189:0/2271486111 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f75ac0557c0 0x7f75ac057c80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:01.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.554+0000 7f75dcc7c700 1 --2- 172.21.15.189:0/2271486111 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75d80ae3a0 0x7f75d80ad3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:01.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.554+0000 7f75dcc7c700 1 -- 172.21.15.189:0/2271486111 >> 172.21.15.189:0/2271486111 conn(0x7f75d8065fe0 msgr2=0x7f75d813d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:01.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.554+0000 7f75dcc7c700 1 -- 172.21.15.189:0/2271486111 shutdown_connections 2020-11-20T12:13:01.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:01.554+0000 7f75dcc7c700 1 -- 172.21.15.189:0/2271486111 wait complete. 2020-11-20T12:13:01.570 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 36 2020-11-20T12:13:01.578 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:13:03.579 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:03.713 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58e59f8700 1 Processor -- start 2020-11-20T12:13:03.713 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58e59f8700 1 -- start start 2020-11-20T12:13:03.715 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58e59f8700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f58e00ad000 0x7f58e013e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:03.715 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58e59f8700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e0067730 con 0x7f58e013a6e0 2020-11-20T12:13:03.715 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58e59f8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e013dd40 con 0x7f58e00ad3e0 2020-11-20T12:13:03.716 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58e59f8700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e0138190 con 0x7f58e00ad000 2020-11-20T12:13:03.716 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58dffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f58e00ad3e0 0x7f58e0066d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52242/0 (socket says 172.21.15.189:52242) 2020-11-20T12:13:03.716 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58dffff700 1 -- 172.21.15.189:0/1150804343 learned_addr learned my addr 172.21.15.189:0/1150804343 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:13:03.716 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58e51f7700 1 --1- 172.21.15.189:0/1150804343 >> v1:172.21.15.47:6789/0 conn(0x7f58e013a6e0 0x7f58e013ad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.189:46910/0 (socket says 172.21.15.189:46910) 2020-11-20T12:13:03.716 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58e49f6700 1 --2- 172.21.15.189:0/1150804343 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f58e00ad000 0x7f58e013e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:03.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58df7fe700 1 -- 172.21.15.189:0/1150804343 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4132002898 0 0) 0x7f58e013dd40 con 0x7f58e00ad3e0 2020-11-20T12:13:03.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58df7fe700 1 -- 172.21.15.189:0/1150804343 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58cc003240 con 0x7f58e00ad3e0 2020-11-20T12:13:03.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58df7fe700 1 -- 172.21.15.189:0/1150804343 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820729180 0 0) 0x7f58e0067730 con 0x7f58e013a6e0 2020-11-20T12:13:03.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58df7fe700 1 -- 172.21.15.189:0/1150804343 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58e013dd40 con 0x7f58e013a6e0 2020-11-20T12:13:03.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58df7fe700 1 -- 172.21.15.189:0/1150804343 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3792965632 0 0) 0x7f58cc003240 con 0x7f58e00ad3e0 2020-11-20T12:13:03.718 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58e49f6700 1 -- 172.21.15.189:0/1150804343 >> v1:172.21.15.189:6789/0 conn(0x7f58e00ad3e0 legacy=0x7f58e0066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:03.718 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58e49f6700 1 -- 172.21.15.189:0/1150804343 >> v1:172.21.15.47:6789/0 conn(0x7f58e013a6e0 legacy=0x7f58e013ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:03.718 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58e49f6700 1 -- 172.21.15.189:0/1150804343 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e013dfe0 con 0x7f58e00ad000 2020-11-20T12:13:03.721 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58e49f6700 1 --2- 172.21.15.189:0/1150804343 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f58e00ad000 0x7f58e013e480 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f58d400b950 tx=0x7f58d4006df0).ready entity=mon.2 client_cookie=c42926fe42dc991f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:03.721 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58df7fe700 1 -- 172.21.15.189:0/1150804343 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d00014e0 con 0x7f58e00ad000 2020-11-20T12:13:03.721 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58df7fe700 1 -- 172.21.15.189:0/1150804343 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d4011400 con 0x7f58e00ad000 2020-11-20T12:13:03.722 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58df7fe700 1 -- 172.21.15.189:0/1150804343 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d4011780 con 0x7f58e00ad000 2020-11-20T12:13:03.722 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58e59f8700 1 -- 172.21.15.189:0/1150804343 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f58e00ad000 msgr2=0x7f58e013e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:03.722 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58e59f8700 1 --2- 172.21.15.189:0/1150804343 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f58e00ad000 0x7f58e013e480 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f58d400b950 tx=0x7f58d4006df0).stop 2020-11-20T12:13:03.722 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58e59f8700 1 -- 172.21.15.189:0/1150804343 shutdown_connections 2020-11-20T12:13:03.722 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58e59f8700 1 --2- 172.21.15.189:0/1150804343 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f58e00ad000 0x7f58e013e480 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:03.723 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58e59f8700 1 -- 172.21.15.189:0/1150804343 >> 172.21.15.189:0/1150804343 conn(0x7f58e0062080 msgr2=0x7f58e0062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:03.723 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58e59f8700 1 -- 172.21.15.189:0/1150804343 shutdown_connections 2020-11-20T12:13:03.723 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.705+0000 7f58e59f8700 1 -- 172.21.15.189:0/1150804343 wait complete. 2020-11-20T12:13:03.723 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.709+0000 7f58e59f8700 1 Processor -- start 2020-11-20T12:13:03.724 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.709+0000 7f58e59f8700 1 -- start start 2020-11-20T12:13:03.724 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.709+0000 7f58e59f8700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58e00ad000 0x7f58e00a9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:03.724 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.709+0000 7f58e59f8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58e00ad3e0 0x7f58e00a7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:03.724 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.709+0000 7f58e59f8700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f58e01498c0 0x7f58e0149d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:03.724 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.709+0000 7f58e59f8700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e00b04f0 con 0x7f58e00ad000 2020-11-20T12:13:03.725 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.709+0000 7f58e59f8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e00afdb0 con 0x7f58e00ad3e0 2020-11-20T12:13:03.725 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.709+0000 7f58e59f8700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e00b0050 con 0x7f58e01498c0 2020-11-20T12:13:03.725 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.709+0000 7f58dffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58e00ad3e0 0x7f58e00a7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:03.725 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.709+0000 7f58dffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58e00ad3e0 0x7f58e00a7090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50630/0 (socket says 172.21.15.189:50630) 2020-11-20T12:13:03.726 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.709+0000 7f58dffff700 1 -- 172.21.15.189:0/3956366256 learned_addr learned my addr 172.21.15.189:0/3956366256 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:13:03.726 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.709+0000 7f58e51f7700 1 --2- 172.21.15.189:0/3956366256 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f58e01498c0 0x7f58e0149d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:03.726 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.709+0000 7f58e49f6700 1 --2- 172.21.15.189:0/3956366256 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58e00ad000 0x7f58e00a9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:03.726 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.709+0000 7f58dffff700 1 -- 172.21.15.189:0/3956366256 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f58e01498c0 msgr2=0x7f58e0149d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:03.726 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.709+0000 7f58dffff700 1 --2- 172.21.15.189:0/3956366256 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f58e01498c0 0x7f58e0149d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:03.727 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.709+0000 7f58dffff700 1 -- 172.21.15.189:0/3956366256 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58e00ad000 msgr2=0x7f58e00a9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:03.727 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.709+0000 7f58dffff700 1 --2- 172.21.15.189:0/3956366256 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58e00ad000 0x7f58e00a9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:03.727 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.709+0000 7f58dffff700 1 -- 172.21.15.189:0/3956366256 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e00a7af0 con 0x7f58e00ad3e0 2020-11-20T12:13:03.727 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.709+0000 7f58dffff700 1 --2- 172.21.15.189:0/3956366256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58e00ad3e0 0x7f58e00a7090 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f58e00ab420 tx=0x7f58d00015a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:03.727 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.709+0000 7f58dcff9700 1 -- 172.21.15.189:0/3956366256 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d0007320 con 0x7f58e00ad3e0 2020-11-20T12:13:03.728 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.709+0000 7f58dcff9700 1 -- 172.21.15.189:0/3956366256 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d0007480 con 0x7f58e00ad3e0 2020-11-20T12:13:03.728 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.709+0000 7f58dcff9700 1 -- 172.21.15.189:0/3956366256 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d00057c0 con 0x7f58e00ad3e0 2020-11-20T12:13:03.728 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.709+0000 7f58e59f8700 1 -- 172.21.15.189:0/3956366256 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e00a7580 con 0x7f58e00ad3e0 2020-11-20T12:13:03.728 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.709+0000 7f58e59f8700 1 -- 172.21.15.189:0/3956366256 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e00a61e0 con 0x7f58e00ad3e0 2020-11-20T12:13:03.729 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.713+0000 7f58e59f8700 1 -- 172.21.15.189:0/3956366256 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e00a7ce0 con 0x7f58e00ad3e0 2020-11-20T12:13:03.729 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.713+0000 7f58dcff9700 1 -- 172.21.15.189:0/3956366256 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f58d0010030 con 0x7f58e00ad3e0 2020-11-20T12:13:03.730 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.713+0000 7f58dcff9700 1 --2- 172.21.15.189:0/3956366256 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f58b80557c0 0x7f58b8057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:03.730 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.713+0000 7f58dcff9700 1 -- 172.21.15.189:0/3956366256 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6589+0+0 (secure 0 0 0) 0x7f58d006e610 con 0x7f58e00ad3e0 2020-11-20T12:13:03.730 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.713+0000 7f58e49f6700 1 --2- 172.21.15.189:0/3956366256 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f58b80557c0 0x7f58b8057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:03.730 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.713+0000 7f58e49f6700 1 --2- 172.21.15.189:0/3956366256 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f58b80557c0 0x7f58b8057c80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f58d400b150 tx=0x7f58d4015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:03.731 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.725+0000 7f58dcff9700 1 -- 172.21.15.189:0/3956366256 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f58d00442b0 con 0x7f58e00ad3e0 2020-11-20T12:13:03.938 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.929+0000 7f58e59f8700 1 -- 172.21.15.189:0/3956366256 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58e00b0210 con 0x7f58e00ad3e0 2020-11-20T12:13:03.939 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.929+0000 7f58dcff9700 1 -- 172.21.15.189:0/3956366256 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f58d00478d0 con 0x7f58e00ad3e0 2020-11-20T12:13:03.939 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:13:03.940 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/1811549340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1811549340},{"type":"v1","addr":"172.21.15.47:6833","nonce":1811549340}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:40.691744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5203":{"gid":5203,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/2651986971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2651986971},{"type":"v1","addr":"172.21.15.189:6835","nonce":2651986971}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:13:03.941 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.929+0000 7f58e59f8700 1 -- 172.21.15.189:0/3956366256 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f58b80557c0 msgr2=0x7f58b8057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:03.941 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.929+0000 7f58e59f8700 1 --2- 172.21.15.189:0/3956366256 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f58b80557c0 0x7f58b8057c80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f58d400b150 tx=0x7f58d4015000).stop 2020-11-20T12:13:03.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.929+0000 7f58e59f8700 1 -- 172.21.15.189:0/3956366256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58e00ad3e0 msgr2=0x7f58e00a7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:03.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.929+0000 7f58e59f8700 1 --2- 172.21.15.189:0/3956366256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58e00ad3e0 0x7f58e00a7090 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f58e00ab420 tx=0x7f58d00015a0).stop 2020-11-20T12:13:03.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.929+0000 7f58e59f8700 1 -- 172.21.15.189:0/3956366256 shutdown_connections 2020-11-20T12:13:03.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.929+0000 7f58e59f8700 1 --2- 172.21.15.189:0/3956366256 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f58b80557c0 0x7f58b8057c80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:03.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.929+0000 7f58e59f8700 1 --2- 172.21.15.189:0/3956366256 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f58e00ad000 0x7f58e00a9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:03.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.929+0000 7f58e59f8700 1 --2- 172.21.15.189:0/3956366256 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f58e01498c0 0x7f58e0149d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:03.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.929+0000 7f58e59f8700 1 --2- 172.21.15.189:0/3956366256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58e00ad3e0 0x7f58e00a7090 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:03.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.929+0000 7f58e59f8700 1 -- 172.21.15.189:0/3956366256 >> 172.21.15.189:0/3956366256 conn(0x7f58e0062080 msgr2=0x7f58e0139f80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:03.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.929+0000 7f58e59f8700 1 -- 172.21.15.189:0/3956366256 shutdown_connections 2020-11-20T12:13:03.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:03.929+0000 7f58e59f8700 1 -- 172.21.15.189:0/3956366256 wait complete. 2020-11-20T12:13:03.944 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 36 2020-11-20T12:13:03.954 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:13:05.957 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:06.099 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.093+0000 7fb0f9503700 1 Processor -- start 2020-11-20T12:13:06.099 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.093+0000 7fb0f9503700 1 -- start start 2020-11-20T12:13:06.101 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.093+0000 7fb0f9503700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb0f40a74d0 0x7fb0f413be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:06.101 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.093+0000 7fb0f9503700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f40a9680 con 0x7fb0f40aefb0 2020-11-20T12:13:06.101 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.093+0000 7fb0f9503700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f40aae50 con 0x7fb0f40aebd0 2020-11-20T12:13:06.101 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.093+0000 7fb0f9503700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f40aa1f0 con 0x7fb0f40a74d0 2020-11-20T12:13:06.101 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.093+0000 7fb0f3fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb0f40aebd0 0x7fb0f40aa620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52260/0 (socket says 172.21.15.189:52260) 2020-11-20T12:13:06.102 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.093+0000 7fb0f3fff700 1 -- 172.21.15.189:0/2288407753 learned_addr learned my addr 172.21.15.189:0/2288407753 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:13:06.102 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.093+0000 7fb0f8d02700 1 --2- 172.21.15.189:0/2288407753 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb0f40a74d0 0x7fb0f413be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:06.102 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.093+0000 7fb0f2ffd700 1 -- 172.21.15.189:0/2288407753 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 247032780 0 0) 0x7fb0f40aae50 con 0x7fb0f40aebd0 2020-11-20T12:13:06.102 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.093+0000 7fb0f2ffd700 1 -- 172.21.15.189:0/2288407753 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e0003240 con 0x7fb0f40aebd0 2020-11-20T12:13:06.103 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.093+0000 7fb0f2ffd700 1 -- 172.21.15.189:0/2288407753 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1078056730 0 0) 0x7fb0f40a9680 con 0x7fb0f40aefb0 2020-11-20T12:13:06.103 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.093+0000 7fb0f2ffd700 1 -- 172.21.15.189:0/2288407753 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0f40aae50 con 0x7fb0f40aefb0 2020-11-20T12:13:06.103 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.093+0000 7fb0f8d02700 1 -- 172.21.15.189:0/2288407753 >> v1:172.21.15.189:6789/0 conn(0x7fb0f40aebd0 legacy=0x7fb0f40aa620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:06.103 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.093+0000 7fb0f8d02700 1 -- 172.21.15.189:0/2288407753 >> v1:172.21.15.47:6789/0 conn(0x7fb0f40aefb0 legacy=0x7fb0f40a9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:06.103 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.093+0000 7fb0f8d02700 1 -- 172.21.15.189:0/2288407753 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f413ba10 con 0x7fb0f40a74d0 2020-11-20T12:13:06.106 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.093+0000 7fb0f8d02700 1 --2- 172.21.15.189:0/2288407753 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb0f40a74d0 0x7fb0f413be70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fb0e400b9d0 tx=0x7fb0e4006df0).ready entity=mon.2 client_cookie=42d3113252d0d998 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:06.106 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.093+0000 7fb0f2ffd700 1 -- 172.21.15.189:0/2288407753 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4009680 con 0x7fb0f40a74d0 2020-11-20T12:13:06.106 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.093+0000 7fb0f2ffd700 1 -- 172.21.15.189:0/2288407753 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e40097e0 con 0x7fb0f40a74d0 2020-11-20T12:13:06.107 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.093+0000 7fb0f2ffd700 1 -- 172.21.15.189:0/2288407753 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4009b60 con 0x7fb0f40a74d0 2020-11-20T12:13:06.107 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.093+0000 7fb0f9503700 1 -- 172.21.15.189:0/2288407753 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb0f40a74d0 msgr2=0x7fb0f413be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:06.107 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.093+0000 7fb0f9503700 1 --2- 172.21.15.189:0/2288407753 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb0f40a74d0 0x7fb0f413be70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fb0e400b9d0 tx=0x7fb0e4006df0).stop 2020-11-20T12:13:06.107 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.093+0000 7fb0f9503700 1 -- 172.21.15.189:0/2288407753 shutdown_connections 2020-11-20T12:13:06.107 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.093+0000 7fb0f9503700 1 --2- 172.21.15.189:0/2288407753 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb0f40a74d0 0x7fb0f413be70 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:06.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.093+0000 7fb0f9503700 1 -- 172.21.15.189:0/2288407753 >> 172.21.15.189:0/2288407753 conn(0x7fb0f4062080 msgr2=0x7fb0f4062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:06.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.097+0000 7fb0f9503700 1 -- 172.21.15.189:0/2288407753 shutdown_connections 2020-11-20T12:13:06.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.097+0000 7fb0f9503700 1 -- 172.21.15.189:0/2288407753 wait complete. 2020-11-20T12:13:06.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.097+0000 7fb0f9503700 1 Processor -- start 2020-11-20T12:13:06.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.097+0000 7fb0f9503700 1 -- start start 2020-11-20T12:13:06.109 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.097+0000 7fb0f9503700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0f40a74d0 0x7fb0f40aca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:06.109 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.097+0000 7fb0f9503700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb0f40aebd0 0x7fb0f40633a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:06.109 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.097+0000 7fb0f9503700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0f40aefb0 0x7fb0f414cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:06.109 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.097+0000 7fb0f9503700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f40a1ab0 con 0x7fb0f40a74d0 2020-11-20T12:13:06.110 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.097+0000 7fb0f9503700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f40a1d50 con 0x7fb0f40aefb0 2020-11-20T12:13:06.110 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.097+0000 7fb0f9503700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f40a1ff0 con 0x7fb0f40aebd0 2020-11-20T12:13:06.110 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.097+0000 7fb0f8d02700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0f40aefb0 0x7fb0f414cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:06.110 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.097+0000 7fb0f8d02700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0f40aefb0 0x7fb0f414cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50644/0 (socket says 172.21.15.189:50644) 2020-11-20T12:13:06.110 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.097+0000 7fb0f8d02700 1 -- 172.21.15.189:0/397493181 learned_addr learned my addr 172.21.15.189:0/397493181 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:13:06.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.097+0000 7fb0f37fe700 1 --2- 172.21.15.189:0/397493181 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb0f40aebd0 0x7fb0f40633a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:06.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.097+0000 7fb0f3fff700 1 --2- 172.21.15.189:0/397493181 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0f40a74d0 0x7fb0f40aca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:06.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.097+0000 7fb0f8d02700 1 -- 172.21.15.189:0/397493181 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb0f40aebd0 msgr2=0x7fb0f40633a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:06.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.097+0000 7fb0f8d02700 1 --2- 172.21.15.189:0/397493181 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb0f40aebd0 0x7fb0f40633a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:06.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.097+0000 7fb0f8d02700 1 -- 172.21.15.189:0/397493181 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0f40a74d0 msgr2=0x7fb0f40aca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:06.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.097+0000 7fb0f8d02700 1 --2- 172.21.15.189:0/397493181 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0f40a74d0 0x7fb0f40aca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:06.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.097+0000 7fb0f8d02700 1 -- 172.21.15.189:0/397493181 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f40a0fd0 con 0x7fb0f40aefb0 2020-11-20T12:13:06.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.097+0000 7fb0f8d02700 1 --2- 172.21.15.189:0/397493181 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0f40aefb0 0x7fb0f414cdf0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7fb0e4010000 tx=0x7fb0e4015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:06.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.097+0000 7fb0d3fff700 1 -- 172.21.15.189:0/397493181 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4006b20 con 0x7fb0f40aefb0 2020-11-20T12:13:06.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.097+0000 7fb0f9503700 1 -- 172.21.15.189:0/397493181 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0f40acf60 con 0x7fb0f40aefb0 2020-11-20T12:13:06.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.097+0000 7fb0d3fff700 1 -- 172.21.15.189:0/397493181 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e401d450 con 0x7fb0f40aefb0 2020-11-20T12:13:06.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.097+0000 7fb0d3fff700 1 -- 172.21.15.189:0/397493181 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e401d830 con 0x7fb0f40aefb0 2020-11-20T12:13:06.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.097+0000 7fb0f9503700 1 -- 172.21.15.189:0/397493181 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0f40a26b0 con 0x7fb0f40aefb0 2020-11-20T12:13:06.114 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.101+0000 7fb0f9503700 1 -- 172.21.15.189:0/397493181 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0f40a21b0 con 0x7fb0f40aefb0 2020-11-20T12:13:06.115 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.101+0000 7fb0d3fff700 1 -- 172.21.15.189:0/397493181 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fb0e400f070 con 0x7fb0f40aefb0 2020-11-20T12:13:06.115 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.101+0000 7fb0d3fff700 1 --2- 172.21.15.189:0/397493181 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb0cc0557c0 0x7fb0cc057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:06.115 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.101+0000 7fb0d3fff700 1 -- 172.21.15.189:0/397493181 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6589+0+0 (secure 0 0 0) 0x7fb0e406fb30 con 0x7fb0f40aefb0 2020-11-20T12:13:06.115 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.101+0000 7fb0f3fff700 1 --2- 172.21.15.189:0/397493181 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb0cc0557c0 0x7fb0cc057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:06.116 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.101+0000 7fb0f3fff700 1 --2- 172.21.15.189:0/397493181 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb0cc0557c0 0x7fb0cc057c80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fb0e80055a0 tx=0x7fb0e8009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:06.116 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.105+0000 7fb0d3fff700 1 -- 172.21.15.189:0/397493181 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb0e4045880 con 0x7fb0f40aefb0 2020-11-20T12:13:06.313 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.305+0000 7fb0f9503700 1 -- 172.21.15.189:0/397493181 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0f40ab0f0 con 0x7fb0f40aefb0 2020-11-20T12:13:06.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.309+0000 7fb0d3fff700 1 -- 172.21.15.189:0/397493181 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fb0e4022070 con 0x7fb0f40aefb0 2020-11-20T12:13:06.314 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:13:06.314 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/1811549340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1811549340},{"type":"v1","addr":"172.21.15.47:6833","nonce":1811549340}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:40.691744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5203":{"gid":5203,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/2651986971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2651986971},{"type":"v1","addr":"172.21.15.189:6835","nonce":2651986971}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:13:06.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.309+0000 7fb0f9503700 1 -- 172.21.15.189:0/397493181 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb0cc0557c0 msgr2=0x7fb0cc057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:06.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.309+0000 7fb0f9503700 1 --2- 172.21.15.189:0/397493181 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb0cc0557c0 0x7fb0cc057c80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fb0e80055a0 tx=0x7fb0e8009040).stop 2020-11-20T12:13:06.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.309+0000 7fb0f9503700 1 -- 172.21.15.189:0/397493181 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0f40aefb0 msgr2=0x7fb0f414cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:06.317 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.309+0000 7fb0f9503700 1 --2- 172.21.15.189:0/397493181 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0f40aefb0 0x7fb0f414cdf0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7fb0e4010000 tx=0x7fb0e4015000).stop 2020-11-20T12:13:06.317 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.309+0000 7fb0f9503700 1 -- 172.21.15.189:0/397493181 shutdown_connections 2020-11-20T12:13:06.317 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.309+0000 7fb0f9503700 1 --2- 172.21.15.189:0/397493181 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb0cc0557c0 0x7fb0cc057c80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:06.317 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.309+0000 7fb0f9503700 1 --2- 172.21.15.189:0/397493181 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb0f40a74d0 0x7fb0f40aca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:06.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.309+0000 7fb0f9503700 1 --2- 172.21.15.189:0/397493181 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0f40aefb0 0x7fb0f414cdf0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:06.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.309+0000 7fb0f9503700 1 --2- 172.21.15.189:0/397493181 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb0f40aebd0 0x7fb0f40633a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:06.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.309+0000 7fb0f9503700 1 -- 172.21.15.189:0/397493181 >> 172.21.15.189:0/397493181 conn(0x7fb0f4062080 msgr2=0x7fb0f405ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:06.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.309+0000 7fb0f9503700 1 -- 172.21.15.189:0/397493181 shutdown_connections 2020-11-20T12:13:06.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:06.309+0000 7fb0f9503700 1 -- 172.21.15.189:0/397493181 wait complete. 2020-11-20T12:13:06.319 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 36 2020-11-20T12:13:06.329 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:13:08.329 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:08.467 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.461+0000 7f19c787b700 1 Processor -- start 2020-11-20T12:13:08.467 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.461+0000 7f19c787b700 1 -- start start 2020-11-20T12:13:08.468 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.461+0000 7f19c787b700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f19c00a9a40 0x7f19c013e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:08.469 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.461+0000 7f19c787b700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c0060f90 con 0x7f19c00a9660 2020-11-20T12:13:08.469 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.461+0000 7f19c787b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c013dc10 con 0x7f19c00a7760 2020-11-20T12:13:08.469 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.461+0000 7f19c787b700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c013deb0 con 0x7f19c00a9a40 2020-11-20T12:13:08.469 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.461+0000 7f19c5e18700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f19c00a7760 0x7f19c0060880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52276/0 (socket says 172.21.15.189:52276) 2020-11-20T12:13:08.469 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.461+0000 7f19c5e18700 1 -- 172.21.15.189:0/530769548 learned_addr learned my addr 172.21.15.189:0/530769548 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:13:08.470 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.461+0000 7f19c4e16700 1 --2- 172.21.15.189:0/530769548 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f19c00a9a40 0x7f19c013e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:08.470 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.461+0000 7f19bffff700 1 -- 172.21.15.189:0/530769548 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164762136 0 0) 0x7f19c013dc10 con 0x7f19c00a7760 2020-11-20T12:13:08.470 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.461+0000 7f19bffff700 1 -- 172.21.15.189:0/530769548 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19ac003240 con 0x7f19c00a7760 2020-11-20T12:13:08.472 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.461+0000 7f19bffff700 1 -- 172.21.15.189:0/530769548 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528696544 0 0) 0x7f19c0060f90 con 0x7f19c00a9660 2020-11-20T12:13:08.472 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.461+0000 7f19bffff700 1 -- 172.21.15.189:0/530769548 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19c013dc10 con 0x7f19c00a9660 2020-11-20T12:13:08.472 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.461+0000 7f19bffff700 1 -- 172.21.15.189:0/530769548 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1826206818 0 0) 0x7f19ac003240 con 0x7f19c00a7760 2020-11-20T12:13:08.472 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.461+0000 7f19c4e16700 1 -- 172.21.15.189:0/530769548 >> v1:172.21.15.189:6789/0 conn(0x7f19c00a7760 legacy=0x7f19c0060880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:08.473 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.461+0000 7f19c4e16700 1 -- 172.21.15.189:0/530769548 >> v1:172.21.15.47:6789/0 conn(0x7f19c00a9660 legacy=0x7f19c0060170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:08.473 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.461+0000 7f19c4e16700 1 -- 172.21.15.189:0/530769548 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c013e150 con 0x7f19c00a9a40 2020-11-20T12:13:08.473 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.461+0000 7f19c4e16700 1 --2- 172.21.15.189:0/530769548 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f19c00a9a40 0x7f19c013e5f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f19b400b950 tx=0x7f19b4006df0).ready entity=mon.2 client_cookie=7cac40002e8f1a5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:08.473 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.461+0000 7f19bffff700 1 -- 172.21.15.189:0/530769548 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b8001860 con 0x7f19c00a9a40 2020-11-20T12:13:08.474 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.461+0000 7f19bffff700 1 -- 172.21.15.189:0/530769548 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b4011400 con 0x7f19c00a9a40 2020-11-20T12:13:08.474 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.461+0000 7f19bffff700 1 -- 172.21.15.189:0/530769548 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b4011780 con 0x7f19c00a9a40 2020-11-20T12:13:08.474 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.465+0000 7f19c787b700 1 -- 172.21.15.189:0/530769548 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f19c00a9a40 msgr2=0x7f19c013e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:08.474 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.465+0000 7f19c787b700 1 --2- 172.21.15.189:0/530769548 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f19c00a9a40 0x7f19c013e5f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f19b400b950 tx=0x7f19b4006df0).stop 2020-11-20T12:13:08.474 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.465+0000 7f19c787b700 1 -- 172.21.15.189:0/530769548 shutdown_connections 2020-11-20T12:13:08.475 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.465+0000 7f19c787b700 1 --2- 172.21.15.189:0/530769548 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f19c00a9a40 0x7f19c013e5f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:08.475 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.465+0000 7f19c787b700 1 -- 172.21.15.189:0/530769548 >> 172.21.15.189:0/530769548 conn(0x7f19c0061f60 msgr2=0x7f19c0062370 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:08.475 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.465+0000 7f19c787b700 1 -- 172.21.15.189:0/530769548 shutdown_connections 2020-11-20T12:13:08.475 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.465+0000 7f19c787b700 1 -- 172.21.15.189:0/530769548 wait complete. 2020-11-20T12:13:08.477 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.465+0000 7f19c787b700 1 Processor -- start 2020-11-20T12:13:08.477 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.465+0000 7f19c787b700 1 -- start start 2020-11-20T12:13:08.478 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.465+0000 7f19c787b700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f19c00a7760 0x7f19c013a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:08.478 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.465+0000 7f19c787b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19c00a9660 0x7f19c00a67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:08.478 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.465+0000 7f19c787b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f19c00a9a40 0x7f19c00a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:08.478 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.465+0000 7f19c787b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c0149610 con 0x7f19c00a9a40 2020-11-20T12:13:08.479 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.465+0000 7f19c787b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c00a6ca0 con 0x7f19c00a9660 2020-11-20T12:13:08.479 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.465+0000 7f19c787b700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c00a19b0 con 0x7f19c00a7760 2020-11-20T12:13:08.479 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.469+0000 7f19c4e16700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19c00a9660 0x7f19c00a67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:08.479 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.469+0000 7f19c5e18700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f19c00a9a40 0x7f19c00a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:08.479 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.469+0000 7f19c4e16700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19c00a9660 0x7f19c00a67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50662/0 (socket says 172.21.15.189:50662) 2020-11-20T12:13:08.480 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.469+0000 7f19c4e16700 1 -- 172.21.15.189:0/1180741377 learned_addr learned my addr 172.21.15.189:0/1180741377 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:13:08.480 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.469+0000 7f19c5617700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f19c00a7760 0x7f19c013a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:08.480 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.469+0000 7f19c4e16700 1 -- 172.21.15.189:0/1180741377 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f19c00a7760 msgr2=0x7f19c013a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:08.480 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.469+0000 7f19c4e16700 1 --2- 172.21.15.189:0/1180741377 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f19c00a7760 0x7f19c013a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:08.480 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.469+0000 7f19c4e16700 1 -- 172.21.15.189:0/1180741377 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f19c00a9a40 msgr2=0x7f19c00a5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:08.481 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.469+0000 7f19c4e16700 1 --2- 172.21.15.189:0/1180741377 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f19c00a9a40 0x7f19c00a5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:08.481 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.469+0000 7f19c4e16700 1 -- 172.21.15.189:0/1180741377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c00a1c50 con 0x7f19c00a9660 2020-11-20T12:13:08.481 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.469+0000 7f19c4e16700 1 --2- 172.21.15.189:0/1180741377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19c00a9660 0x7f19c00a67b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f19c0068950 tx=0x7f19b4015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:08.481 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.469+0000 7f19bd7fa700 1 -- 172.21.15.189:0/1180741377 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b4009af0 con 0x7f19c00a9660 2020-11-20T12:13:08.481 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.469+0000 7f19c787b700 1 -- 172.21.15.189:0/1180741377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c00a1ef0 con 0x7f19c00a9660 2020-11-20T12:13:08.482 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.469+0000 7f19bd7fa700 1 -- 172.21.15.189:0/1180741377 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b4009c50 con 0x7f19c00a9660 2020-11-20T12:13:08.482 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.469+0000 7f19bd7fa700 1 -- 172.21.15.189:0/1180741377 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b401e460 con 0x7f19c00a9660 2020-11-20T12:13:08.482 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.469+0000 7f19c787b700 1 -- 172.21.15.189:0/1180741377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c00a5750 con 0x7f19c00a9660 2020-11-20T12:13:08.482 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.469+0000 7f19c787b700 1 -- 172.21.15.189:0/1180741377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19c0044f60 con 0x7f19c00a9660 2020-11-20T12:13:08.482 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.469+0000 7f19bd7fa700 1 -- 172.21.15.189:0/1180741377 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f19b4025030 con 0x7f19c00a9660 2020-11-20T12:13:08.483 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.473+0000 7f19bd7fa700 1 --2- 172.21.15.189:0/1180741377 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f19980557c0 0x7f1998057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:08.484 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.473+0000 7f19bd7fa700 1 -- 172.21.15.189:0/1180741377 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6589+0+0 (secure 0 0 0) 0x7f19b406fd20 con 0x7f19c00a9660 2020-11-20T12:13:08.484 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.473+0000 7f19c5617700 1 --2- 172.21.15.189:0/1180741377 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f19980557c0 0x7f1998057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:08.484 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.473+0000 7f19c5617700 1 --2- 172.21.15.189:0/1180741377 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f19980557c0 0x7f1998057c80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f19b0007fd0 tx=0x7f19b0007a20).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:08.484 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.477+0000 7f19bd7fa700 1 -- 172.21.15.189:0/1180741377 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f19b4045a70 con 0x7f19c00a9660 2020-11-20T12:13:08.686 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.677+0000 7f19c787b700 1 -- 172.21.15.189:0/1180741377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19c00a2200 con 0x7f19c00a9660 2020-11-20T12:13:08.686 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.681+0000 7f19bd7fa700 1 -- 172.21.15.189:0/1180741377 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f19b4023020 con 0x7f19c00a9660 2020-11-20T12:13:08.687 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:13:08.687 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/1811549340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1811549340},{"type":"v1","addr":"172.21.15.47:6833","nonce":1811549340}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:40.691744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5203":{"gid":5203,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/2651986971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2651986971},{"type":"v1","addr":"172.21.15.189:6835","nonce":2651986971}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:13:08.689 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.681+0000 7f19c787b700 1 -- 172.21.15.189:0/1180741377 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f19980557c0 msgr2=0x7f1998057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:08.689 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.681+0000 7f19c787b700 1 --2- 172.21.15.189:0/1180741377 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f19980557c0 0x7f1998057c80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f19b0007fd0 tx=0x7f19b0007a20).stop 2020-11-20T12:13:08.689 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.681+0000 7f19c787b700 1 -- 172.21.15.189:0/1180741377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19c00a9660 msgr2=0x7f19c00a67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:08.689 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.681+0000 7f19c787b700 1 --2- 172.21.15.189:0/1180741377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19c00a9660 0x7f19c00a67b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f19c0068950 tx=0x7f19b4015000).stop 2020-11-20T12:13:08.689 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.681+0000 7f19c787b700 1 -- 172.21.15.189:0/1180741377 shutdown_connections 2020-11-20T12:13:08.690 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.681+0000 7f19c787b700 1 --2- 172.21.15.189:0/1180741377 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f19980557c0 0x7f1998057c80 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:08.690 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.681+0000 7f19c787b700 1 --2- 172.21.15.189:0/1180741377 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f19c00a9a40 0x7f19c00a5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:08.690 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.681+0000 7f19c787b700 1 --2- 172.21.15.189:0/1180741377 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f19c00a7760 0x7f19c013a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:08.690 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.681+0000 7f19c787b700 1 --2- 172.21.15.189:0/1180741377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19c00a9660 0x7f19c00a67b0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:08.690 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.681+0000 7f19c787b700 1 -- 172.21.15.189:0/1180741377 >> 172.21.15.189:0/1180741377 conn(0x7f19c0061f60 msgr2=0x7f19c00ac9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:08.691 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.681+0000 7f19c787b700 1 -- 172.21.15.189:0/1180741377 shutdown_connections 2020-11-20T12:13:08.691 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:08.681+0000 7f19c787b700 1 -- 172.21.15.189:0/1180741377 wait complete. 2020-11-20T12:13:08.691 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 36 2020-11-20T12:13:08.702 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:13:10.704 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:10.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.833+0000 7fef8f84a700 1 Processor -- start 2020-11-20T12:13:10.839 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.833+0000 7fef8f84a700 1 -- start start 2020-11-20T12:13:10.840 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.833+0000 7fef8f84a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fef880a9800 0x7fef8813ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:10.841 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.833+0000 7fef8f84a700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef8813f260 0x7fef8813b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:10.841 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.833+0000 7fef8f84a700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef880676a0 con 0x7fef8813be50 2020-11-20T12:13:10.841 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.833+0000 7fef8f84a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef88065ec0 con 0x7fef880a9800 2020-11-20T12:13:10.841 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.833+0000 7fef8f84a700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef88067c50 con 0x7fef8813f260 2020-11-20T12:13:10.841 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.833+0000 7fef8d5e6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fef880a9800 0x7fef8813ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:10.842 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.833+0000 7fef8dde7700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fef8813be50 0x7fef88060170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52290/0 (socket says 172.21.15.189:52290) 2020-11-20T12:13:10.842 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.833+0000 7fef8dde7700 1 -- 172.21.15.189:0/1477663699 learned_addr learned my addr 172.21.15.189:0/1477663699 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:13:10.842 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.833+0000 7fef8cde5700 1 --2- 172.21.15.189:0/1477663699 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef8813f260 0x7fef8813b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:10.842 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.833+0000 7fef8d5e6700 1 -- 172.21.15.189:0/1477663699 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef8813f260 msgr2=0x7fef8813b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:10.843 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.833+0000 7fef7ffff700 1 -- 172.21.15.189:0/1477663699 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842170631 0 0) 0x7fef880676a0 con 0x7fef8813be50 2020-11-20T12:13:10.843 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.833+0000 7fef8d5e6700 1 --2- 172.21.15.189:0/1477663699 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef8813f260 0x7fef8813b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:10.843 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.833+0000 7fef8d5e6700 1 -- 172.21.15.189:0/1477663699 >> v1:172.21.15.189:6789/0 conn(0x7fef8813be50 legacy=0x7fef88060170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:10.843 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.833+0000 7fef8d5e6700 1 -- 172.21.15.189:0/1477663699 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef88067160 con 0x7fef880a9800 2020-11-20T12:13:10.843 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.833+0000 7fef8cde5700 1 --2- 172.21.15.189:0/1477663699 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef8813f260 0x7fef8813b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:13:10.844 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.833+0000 7fef8d5e6700 1 --2- 172.21.15.189:0/1477663699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef880a9800 0x7fef8813ed70 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7fef8400b9a0 tx=0x7fef84006df0).ready entity=mon.1 client_cookie=d5d4f240478873f9 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:10.844 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.833+0000 7fef7ffff700 1 -- 172.21.15.189:0/1477663699 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef84009840 con 0x7fef880a9800 2020-11-20T12:13:10.844 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.833+0000 7fef7ffff700 1 -- 172.21.15.189:0/1477663699 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef840099a0 con 0x7fef880a9800 2020-11-20T12:13:10.844 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.833+0000 7fef7ffff700 1 -- 172.21.15.189:0/1477663699 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef84009d20 con 0x7fef880a9800 2020-11-20T12:13:10.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.833+0000 7fef8f84a700 1 -- 172.21.15.189:0/1477663699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef880a9800 msgr2=0x7fef8813ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:10.846 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.833+0000 7fef8f84a700 1 --2- 172.21.15.189:0/1477663699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef880a9800 0x7fef8813ed70 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7fef8400b9a0 tx=0x7fef84006df0).stop 2020-11-20T12:13:10.846 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.837+0000 7fef8f84a700 1 -- 172.21.15.189:0/1477663699 shutdown_connections 2020-11-20T12:13:10.846 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.837+0000 7fef8f84a700 1 --2- 172.21.15.189:0/1477663699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef880a9800 0x7fef8813ed70 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:10.846 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.837+0000 7fef8f84a700 1 --2- 172.21.15.189:0/1477663699 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef8813f260 0x7fef8813b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:10.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.837+0000 7fef8f84a700 1 -- 172.21.15.189:0/1477663699 >> 172.21.15.189:0/1477663699 conn(0x7fef88061f60 msgr2=0x7fef88062370 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:10.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.837+0000 7fef8f84a700 1 -- 172.21.15.189:0/1477663699 shutdown_connections 2020-11-20T12:13:10.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.837+0000 7fef8f84a700 1 -- 172.21.15.189:0/1477663699 wait complete. 2020-11-20T12:13:10.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.837+0000 7fef8f84a700 1 Processor -- start 2020-11-20T12:13:10.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.837+0000 7fef8f84a700 1 -- start start 2020-11-20T12:13:10.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.841+0000 7fef8f84a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef880a9800 0x7fef880aacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:10.850 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.841+0000 7fef8f84a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef8813be50 0x7fef880a4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:10.850 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.841+0000 7fef8f84a700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef8813f260 0x7fef880abd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:10.850 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.841+0000 7fef8f84a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef880ab2e0 con 0x7fef8813be50 2020-11-20T12:13:10.850 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.841+0000 7fef8f84a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef880ab580 con 0x7fef880a9800 2020-11-20T12:13:10.851 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.841+0000 7fef8f84a700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef880ab820 con 0x7fef8813f260 2020-11-20T12:13:10.851 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.841+0000 7fef8d5e6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef880a9800 0x7fef880aacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:10.851 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.841+0000 7fef8d5e6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef880a9800 0x7fef880aacb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50672/0 (socket says 172.21.15.189:50672) 2020-11-20T12:13:10.851 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.841+0000 7fef8d5e6700 1 -- 172.21.15.189:0/240483023 learned_addr learned my addr 172.21.15.189:0/240483023 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:13:10.851 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.841+0000 7fef8cde5700 1 --2- 172.21.15.189:0/240483023 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef8813be50 0x7fef880a4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:10.852 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.841+0000 7fef8dde7700 1 --2- 172.21.15.189:0/240483023 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef8813f260 0x7fef880abd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:10.852 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.841+0000 7fef8d5e6700 1 -- 172.21.15.189:0/240483023 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef8813f260 msgr2=0x7fef880abd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:10.852 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.841+0000 7fef8d5e6700 1 --2- 172.21.15.189:0/240483023 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef8813f260 0x7fef880abd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:10.852 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.841+0000 7fef8d5e6700 1 -- 172.21.15.189:0/240483023 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef8813be50 msgr2=0x7fef880a4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:10.852 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.841+0000 7fef8d5e6700 1 --2- 172.21.15.189:0/240483023 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef8813be50 0x7fef880a4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:10.853 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.841+0000 7fef8d5e6700 1 -- 172.21.15.189:0/240483023 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef8813fbd0 con 0x7fef880a9800 2020-11-20T12:13:10.853 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.841+0000 7fef8d5e6700 1 --2- 172.21.15.189:0/240483023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef880a9800 0x7fef880aacb0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fef8400b150 tx=0x7fef84015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:10.853 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.841+0000 7fef7d7fa700 1 -- 172.21.15.189:0/240483023 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef84011d50 con 0x7fef880a9800 2020-11-20T12:13:10.853 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.841+0000 7fef7d7fa700 1 -- 172.21.15.189:0/240483023 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef8401e460 con 0x7fef880a9800 2020-11-20T12:13:10.853 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.841+0000 7fef8f84a700 1 -- 172.21.15.189:0/240483023 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef88142790 con 0x7fef880a9800 2020-11-20T12:13:10.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.841+0000 7fef7d7fa700 1 -- 172.21.15.189:0/240483023 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8401e7e0 con 0x7fef880a9800 2020-11-20T12:13:10.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.841+0000 7fef8f84a700 1 -- 172.21.15.189:0/240483023 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef880ac3e0 con 0x7fef880a9800 2020-11-20T12:13:10.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.841+0000 7fef7d7fa700 1 -- 172.21.15.189:0/240483023 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fef84025030 con 0x7fef880a9800 2020-11-20T12:13:10.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.841+0000 7fef8f84a700 1 -- 172.21.15.189:0/240483023 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef880abb50 con 0x7fef880a9800 2020-11-20T12:13:10.855 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.845+0000 7fef7d7fa700 1 --2- 172.21.15.189:0/240483023 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fef600557c0 0x7fef60057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:10.855 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.845+0000 7fef8cde5700 1 --2- 172.21.15.189:0/240483023 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fef600557c0 0x7fef60057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:10.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.845+0000 7fef7d7fa700 1 -- 172.21.15.189:0/240483023 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6589+0+0 (secure 0 0 0) 0x7fef8406ff90 con 0x7fef880a9800 2020-11-20T12:13:10.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.845+0000 7fef8cde5700 1 --2- 172.21.15.189:0/240483023 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fef600557c0 0x7fef60057c80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fef880a87b0 tx=0x7fef78008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:10.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:10.849+0000 7fef7d7fa700 1 -- 172.21.15.189:0/240483023 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fef84045bd0 con 0x7fef880a9800 2020-11-20T12:13:11.056 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:11.049+0000 7fef8f84a700 1 -- 172.21.15.189:0/240483023 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef88044f60 con 0x7fef880a9800 2020-11-20T12:13:11.057 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:11.049+0000 7fef7d7fa700 1 -- 172.21.15.189:0/240483023 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fef84023070 con 0x7fef880a9800 2020-11-20T12:13:11.057 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:13:11.057 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/1811549340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1811549340},{"type":"v1","addr":"172.21.15.47:6833","nonce":1811549340}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:40.691744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5203":{"gid":5203,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/2651986971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2651986971},{"type":"v1","addr":"172.21.15.189:6835","nonce":2651986971}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:13:11.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:11.053+0000 7fef8f84a700 1 -- 172.21.15.189:0/240483023 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fef600557c0 msgr2=0x7fef60057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:11.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:11.053+0000 7fef8f84a700 1 --2- 172.21.15.189:0/240483023 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fef600557c0 0x7fef60057c80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fef880a87b0 tx=0x7fef78008040).stop 2020-11-20T12:13:11.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:11.053+0000 7fef8f84a700 1 -- 172.21.15.189:0/240483023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef880a9800 msgr2=0x7fef880aacb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:11.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:11.053+0000 7fef8f84a700 1 --2- 172.21.15.189:0/240483023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef880a9800 0x7fef880aacb0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fef8400b150 tx=0x7fef84015000).stop 2020-11-20T12:13:11.061 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:11.053+0000 7fef8f84a700 1 -- 172.21.15.189:0/240483023 shutdown_connections 2020-11-20T12:13:11.061 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:11.053+0000 7fef8f84a700 1 --2- 172.21.15.189:0/240483023 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef8813f260 0x7fef880abd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:11.061 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:11.053+0000 7fef8f84a700 1 --2- 172.21.15.189:0/240483023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef880a9800 0x7fef880aacb0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:11.061 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:11.053+0000 7fef8f84a700 1 --2- 172.21.15.189:0/240483023 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fef600557c0 0x7fef60057c80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:11.053+0000 7fef8f84a700 1 --2- 172.21.15.189:0/240483023 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef8813be50 0x7fef880a4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:11.053+0000 7fef8f84a700 1 -- 172.21.15.189:0/240483023 >> 172.21.15.189:0/240483023 conn(0x7fef88061f60 msgr2=0x7fef880608c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:11.053+0000 7fef8f84a700 1 -- 172.21.15.189:0/240483023 shutdown_connections 2020-11-20T12:13:11.062 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:11.053+0000 7fef8f84a700 1 -- 172.21.15.189:0/240483023 wait complete. 2020-11-20T12:13:11.063 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 36 2020-11-20T12:13:11.071 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:13:13.073 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:13.218 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.209+0000 7fb06cc11700 1 Processor -- start 2020-11-20T12:13:13.241 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.209+0000 7fb06cc11700 1 -- start start 2020-11-20T12:13:13.243 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06cc11700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb0680aa680 0x7fb0680a70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:13.243 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06cc11700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0680a90c0 con 0x7fb0680a9d80 2020-11-20T12:13:13.244 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06cc11700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0680ab630 con 0x7fb0680a75c0 2020-11-20T12:13:13.244 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06cc11700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0680ab890 con 0x7fb0680aa680 2020-11-20T12:13:13.244 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb065d9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb0680a75c0 0x7fb0680a9670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52302/0 (socket says 172.21.15.189:52302) 2020-11-20T12:13:13.244 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb065d9b700 1 -- 172.21.15.189:0/3564081527 learned_addr learned my addr 172.21.15.189:0/3564081527 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:13:13.245 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06659c700 1 --2- 172.21.15.189:0/3564081527 >> v2:172.21.15.47:3300/0 conn(0x7fb0680aa680 0x7fb0680a70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:13.245 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06559a700 1 -- 172.21.15.189:0/3564081527 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164149549 0 0) 0x7fb0680ab630 con 0x7fb0680a75c0 2020-11-20T12:13:13.246 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06559a700 1 -- 172.21.15.189:0/3564081527 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb04c003240 con 0x7fb0680a75c0 2020-11-20T12:13:13.246 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06559a700 1 -- 172.21.15.189:0/3564081527 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2823367458 0 0) 0x7fb04c003240 con 0x7fb0680a75c0 2020-11-20T12:13:13.247 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06559a700 1 -- 172.21.15.189:0/3564081527 >> v2:172.21.15.47:3300/0 conn(0x7fb0680aa680 msgr2=0x7fb0680a70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:13.247 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06559a700 1 --2- 172.21.15.189:0/3564081527 >> v2:172.21.15.47:3300/0 conn(0x7fb0680aa680 0x7fb0680a70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:13.248 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06559a700 1 -- 172.21.15.189:0/3564081527 >> v1:172.21.15.47:6789/0 conn(0x7fb0680a9d80 legacy=0x7fb068135030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:13.248 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06559a700 1 -- 172.21.15.189:0/3564081527 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0680abb30 con 0x7fb0680a75c0 2020-11-20T12:13:13.248 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06559a700 1 -- 172.21.15.189:0/3564081527 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7fb050001860 con 0x7fb0680a75c0 2020-11-20T12:13:13.248 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06559a700 1 -- 172.21.15.189:0/3564081527 >> v1:172.21.15.189:6789/0 conn(0x7fb0680a75c0 legacy=0x7fb0680a9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:13.249 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06559a700 1 --2- 172.21.15.189:0/3564081527 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb04c003730 0x7fb04c003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:13.249 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06559a700 1 --2- 172.21.15.189:0/3564081527 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb04c0044e0 0x7fb04c004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:13.249 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06559a700 1 --2- 172.21.15.189:0/3564081527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb04c004e50 0x7fb04c00b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:13.249 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06559a700 1 -- 172.21.15.189:0/3564081527 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0680ab890 con 0x7fb04c003730 2020-11-20T12:13:13.250 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06559a700 1 -- 172.21.15.189:0/3564081527 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c0041f0 con 0x7fb04c004e50 2020-11-20T12:13:13.250 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06559a700 1 -- 172.21.15.189:0/3564081527 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c00bc20 con 0x7fb04c0044e0 2020-11-20T12:13:13.250 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb066d9d700 1 --2- 172.21.15.189:0/3564081527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb04c004e50 0x7fb04c00b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:13.254 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06659c700 1 --2- 172.21.15.189:0/3564081527 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb04c003730 0x7fb04c003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:13.254 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb066d9d700 1 -- 172.21.15.189:0/3564081527 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb04c0044e0 msgr2=0x7fb04c004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:13.254 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb066d9d700 1 --2- 172.21.15.189:0/3564081527 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb04c0044e0 0x7fb04c004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:13:13.254 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb066d9d700 1 -- 172.21.15.189:0/3564081527 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb04c003730 msgr2=0x7fb04c003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:13.254 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb066d9d700 1 --2- 172.21.15.189:0/3564081527 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb04c003730 0x7fb04c003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:13.255 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb066d9d700 1 -- 172.21.15.189:0/3564081527 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04c003240 con 0x7fb04c004e50 2020-11-20T12:13:13.255 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06659c700 1 --2- 172.21.15.189:0/3564081527 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb04c003730 0x7fb04c003b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:13:13.255 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb066d9d700 1 -- 172.21.15.189:0/3564081527 reap_dead start 2020-11-20T12:13:13.255 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb066d9d700 1 --2- 172.21.15.189:0/3564081527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb04c004e50 0x7fb04c00b730 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fb060001a00 tx=0x7fb060007b70).ready entity=mon.1 client_cookie=e9304f7da0a7f41b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:13.256 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06559a700 1 -- 172.21.15.189:0/3564081527 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06000c830 con 0x7fb04c004e50 2020-11-20T12:13:13.256 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06559a700 1 -- 172.21.15.189:0/3564081527 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb06000c990 con 0x7fb04c004e50 2020-11-20T12:13:13.256 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06559a700 1 -- 172.21.15.189:0/3564081527 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06000cd50 con 0x7fb04c004e50 2020-11-20T12:13:13.256 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06cc11700 1 -- 172.21.15.189:0/3564081527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb04c004e50 msgr2=0x7fb04c00b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:13.256 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06cc11700 1 --2- 172.21.15.189:0/3564081527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb04c004e50 0x7fb04c00b730 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fb060001a00 tx=0x7fb060007b70).stop 2020-11-20T12:13:13.257 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06cc11700 1 -- 172.21.15.189:0/3564081527 shutdown_connections 2020-11-20T12:13:13.257 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06cc11700 1 --2- 172.21.15.189:0/3564081527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb04c004e50 0x7fb04c00b730 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:13.257 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06cc11700 1 -- 172.21.15.189:0/3564081527 >> 172.21.15.189:0/3564081527 conn(0x7fb068061f60 msgr2=0x7fb068062340 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:13.257 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06cc11700 1 -- 172.21.15.189:0/3564081527 shutdown_connections 2020-11-20T12:13:13.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.213+0000 7fb06cc11700 1 -- 172.21.15.189:0/3564081527 wait complete. 2020-11-20T12:13:13.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.217+0000 7fb06cc11700 1 Processor -- start 2020-11-20T12:13:13.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.217+0000 7fb06cc11700 1 -- start start 2020-11-20T12:13:13.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.217+0000 7fb06cc11700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb04c004e50 0x7fb0680a2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:13.258 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.217+0000 7fb06cc11700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb068147840 0x7fb0680addb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:13.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.217+0000 7fb06cc11700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb0680ad320 0x7fb068131ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:13.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.217+0000 7fb06cc11700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0680af0b0 con 0x7fb04c004e50 2020-11-20T12:13:13.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.217+0000 7fb06cc11700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb068062d40 con 0x7fb068147840 2020-11-20T12:13:13.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.217+0000 7fb06cc11700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0680a3610 con 0x7fb0680ad320 2020-11-20T12:13:13.259 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.217+0000 7fb065d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb068147840 0x7fb0680addb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:13.260 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.217+0000 7fb065d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb068147840 0x7fb0680addb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50694/0 (socket says 172.21.15.189:50694) 2020-11-20T12:13:13.260 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.217+0000 7fb065d9b700 1 -- 172.21.15.189:0/2633979907 learned_addr learned my addr 172.21.15.189:0/2633979907 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:13:13.260 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.217+0000 7fb066d9d700 1 --2- 172.21.15.189:0/2633979907 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb0680ad320 0x7fb068131ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:13.260 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.217+0000 7fb06659c700 1 --2- 172.21.15.189:0/2633979907 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb04c004e50 0x7fb0680a2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:13.260 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.217+0000 7fb065d9b700 1 -- 172.21.15.189:0/2633979907 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb0680ad320 msgr2=0x7fb068131ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:13.261 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.217+0000 7fb065d9b700 1 --2- 172.21.15.189:0/2633979907 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb0680ad320 0x7fb068131ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:13.261 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.217+0000 7fb065d9b700 1 -- 172.21.15.189:0/2633979907 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb04c004e50 msgr2=0x7fb0680a2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:13.261 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.217+0000 7fb065d9b700 1 --2- 172.21.15.189:0/2633979907 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb04c004e50 0x7fb0680a2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:13.261 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.217+0000 7fb065d9b700 1 -- 172.21.15.189:0/2633979907 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0680a3370 con 0x7fb068147840 2020-11-20T12:13:13.261 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.217+0000 7fb066d9d700 1 --2- 172.21.15.189:0/2633979907 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb0680ad320 0x7fb068131ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:13:13.262 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.217+0000 7fb06659c700 1 --2- 172.21.15.189:0/2633979907 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb04c004e50 0x7fb0680a2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:13:13.262 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.221+0000 7fb065d9b700 1 --2- 172.21.15.189:0/2633979907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb068147840 0x7fb0680addb0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fb05000cde0 tx=0x7fb0680abcc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:13.262 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.221+0000 7fb05e7fc700 1 -- 172.21.15.189:0/2633979907 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05000aac0 con 0x7fb068147840 2020-11-20T12:13:13.262 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.221+0000 7fb05e7fc700 1 -- 172.21.15.189:0/2633979907 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05000ac40 con 0x7fb068147840 2020-11-20T12:13:13.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.221+0000 7fb06cc11700 1 -- 172.21.15.189:0/2633979907 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0680ace80 con 0x7fb068147840 2020-11-20T12:13:13.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.221+0000 7fb05e7fc700 1 -- 172.21.15.189:0/2633979907 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb050007170 con 0x7fb068147840 2020-11-20T12:13:13.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.221+0000 7fb06cc11700 1 -- 172.21.15.189:0/2633979907 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0680ad920 con 0x7fb068147840 2020-11-20T12:13:13.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.221+0000 7fb06cc11700 1 -- 172.21.15.189:0/2633979907 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb068044f60 con 0x7fb068147840 2020-11-20T12:13:13.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.221+0000 7fb05e7fc700 1 -- 172.21.15.189:0/2633979907 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fb050010070 con 0x7fb068147840 2020-11-20T12:13:13.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.221+0000 7fb05e7fc700 1 --2- 172.21.15.189:0/2633979907 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb0380557c0 0x7fb038057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:13.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.221+0000 7fb05e7fc700 1 -- 172.21.15.189:0/2633979907 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6589+0+0 (secure 0 0 0) 0x7fb05006e560 con 0x7fb068147840 2020-11-20T12:13:13.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.221+0000 7fb06659c700 1 --2- 172.21.15.189:0/2633979907 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb0380557c0 0x7fb038057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:13.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.225+0000 7fb06659c700 1 --2- 172.21.15.189:0/2633979907 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb0380557c0 0x7fb038057c80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fb058006a90 tx=0x7fb058008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:13.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.225+0000 7fb05e7fc700 1 -- 172.21.15.189:0/2633979907 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb0500441a0 con 0x7fb068147840 2020-11-20T12:13:13.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.429+0000 7fb06cc11700 1 -- 172.21.15.189:0/2633979907 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0680ad010 con 0x7fb068147840 2020-11-20T12:13:13.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.429+0000 7fb05e7fc700 1 -- 172.21.15.189:0/2633979907 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fb0500477c0 con 0x7fb068147840 2020-11-20T12:13:13.435 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:13:13.435 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/1811549340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1811549340},{"type":"v1","addr":"172.21.15.47:6833","nonce":1811549340}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:40.691744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5203":{"gid":5203,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/2651986971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2651986971},{"type":"v1","addr":"172.21.15.189:6835","nonce":2651986971}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:13:13.437 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.429+0000 7fb06cc11700 1 -- 172.21.15.189:0/2633979907 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb0380557c0 msgr2=0x7fb038057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:13.437 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.429+0000 7fb06cc11700 1 --2- 172.21.15.189:0/2633979907 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb0380557c0 0x7fb038057c80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fb058006a90 tx=0x7fb058008040).stop 2020-11-20T12:13:13.437 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.429+0000 7fb06cc11700 1 -- 172.21.15.189:0/2633979907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb068147840 msgr2=0x7fb0680addb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:13.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.429+0000 7fb06cc11700 1 --2- 172.21.15.189:0/2633979907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb068147840 0x7fb0680addb0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fb05000cde0 tx=0x7fb0680abcc0).stop 2020-11-20T12:13:13.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.429+0000 7fb06cc11700 1 -- 172.21.15.189:0/2633979907 shutdown_connections 2020-11-20T12:13:13.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.429+0000 7fb06cc11700 1 --2- 172.21.15.189:0/2633979907 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb0380557c0 0x7fb038057c80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:13.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.429+0000 7fb06cc11700 1 --2- 172.21.15.189:0/2633979907 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb04c004e50 0x7fb0680a2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:13.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.429+0000 7fb06cc11700 1 --2- 172.21.15.189:0/2633979907 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb0680ad320 0x7fb068131ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:13.439 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.429+0000 7fb06cc11700 1 --2- 172.21.15.189:0/2633979907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb068147840 0x7fb0680addb0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:13.439 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.429+0000 7fb06cc11700 1 -- 172.21.15.189:0/2633979907 >> 172.21.15.189:0/2633979907 conn(0x7fb068061f60 msgr2=0x7fb06805ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:13.439 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.429+0000 7fb06cc11700 1 -- 172.21.15.189:0/2633979907 shutdown_connections 2020-11-20T12:13:13.439 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:13.429+0000 7fb06cc11700 1 -- 172.21.15.189:0/2633979907 wait complete. 2020-11-20T12:13:13.440 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 36 2020-11-20T12:13:13.452 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:13:15.453 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:15.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.581+0000 7fbb84823700 1 Processor -- start 2020-11-20T12:13:15.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.581+0000 7fbb84823700 1 -- start start 2020-11-20T12:13:15.588 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.581+0000 7fbb84823700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fbb800ae390 0x7fbb80137b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:15.588 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.581+0000 7fbb84823700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb800aeb40 con 0x7fbb80067ca0 2020-11-20T12:13:15.589 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.581+0000 7fbb84823700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb800ae8a0 con 0x7fbb800678c0 2020-11-20T12:13:15.589 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.581+0000 7fbb84823700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb80137690 con 0x7fbb800ae390 2020-11-20T12:13:15.589 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.581+0000 7fbb7e59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbb800678c0 0x7fbb800a8c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52324/0 (socket says 172.21.15.189:52324) 2020-11-20T12:13:15.589 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.581+0000 7fbb7e59c700 1 -- 172.21.15.189:0/3562585263 learned_addr learned my addr 172.21.15.189:0/3562585263 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:13:15.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.581+0000 7fbb7ed9d700 1 --2- 172.21.15.189:0/3562585263 >> v2:172.21.15.47:3300/0 conn(0x7fbb800ae390 0x7fbb80137b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:15.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.581+0000 7fbb7dd9b700 1 -- 172.21.15.189:0/3562585263 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2834206624 0 0) 0x7fbb800ae8a0 con 0x7fbb800678c0 2020-11-20T12:13:15.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.581+0000 7fbb7dd9b700 1 -- 172.21.15.189:0/3562585263 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb60003240 con 0x7fbb800678c0 2020-11-20T12:13:15.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.581+0000 7fbb7dd9b700 1 -- 172.21.15.189:0/3562585263 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2742653669 0 0) 0x7fbb800aeb40 con 0x7fbb80067ca0 2020-11-20T12:13:15.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.581+0000 7fbb7dd9b700 1 -- 172.21.15.189:0/3562585263 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb800ae8a0 con 0x7fbb80067ca0 2020-11-20T12:13:15.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.581+0000 7fbb7dd9b700 1 -- 172.21.15.189:0/3562585263 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2229366035 0 0) 0x7fbb60003240 con 0x7fbb800678c0 2020-11-20T12:13:15.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.581+0000 7fbb7ed9d700 1 -- 172.21.15.189:0/3562585263 >> v1:172.21.15.189:6789/0 conn(0x7fbb800678c0 legacy=0x7fbb800a8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:15.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.581+0000 7fbb7ed9d700 1 -- 172.21.15.189:0/3562585263 >> v1:172.21.15.47:6789/0 conn(0x7fbb80067ca0 legacy=0x7fbb8013b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:15.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.581+0000 7fbb7ed9d700 1 -- 172.21.15.189:0/3562585263 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb80138020 con 0x7fbb800ae390 2020-11-20T12:13:15.594 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.581+0000 7fbb7ed9d700 1 --2- 172.21.15.189:0/3562585263 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb800ae390 0x7fbb80137b30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fbb78009c10 tx=0x7fbb780091d0).ready entity=mon.0 client_cookie=f58a7609d4710ae4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:15.594 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.581+0000 7fbb7dd9b700 1 -- 172.21.15.189:0/3562585263 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb700014e0 con 0x7fbb800ae390 2020-11-20T12:13:15.595 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.581+0000 7fbb7dd9b700 1 -- 172.21.15.189:0/3562585263 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb7800b730 con 0x7fbb800ae390 2020-11-20T12:13:15.595 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.581+0000 7fbb7dd9b700 1 -- 172.21.15.189:0/3562585263 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7800bab0 con 0x7fbb800ae390 2020-11-20T12:13:15.595 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb84823700 1 -- 172.21.15.189:0/3562585263 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb800ae390 msgr2=0x7fbb80137b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:15.595 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb84823700 1 --2- 172.21.15.189:0/3562585263 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb800ae390 0x7fbb80137b30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fbb78009c10 tx=0x7fbb780091d0).stop 2020-11-20T12:13:15.595 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb84823700 1 -- 172.21.15.189:0/3562585263 shutdown_connections 2020-11-20T12:13:15.596 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb84823700 1 --2- 172.21.15.189:0/3562585263 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb800ae390 0x7fbb80137b30 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:15.596 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb84823700 1 -- 172.21.15.189:0/3562585263 >> 172.21.15.189:0/3562585263 conn(0x7fbb800625a0 msgr2=0x7fbb800adc80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:15.599 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb84823700 1 -- 172.21.15.189:0/3562585263 shutdown_connections 2020-11-20T12:13:15.600 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb84823700 1 -- 172.21.15.189:0/3562585263 wait complete. 2020-11-20T12:13:15.600 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb84823700 1 Processor -- start 2020-11-20T12:13:15.600 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb84823700 1 -- start start 2020-11-20T12:13:15.600 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb84823700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb800678c0 0x7fbb80139ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:15.601 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb84823700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbb80067ca0 0x7fbb8012f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:15.601 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb84823700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb800ae390 0x7fbb8012fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:15.601 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb84823700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb80138710 con 0x7fbb800678c0 2020-11-20T12:13:15.602 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb84823700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb800ad340 con 0x7fbb800ae390 2020-11-20T12:13:15.602 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb84823700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb800ad5e0 con 0x7fbb80067ca0 2020-11-20T12:13:15.602 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb7ed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb800ae390 0x7fbb8012fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:15.602 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb7ed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb800ae390 0x7fbb8012fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50710/0 (socket says 172.21.15.189:50710) 2020-11-20T12:13:15.602 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb7ed9d700 1 -- 172.21.15.189:0/969695412 learned_addr learned my addr 172.21.15.189:0/969695412 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:13:15.603 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb77fff700 1 --2- 172.21.15.189:0/969695412 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbb80067ca0 0x7fbb8012f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:15.604 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb7e59c700 1 --2- 172.21.15.189:0/969695412 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb800678c0 0x7fbb80139ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:15.604 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb7ed9d700 1 -- 172.21.15.189:0/969695412 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbb80067ca0 msgr2=0x7fbb8012f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:15.604 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb7ed9d700 1 --2- 172.21.15.189:0/969695412 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbb80067ca0 0x7fbb8012f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:15.605 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb7ed9d700 1 -- 172.21.15.189:0/969695412 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb800678c0 msgr2=0x7fbb80139ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:15.605 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb7ed9d700 1 --2- 172.21.15.189:0/969695412 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb800678c0 0x7fbb80139ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:15.605 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb7ed9d700 1 -- 172.21.15.189:0/969695412 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb800ad880 con 0x7fbb800ae390 2020-11-20T12:13:15.605 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb7ed9d700 1 --2- 172.21.15.189:0/969695412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb800ae390 0x7fbb8012fbe0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fbb78011b90 tx=0x7fbb78016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:15.606 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb767fc700 1 -- 172.21.15.189:0/969695412 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7801d450 con 0x7fbb800ae390 2020-11-20T12:13:15.606 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.585+0000 7fbb767fc700 1 -- 172.21.15.189:0/969695412 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb7801d5b0 con 0x7fbb800ae390 2020-11-20T12:13:15.606 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.589+0000 7fbb84823700 1 -- 172.21.15.189:0/969695412 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb800ad030 con 0x7fbb800ae390 2020-11-20T12:13:15.606 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.589+0000 7fbb767fc700 1 -- 172.21.15.189:0/969695412 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7801d930 con 0x7fbb800ae390 2020-11-20T12:13:15.607 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.589+0000 7fbb84823700 1 -- 172.21.15.189:0/969695412 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb800a78b0 con 0x7fbb800ae390 2020-11-20T12:13:15.608 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.589+0000 7fbb84823700 1 -- 172.21.15.189:0/969695412 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb8013a190 con 0x7fbb800ae390 2020-11-20T12:13:15.608 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.589+0000 7fbb767fc700 1 -- 172.21.15.189:0/969695412 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fbb78023020 con 0x7fbb800ae390 2020-11-20T12:13:15.608 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.589+0000 7fbb767fc700 1 --2- 172.21.15.189:0/969695412 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fbb5c0557f0 0x7fbb5c057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:15.609 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.589+0000 7fbb767fc700 1 -- 172.21.15.189:0/969695412 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6589+0+0 (secure 0 0 0) 0x7fbb7806fdd0 con 0x7fbb800ae390 2020-11-20T12:13:15.609 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.589+0000 7fbb7e59c700 1 --2- 172.21.15.189:0/969695412 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fbb5c0557f0 0x7fbb5c057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:15.609 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.593+0000 7fbb7e59c700 1 --2- 172.21.15.189:0/969695412 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fbb5c0557f0 0x7fbb5c057cb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fbb70000f70 tx=0x7fbb70001a40).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:15.609 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.601+0000 7fbb767fc700 1 -- 172.21.15.189:0/969695412 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbb78045af0 con 0x7fbb800ae390 2020-11-20T12:13:15.810 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.805+0000 7fbb84823700 1 -- 172.21.15.189:0/969695412 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb800ada10 con 0x7fbb800ae390 2020-11-20T12:13:15.811 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.805+0000 7fbb767fc700 1 -- 172.21.15.189:0/969695412 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fbb78015070 con 0x7fbb800ae390 2020-11-20T12:13:15.811 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:13:15.812 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4937,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/1811549340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1811549340},{"type":"v1","addr":"172.21.15.47:6833","nonce":1811549340}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:12:40.691744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":28,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5203},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5203":{"gid":5203,"name":"a","rank":0,"incarnation":32,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/2651986971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2651986971},{"type":"v1","addr":"172.21.15.189:6835","nonce":2651986971}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:13:15.813 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.805+0000 7fbb84823700 1 -- 172.21.15.189:0/969695412 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fbb5c0557f0 msgr2=0x7fbb5c057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:15.813 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.805+0000 7fbb84823700 1 --2- 172.21.15.189:0/969695412 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fbb5c0557f0 0x7fbb5c057cb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fbb70000f70 tx=0x7fbb70001a40).stop 2020-11-20T12:13:15.813 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.805+0000 7fbb84823700 1 -- 172.21.15.189:0/969695412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb800ae390 msgr2=0x7fbb8012fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:15.814 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.805+0000 7fbb84823700 1 --2- 172.21.15.189:0/969695412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb800ae390 0x7fbb8012fbe0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fbb78011b90 tx=0x7fbb78016000).stop 2020-11-20T12:13:15.814 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.805+0000 7fbb84823700 1 -- 172.21.15.189:0/969695412 shutdown_connections 2020-11-20T12:13:15.814 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.805+0000 7fbb84823700 1 --2- 172.21.15.189:0/969695412 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fbb5c0557f0 0x7fbb5c057cb0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:15.815 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.805+0000 7fbb84823700 1 --2- 172.21.15.189:0/969695412 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fbb800678c0 0x7fbb80139ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:15.815 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.805+0000 7fbb84823700 1 --2- 172.21.15.189:0/969695412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb800ae390 0x7fbb8012fbe0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:15.815 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.805+0000 7fbb84823700 1 --2- 172.21.15.189:0/969695412 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fbb80067ca0 0x7fbb8012f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:15.815 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.805+0000 7fbb84823700 1 -- 172.21.15.189:0/969695412 >> 172.21.15.189:0/969695412 conn(0x7fbb800625a0 msgr2=0x7fbb80063b60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:15.816 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.805+0000 7fbb84823700 1 -- 172.21.15.189:0/969695412 shutdown_connections 2020-11-20T12:13:15.816 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:15.805+0000 7fbb84823700 1 -- 172.21.15.189:0/969695412 wait complete. 2020-11-20T12:13:15.816 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 36 2020-11-20T12:13:15.826 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:13:17.827 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:17.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.953+0000 7f0aeec8e700 1 Processor -- start 2020-11-20T12:13:17.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.953+0000 7f0aeec8e700 1 -- start start 2020-11-20T12:13:17.960 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.953+0000 7f0aeec8e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ae80678c0 0x7f0ae80a8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:17.960 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.953+0000 7f0aeec8e700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0ae80ae390 0x7f0ae8137b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:17.960 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.953+0000 7f0aeec8e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae80aeb40 con 0x7f0ae8067ca0 2020-11-20T12:13:17.960 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.953+0000 7f0aeec8e700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ae80ae8a0 con 0x7f0ae80ae390 2020-11-20T12:13:17.961 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.953+0000 7f0aeec8e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae8137690 con 0x7f0ae80678c0 2020-11-20T12:13:17.961 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.953+0000 7f0ae7fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0ae8067ca0 0x7f0ae813b330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52336/0 (socket says 172.21.15.189:52336) 2020-11-20T12:13:17.961 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.953+0000 7f0ae7fff700 1 -- 172.21.15.189:0/2026227733 learned_addr learned my addr 172.21.15.189:0/2026227733 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:13:17.961 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.953+0000 7f0aed22b700 1 --2- 172.21.15.189:0/2026227733 >> v2:172.21.15.47:3300/0 conn(0x7f0ae80ae390 0x7f0ae8137b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:17.961 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.953+0000 7f0aeca2a700 1 --2- 172.21.15.189:0/2026227733 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ae80678c0 0x7f0ae80a8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:17.963 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.953+0000 7f0ae77fe700 1 -- 172.21.15.189:0/2026227733 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950838923 0 0) 0x7f0ae80aeb40 con 0x7f0ae8067ca0 2020-11-20T12:13:17.963 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.953+0000 7f0ae77fe700 1 -- 172.21.15.189:0/2026227733 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ac8003240 con 0x7f0ae8067ca0 2020-11-20T12:13:17.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.953+0000 7f0aeca2a700 1 -- 172.21.15.189:0/2026227733 >> v2:172.21.15.47:3300/0 conn(0x7f0ae80ae390 msgr2=0x7f0ae8137b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:17.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.953+0000 7f0aeca2a700 1 --2- 172.21.15.189:0/2026227733 >> v2:172.21.15.47:3300/0 conn(0x7f0ae80ae390 0x7f0ae8137b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:17.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.953+0000 7f0aeca2a700 1 -- 172.21.15.189:0/2026227733 >> v1:172.21.15.189:6789/0 conn(0x7f0ae8067ca0 legacy=0x7f0ae813b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:17.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.953+0000 7f0aeca2a700 1 -- 172.21.15.189:0/2026227733 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae80a9170 con 0x7f0ae80678c0 2020-11-20T12:13:17.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.953+0000 7f0aed22b700 1 --2- 172.21.15.189:0/2026227733 >> v2:172.21.15.47:3300/0 conn(0x7f0ae80ae390 0x7f0ae8137b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:13:17.965 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.953+0000 7f0aeca2a700 1 --2- 172.21.15.189:0/2026227733 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ae80678c0 0x7f0ae80a8c80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f0ad000b9a0 tx=0x7f0ad0006df0).ready entity=mon.2 client_cookie=45e8fc8fee97ffef server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:17.965 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.953+0000 7f0ae77fe700 1 -- 172.21.15.189:0/2026227733 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad0009680 con 0x7f0ae80678c0 2020-11-20T12:13:17.965 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.953+0000 7f0ae77fe700 1 -- 172.21.15.189:0/2026227733 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ad00097e0 con 0x7f0ae80678c0 2020-11-20T12:13:17.965 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.953+0000 7f0ae77fe700 1 -- 172.21.15.189:0/2026227733 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad0009b60 con 0x7f0ae80678c0 2020-11-20T12:13:17.966 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.957+0000 7f0aeec8e700 1 -- 172.21.15.189:0/2026227733 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ae80678c0 msgr2=0x7f0ae80a8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:17.966 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.957+0000 7f0aeec8e700 1 --2- 172.21.15.189:0/2026227733 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ae80678c0 0x7f0ae80a8c80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f0ad000b9a0 tx=0x7f0ad0006df0).stop 2020-11-20T12:13:17.966 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.957+0000 7f0aeec8e700 1 -- 172.21.15.189:0/2026227733 shutdown_connections 2020-11-20T12:13:17.966 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.957+0000 7f0aeec8e700 1 --2- 172.21.15.189:0/2026227733 >> v2:172.21.15.47:3300/0 conn(0x7f0ae80ae390 0x7f0ae8137b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:17.966 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.957+0000 7f0aeec8e700 1 --2- 172.21.15.189:0/2026227733 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ae80678c0 0x7f0ae80a8c80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:17.966 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.957+0000 7f0aeec8e700 1 -- 172.21.15.189:0/2026227733 >> 172.21.15.189:0/2026227733 conn(0x7f0ae80625a0 msgr2=0x7f0ae80adc80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:17.967 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.957+0000 7f0aeec8e700 1 -- 172.21.15.189:0/2026227733 shutdown_connections 2020-11-20T12:13:17.967 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.957+0000 7f0aeec8e700 1 -- 172.21.15.189:0/2026227733 wait complete. 2020-11-20T12:13:17.969 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.957+0000 7f0aeec8e700 1 Processor -- start 2020-11-20T12:13:17.969 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.957+0000 7f0aeec8e700 1 -- start start 2020-11-20T12:13:17.969 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.957+0000 7f0aeec8e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ae80678c0 0x7f0ae8139840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:17.969 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.957+0000 7f0aeec8e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ae8067ca0 0x7f0ae80a77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:17.970 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.957+0000 7f0aeec8e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ae80ae390 0x7f0ae8146820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:17.970 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.957+0000 7f0aeec8e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae80a3bd0 con 0x7f0ae8067ca0 2020-11-20T12:13:17.970 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.957+0000 7f0aeec8e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae80a3e70 con 0x7f0ae80ae390 2020-11-20T12:13:17.970 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.957+0000 7f0aeec8e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae80a4310 con 0x7f0ae80678c0 2020-11-20T12:13:17.970 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.961+0000 7f0aed22b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ae80ae390 0x7f0ae8146820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:17.971 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.961+0000 7f0aeca2a700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ae80678c0 0x7f0ae8139840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:17.971 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.961+0000 7f0aed22b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ae80ae390 0x7f0ae8146820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50724/0 (socket says 172.21.15.189:50724) 2020-11-20T12:13:17.971 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.961+0000 7f0aeca2a700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ae80678c0 0x7f0ae8139840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.189:43754/0 (socket says 172.21.15.189:43754) 2020-11-20T12:13:17.971 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.961+0000 7f0aed22b700 1 -- 172.21.15.189:0/3822232211 learned_addr learned my addr 172.21.15.189:0/3822232211 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:13:17.971 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.961+0000 7f0ae7fff700 1 --2- 172.21.15.189:0/3822232211 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ae8067ca0 0x7f0ae80a77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:17.972 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.961+0000 7f0aed22b700 1 -- 172.21.15.189:0/3822232211 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ae80678c0 msgr2=0x7f0ae8139840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:17.972 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.961+0000 7f0aed22b700 1 --2- 172.21.15.189:0/3822232211 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ae80678c0 0x7f0ae8139840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:17.972 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.961+0000 7f0aed22b700 1 -- 172.21.15.189:0/3822232211 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ae8067ca0 msgr2=0x7f0ae80a77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:17.972 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.961+0000 7f0aed22b700 1 --2- 172.21.15.189:0/3822232211 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ae8067ca0 0x7f0ae80a77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:17.972 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.961+0000 7f0aed22b700 1 -- 172.21.15.189:0/3822232211 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae80a7ca0 con 0x7f0ae80ae390 2020-11-20T12:13:17.973 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.961+0000 7f0aed22b700 1 --2- 172.21.15.189:0/3822232211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ae80ae390 0x7f0ae8146820 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f0ae000ef50 tx=0x7f0ae00091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:17.973 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.961+0000 7f0ae4ff9700 1 -- 172.21.15.189:0/3822232211 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae000cc10 con 0x7f0ae80ae390 2020-11-20T12:13:17.973 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.961+0000 7f0ae4ff9700 1 -- 172.21.15.189:0/3822232211 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ae000cd70 con 0x7f0ae80ae390 2020-11-20T12:13:17.973 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.961+0000 7f0ae4ff9700 1 -- 172.21.15.189:0/3822232211 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0012620 con 0x7f0ae80ae390 2020-11-20T12:13:17.973 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.961+0000 7f0aeec8e700 1 -- 172.21.15.189:0/3822232211 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ae80a5e50 con 0x7f0ae80ae390 2020-11-20T12:13:17.974 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.961+0000 7f0aeec8e700 1 -- 172.21.15.189:0/3822232211 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ae8146eb0 con 0x7f0ae80ae390 2020-11-20T12:13:17.974 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.961+0000 7f0ae4ff9700 1 -- 172.21.15.189:0/3822232211 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f0ae0022070 con 0x7f0ae80ae390 2020-11-20T12:13:17.974 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.961+0000 7f0aeec8e700 1 -- 172.21.15.189:0/3822232211 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ae80a4030 con 0x7f0ae80ae390 2020-11-20T12:13:17.975 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.961+0000 7f0ae4ff9700 1 --2- 172.21.15.189:0/3822232211 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0ac40557f0 0x7f0ac4057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:17.975 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.961+0000 7f0aeca2a700 1 --2- 172.21.15.189:0/3822232211 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0ac40557f0 0x7f0ac4057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:17.975 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.961+0000 7f0ae4ff9700 1 -- 172.21.15.189:0/3822232211 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f0ae006f1b0 con 0x7f0ae80ae390 2020-11-20T12:13:17.975 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.965+0000 7f0aeca2a700 1 --2- 172.21.15.189:0/3822232211 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0ac40557f0 0x7f0ac4057cb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f0ad0006e30 tx=0x7f0ad0015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:17.976 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:17.969+0000 7f0ae4ff9700 1 -- 172.21.15.189:0/3822232211 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0ae0044ce0 con 0x7f0ae80ae390 2020-11-20T12:13:18.178 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.173+0000 7f0aeec8e700 1 -- 172.21.15.189:0/3822232211 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ae80a6040 con 0x7f0ae80ae390 2020-11-20T12:13:18.179 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.173+0000 7f0ae4ff9700 1 -- 172.21.15.189:0/3822232211 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+1784 (secure 0 0 0) 0x7f0ae0016090 con 0x7f0ae80ae390 2020-11-20T12:13:18.179 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:13:18.179 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:13:17.650030+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"b","rank":0,"incarnation":37,"state":"up:replay","state_seq":1,"addr":"172.21.15.47:6833/1811549340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1811549340},{"type":"v1","addr":"172.21.15.47:6833","nonce":1811549340}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:13:18.180 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.173+0000 7f0aeec8e700 1 -- 172.21.15.189:0/3822232211 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0ac40557f0 msgr2=0x7f0ac4057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:18.180 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.173+0000 7f0aeec8e700 1 --2- 172.21.15.189:0/3822232211 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0ac40557f0 0x7f0ac4057cb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f0ad0006e30 tx=0x7f0ad0015000).stop 2020-11-20T12:13:18.180 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.173+0000 7f0aeec8e700 1 -- 172.21.15.189:0/3822232211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ae80ae390 msgr2=0x7f0ae8146820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:18.180 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.173+0000 7f0aeec8e700 1 --2- 172.21.15.189:0/3822232211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ae80ae390 0x7f0ae8146820 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f0ae000ef50 tx=0x7f0ae00091d0).stop 2020-11-20T12:13:18.181 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.173+0000 7f0aeec8e700 1 -- 172.21.15.189:0/3822232211 shutdown_connections 2020-11-20T12:13:18.182 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.173+0000 7f0aeec8e700 1 --2- 172.21.15.189:0/3822232211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ae80ae390 0x7f0ae8146820 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:18.182 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.173+0000 7f0aeec8e700 1 --2- 172.21.15.189:0/3822232211 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0ae80678c0 0x7f0ae8139840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:18.182 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.173+0000 7f0aeec8e700 1 --2- 172.21.15.189:0/3822232211 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0ac40557f0 0x7f0ac4057cb0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:18.182 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.173+0000 7f0aeec8e700 1 --2- 172.21.15.189:0/3822232211 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0ae8067ca0 0x7f0ae80a77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:18.182 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.173+0000 7f0aeec8e700 1 -- 172.21.15.189:0/3822232211 >> 172.21.15.189:0/3822232211 conn(0x7f0ae80625a0 msgr2=0x7f0ae80aa690 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:18.183 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.173+0000 7f0aeec8e700 1 -- 172.21.15.189:0/3822232211 shutdown_connections 2020-11-20T12:13:18.183 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.173+0000 7f0aeec8e700 1 -- 172.21.15.189:0/3822232211 wait complete. 2020-11-20T12:13:18.183 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 37 2020-11-20T12:13:18.193 INFO:tasks.mds_thrash.fs.[None]:mds.a down, removed from mdsmap 2020-11-20T12:13:18.194 INFO:tasks.mds_thrash.fs.[None]:waiting for mds cluster to stabilize... 2020-11-20T12:13:18.194 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:18.320 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.313+0000 7f4a3edbf700 1 Processor -- start 2020-11-20T12:13:18.320 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.313+0000 7f4a3edbf700 1 -- start start 2020-11-20T12:13:18.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.313+0000 7f4a3edbf700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4a380a8840 0x7f4a3813dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:18.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.313+0000 7f4a3edbf700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4a3813e100 0x7f4a381371b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:18.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.313+0000 7f4a3edbf700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a38066800 con 0x7f4a381376a0 2020-11-20T12:13:18.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.313+0000 7f4a3edbf700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a38066aa0 con 0x7f4a3813e100 2020-11-20T12:13:18.328 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.313+0000 7f4a3edbf700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a380672f0 con 0x7f4a380a8840 2020-11-20T12:13:18.328 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.313+0000 7f4a3d5bc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4a3813e100 0x7f4a381371b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:18.328 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.313+0000 7f4a3d5bc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4a3813e100 0x7f4a381371b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50728/0 (socket says 172.21.15.189:50728) 2020-11-20T12:13:18.328 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.313+0000 7f4a3d5bc700 1 -- 172.21.15.189:0/1448374662 learned_addr learned my addr 172.21.15.189:0/1448374662 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:13:18.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.313+0000 7f4a3ddbd700 1 --2- 172.21.15.189:0/1448374662 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4a380a8840 0x7f4a3813dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:18.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.313+0000 7f4a3d5bc700 1 -- 172.21.15.189:0/1448374662 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4a380a8840 msgr2=0x7f4a3813dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:18.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.313+0000 7f4a3d5bc700 1 --2- 172.21.15.189:0/1448374662 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4a380a8840 0x7f4a3813dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:18.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.313+0000 7f4a3d5bc700 1 -- 172.21.15.189:0/1448374662 >> v1:172.21.15.47:6789/0 conn(0x7f4a381376a0 legacy=0x7f4a3813a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:18.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.313+0000 7f4a3d5bc700 1 -- 172.21.15.189:0/1448374662 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a38067590 con 0x7f4a3813e100 2020-11-20T12:13:18.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.313+0000 7f4a3d5bc700 1 --2- 172.21.15.189:0/1448374662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a3813e100 0x7f4a381371b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f4a2c00ba20 tx=0x7f4a2c006eb0).ready entity=mon.1 client_cookie=23cb1d3ac8ac0a6c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:18.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.313+0000 7f4a3cdbb700 1 -- 172.21.15.189:0/1448374662 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c0097d0 con 0x7f4a3813e100 2020-11-20T12:13:18.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.313+0000 7f4a3cdbb700 1 -- 172.21.15.189:0/1448374662 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2c009930 con 0x7f4a3813e100 2020-11-20T12:13:18.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.313+0000 7f4a3cdbb700 1 -- 172.21.15.189:0/1448374662 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c009cb0 con 0x7f4a3813e100 2020-11-20T12:13:18.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.317+0000 7f4a3edbf700 1 -- 172.21.15.189:0/1448374662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a3813e100 msgr2=0x7f4a381371b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:18.331 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.317+0000 7f4a3edbf700 1 --2- 172.21.15.189:0/1448374662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a3813e100 0x7f4a381371b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f4a2c00ba20 tx=0x7f4a2c006eb0).stop 2020-11-20T12:13:18.331 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.317+0000 7f4a3edbf700 1 -- 172.21.15.189:0/1448374662 shutdown_connections 2020-11-20T12:13:18.331 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.317+0000 7f4a3edbf700 1 --2- 172.21.15.189:0/1448374662 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4a380a8840 0x7f4a3813dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:18.331 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.317+0000 7f4a3edbf700 1 --2- 172.21.15.189:0/1448374662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a3813e100 0x7f4a381371b0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:18.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.317+0000 7f4a3edbf700 1 -- 172.21.15.189:0/1448374662 >> 172.21.15.189:0/1448374662 conn(0x7f4a38060170 msgr2=0x7f4a38060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:18.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.317+0000 7f4a3edbf700 1 -- 172.21.15.189:0/1448374662 shutdown_connections 2020-11-20T12:13:18.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.317+0000 7f4a3edbf700 1 -- 172.21.15.189:0/1448374662 wait complete. 2020-11-20T12:13:18.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.317+0000 7f4a3edbf700 1 Processor -- start 2020-11-20T12:13:18.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.317+0000 7f4a3edbf700 1 -- start start 2020-11-20T12:13:18.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.317+0000 7f4a3edbf700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4a380a8840 0x7f4a380a4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:18.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.317+0000 7f4a3edbf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a381376a0 0x7f4a38138e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:18.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.317+0000 7f4a3edbf700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4a3813e100 0x7f4a38149be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:18.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.317+0000 7f4a3edbf700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a38139930 con 0x7f4a380a8840 2020-11-20T12:13:18.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.317+0000 7f4a3edbf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a38139bd0 con 0x7f4a381376a0 2020-11-20T12:13:18.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.317+0000 7f4a3edbf700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a38139e70 con 0x7f4a3813e100 2020-11-20T12:13:18.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.317+0000 7f4a3d5bc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a381376a0 0x7f4a38138e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:18.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.317+0000 7f4a3d5bc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a381376a0 0x7f4a38138e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50734/0 (socket says 172.21.15.189:50734) 2020-11-20T12:13:18.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.317+0000 7f4a3d5bc700 1 -- 172.21.15.189:0/1253226633 learned_addr learned my addr 172.21.15.189:0/1253226633 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:13:18.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.317+0000 7f4a3ddbd700 1 --2- 172.21.15.189:0/1253226633 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4a380a8840 0x7f4a380a4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:18.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.321+0000 7f4a3d5bc700 1 -- 172.21.15.189:0/1253226633 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4a3813e100 msgr2=0x7f4a38149be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:18.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.321+0000 7f4a3e5be700 1 --2- 172.21.15.189:0/1253226633 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4a3813e100 0x7f4a38149be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:18.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.321+0000 7f4a3d5bc700 1 --2- 172.21.15.189:0/1253226633 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4a3813e100 0x7f4a38149be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:18.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.321+0000 7f4a3d5bc700 1 -- 172.21.15.189:0/1253226633 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4a380a8840 msgr2=0x7f4a380a4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:18.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.321+0000 7f4a3d5bc700 1 --2- 172.21.15.189:0/1253226633 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4a380a8840 0x7f4a380a4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:18.336 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.321+0000 7f4a3d5bc700 1 -- 172.21.15.189:0/1253226633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a38138430 con 0x7f4a381376a0 2020-11-20T12:13:18.336 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.321+0000 7f4a3ddbd700 1 --2- 172.21.15.189:0/1253226633 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4a380a8840 0x7f4a380a4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:13:18.336 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.321+0000 7f4a3e5be700 1 --2- 172.21.15.189:0/1253226633 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4a3813e100 0x7f4a38149be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:13:18.336 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.321+0000 7f4a3d5bc700 1 --2- 172.21.15.189:0/1253226633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a381376a0 0x7f4a38138e90 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f4a2c00c010 tx=0x7f4a2c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:18.336 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.321+0000 7f4a29ffb700 1 -- 172.21.15.189:0/1253226633 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c011dd0 con 0x7f4a381376a0 2020-11-20T12:13:18.337 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.321+0000 7f4a29ffb700 1 -- 172.21.15.189:0/1253226633 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2c01e460 con 0x7f4a381376a0 2020-11-20T12:13:18.337 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.321+0000 7f4a3edbf700 1 -- 172.21.15.189:0/1253226633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a38139380 con 0x7f4a381376a0 2020-11-20T12:13:18.337 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.321+0000 7f4a29ffb700 1 -- 172.21.15.189:0/1253226633 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c01e800 con 0x7f4a381376a0 2020-11-20T12:13:18.337 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.321+0000 7f4a3edbf700 1 -- 172.21.15.189:0/1253226633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a3813a4b0 con 0x7f4a381376a0 2020-11-20T12:13:18.338 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.321+0000 7f4a29ffb700 1 -- 172.21.15.189:0/1253226633 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f4a2c01a040 con 0x7f4a381376a0 2020-11-20T12:13:18.339 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.321+0000 7f4a29ffb700 1 --2- 172.21.15.189:0/1253226633 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4a100557c0 0x7f4a10057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:18.339 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.321+0000 7f4a29ffb700 1 -- 172.21.15.189:0/1253226633 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f4a2c070250 con 0x7f4a381376a0 2020-11-20T12:13:18.339 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.321+0000 7f4a3ddbd700 1 --2- 172.21.15.189:0/1253226633 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4a100557c0 0x7f4a10057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:18.339 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.321+0000 7f4a3edbf700 1 -- 172.21.15.189:0/1253226633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a380446d0 con 0x7f4a381376a0 2020-11-20T12:13:18.340 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.321+0000 7f4a3ddbd700 1 --2- 172.21.15.189:0/1253226633 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4a100557c0 0x7f4a10057c80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f4a34006fd0 tx=0x7f4a34008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:18.340 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.329+0000 7f4a29ffb700 1 -- 172.21.15.189:0/1253226633 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4a2c023080 con 0x7f4a381376a0 2020-11-20T12:13:18.534 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.529+0000 7f4a3edbf700 1 -- 172.21.15.189:0/1253226633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a3805cdd0 con 0x7f4a381376a0 2020-11-20T12:13:18.535 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.529+0000 7f4a29ffb700 1 -- 172.21.15.189:0/1253226633 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+1784 (secure 0 0 0) 0x7f4a2c045e90 con 0x7f4a381376a0 2020-11-20T12:13:18.535 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:13:18.535 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:13:17.650030+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"b","rank":0,"incarnation":37,"state":"up:replay","state_seq":1,"addr":"172.21.15.47:6833/1811549340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1811549340},{"type":"v1","addr":"172.21.15.47:6833","nonce":1811549340}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:13:18.540 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.529+0000 7f4a3edbf700 1 -- 172.21.15.189:0/1253226633 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4a100557c0 msgr2=0x7f4a10057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:18.540 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.529+0000 7f4a3edbf700 1 --2- 172.21.15.189:0/1253226633 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4a100557c0 0x7f4a10057c80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f4a34006fd0 tx=0x7f4a34008040).stop 2020-11-20T12:13:18.541 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.529+0000 7f4a3edbf700 1 -- 172.21.15.189:0/1253226633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a381376a0 msgr2=0x7f4a38138e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:18.541 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.529+0000 7f4a3edbf700 1 --2- 172.21.15.189:0/1253226633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a381376a0 0x7f4a38138e90 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f4a2c00c010 tx=0x7f4a2c015000).stop 2020-11-20T12:13:18.542 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.529+0000 7f4a3edbf700 1 -- 172.21.15.189:0/1253226633 shutdown_connections 2020-11-20T12:13:18.542 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.529+0000 7f4a3edbf700 1 --2- 172.21.15.189:0/1253226633 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f4a100557c0 0x7f4a10057c80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:18.542 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.529+0000 7f4a3edbf700 1 --2- 172.21.15.189:0/1253226633 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f4a380a8840 0x7f4a380a4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:18.543 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.529+0000 7f4a3edbf700 1 --2- 172.21.15.189:0/1253226633 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f4a3813e100 0x7f4a38149be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:18.543 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.529+0000 7f4a3edbf700 1 --2- 172.21.15.189:0/1253226633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a381376a0 0x7f4a38138e90 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:18.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.529+0000 7f4a3edbf700 1 -- 172.21.15.189:0/1253226633 >> 172.21.15.189:0/1253226633 conn(0x7f4a38060170 msgr2=0x7f4a38133e60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:18.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.529+0000 7f4a3edbf700 1 -- 172.21.15.189:0/1253226633 shutdown_connections 2020-11-20T12:13:18.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:18.529+0000 7f4a3edbf700 1 -- 172.21.15.189:0/1253226633 wait complete. 2020-11-20T12:13:18.545 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 37 2020-11-20T12:13:18.550 INFO:tasks.mds_thrash.fs.[None]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 37, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2020-11-20T12:04:54.012110+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 37, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4937": { "addr": "172.21.15.47:6833/1811549340", "addrs": { "addrvec": [ { "addr": "172.21.15.47:6832", "nonce": 1811549340, "type": "v2" }, { "addr": "172.21.15.47:6833", "nonce": 1811549340, "type": "v1" } ] }, "export_targets": [], "features": 4540138297136906239, "flags": 0, "gid": 4937, "incarnation": 37, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 29, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2020-11-20T12:13:17.650030+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4937 } } } ], "standbys": [] } 2020-11-20T12:13:20.554 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:20.692 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.685+0000 7f303e328700 1 Processor -- start 2020-11-20T12:13:20.692 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.685+0000 7f303e328700 1 -- start start 2020-11-20T12:13:20.694 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.685+0000 7f303e328700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f30380a81b0 0x7f30380aa3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:20.694 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.685+0000 7f303e328700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30380a8fa0 con 0x7f3038060310 2020-11-20T12:13:20.694 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.685+0000 7f303e328700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30380a9240 con 0x7f30380a7d40 2020-11-20T12:13:20.694 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.685+0000 7f303e328700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30380618b0 con 0x7f30380a81b0 2020-11-20T12:13:20.695 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.685+0000 7f303cb25700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f30380a7d40 0x7f30380a9cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52366/0 (socket says 172.21.15.189:52366) 2020-11-20T12:13:20.695 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.685+0000 7f303cb25700 1 -- 172.21.15.189:0/3582785290 learned_addr learned my addr 172.21.15.189:0/3582785290 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:13:20.695 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.685+0000 7f303db27700 1 --2- 172.21.15.189:0/3582785290 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f30380a81b0 0x7f30380aa3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:20.695 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.685+0000 7f3027fff700 1 -- 172.21.15.189:0/3582785290 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035249953 0 0) 0x7f30380a9240 con 0x7f30380a7d40 2020-11-20T12:13:20.695 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.685+0000 7f3027fff700 1 -- 172.21.15.189:0/3582785290 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3020003240 con 0x7f30380a7d40 2020-11-20T12:13:20.696 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.685+0000 7f3027fff700 1 -- 172.21.15.189:0/3582785290 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026323316 0 0) 0x7f30380a8fa0 con 0x7f3038060310 2020-11-20T12:13:20.699 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.685+0000 7f3027fff700 1 -- 172.21.15.189:0/3582785290 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30380a9240 con 0x7f3038060310 2020-11-20T12:13:20.699 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.685+0000 7f303db27700 1 -- 172.21.15.189:0/3582785290 >> v1:172.21.15.189:6789/0 conn(0x7f30380a7d40 legacy=0x7f30380a9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:20.699 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303db27700 1 -- 172.21.15.189:0/3582785290 >> v1:172.21.15.47:6789/0 conn(0x7f3038060310 legacy=0x7f30380a1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:20.699 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303db27700 1 -- 172.21.15.189:0/3582785290 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30380a94e0 con 0x7f30380a81b0 2020-11-20T12:13:20.699 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303db27700 1 --2- 172.21.15.189:0/3582785290 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f30380a81b0 0x7f30380aa3d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f3030009cc0 tx=0x7f30300091d0).ready entity=mon.2 client_cookie=8682ebb060758664 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:20.700 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f3027fff700 1 -- 172.21.15.189:0/3582785290 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303000bab0 con 0x7f30380a81b0 2020-11-20T12:13:20.700 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f3027fff700 1 -- 172.21.15.189:0/3582785290 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f303000bc10 con 0x7f30380a81b0 2020-11-20T12:13:20.700 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f3027fff700 1 -- 172.21.15.189:0/3582785290 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3030011400 con 0x7f30380a81b0 2020-11-20T12:13:20.700 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303e328700 1 -- 172.21.15.189:0/3582785290 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f30380a81b0 msgr2=0x7f30380aa3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:20.701 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303e328700 1 --2- 172.21.15.189:0/3582785290 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f30380a81b0 0x7f30380aa3d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f3030009cc0 tx=0x7f30300091d0).stop 2020-11-20T12:13:20.701 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303e328700 1 -- 172.21.15.189:0/3582785290 shutdown_connections 2020-11-20T12:13:20.701 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303e328700 1 --2- 172.21.15.189:0/3582785290 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f30380a81b0 0x7f30380aa3d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:20.701 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303e328700 1 -- 172.21.15.189:0/3582785290 >> 172.21.15.189:0/3582785290 conn(0x7f3038063210 msgr2=0x7f3038060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:20.701 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303e328700 1 -- 172.21.15.189:0/3582785290 shutdown_connections 2020-11-20T12:13:20.702 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303e328700 1 -- 172.21.15.189:0/3582785290 wait complete. 2020-11-20T12:13:20.702 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303e328700 1 Processor -- start 2020-11-20T12:13:20.702 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303e328700 1 -- start start 2020-11-20T12:13:20.702 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303e328700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3038060310 0x7f303813e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:20.702 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303e328700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f30380a7d40 0x7f303809c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:20.703 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303e328700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f30380a81b0 0x7f30381476b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:20.703 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303e328700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3038147ea0 con 0x7f30380a81b0 2020-11-20T12:13:20.703 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303e328700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3038148140 con 0x7f3038060310 2020-11-20T12:13:20.703 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303e328700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30381483e0 con 0x7f30380a7d40 2020-11-20T12:13:20.704 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303d326700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3038060310 0x7f303813e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:20.704 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303cb25700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f30380a7d40 0x7f303809c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:20.704 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303d326700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3038060310 0x7f303813e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50750/0 (socket says 172.21.15.189:50750) 2020-11-20T12:13:20.704 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303d326700 1 -- 172.21.15.189:0/2446541352 learned_addr learned my addr 172.21.15.189:0/2446541352 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:13:20.704 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303db27700 1 --2- 172.21.15.189:0/2446541352 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f30380a81b0 0x7f30381476b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:20.705 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303d326700 1 -- 172.21.15.189:0/2446541352 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f30380a7d40 msgr2=0x7f303809c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:20.705 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303d326700 1 --2- 172.21.15.189:0/2446541352 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f30380a7d40 0x7f303809c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:20.705 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303d326700 1 -- 172.21.15.189:0/2446541352 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f30380a81b0 msgr2=0x7f30381476b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:20.705 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303d326700 1 --2- 172.21.15.189:0/2446541352 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f30380a81b0 0x7f30381476b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:20.705 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.689+0000 7f303d326700 1 -- 172.21.15.189:0/2446541352 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3038147ba0 con 0x7f3038060310 2020-11-20T12:13:20.706 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.693+0000 7f303d326700 1 --2- 172.21.15.189:0/2446541352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3038060310 0x7f303813e5d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f302800c710 tx=0x7f3028007a90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:20.706 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.693+0000 7f30257fa700 1 -- 172.21.15.189:0/2446541352 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302800e490 con 0x7f3038060310 2020-11-20T12:13:20.706 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.693+0000 7f303e328700 1 -- 172.21.15.189:0/2446541352 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f303814ad40 con 0x7f3038060310 2020-11-20T12:13:20.706 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.693+0000 7f30257fa700 1 -- 172.21.15.189:0/2446541352 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f302800e5f0 con 0x7f3038060310 2020-11-20T12:13:20.706 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.693+0000 7f30257fa700 1 -- 172.21.15.189:0/2446541352 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f302800e990 con 0x7f3038060310 2020-11-20T12:13:20.707 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.693+0000 7f303e328700 1 -- 172.21.15.189:0/2446541352 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f303814b290 con 0x7f3038060310 2020-11-20T12:13:20.708 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.693+0000 7f303e328700 1 -- 172.21.15.189:0/2446541352 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30381485a0 con 0x7f3038060310 2020-11-20T12:13:20.708 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.693+0000 7f30257fa700 1 -- 172.21.15.189:0/2446541352 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f3028010030 con 0x7f3038060310 2020-11-20T12:13:20.708 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.693+0000 7f30257fa700 1 --2- 172.21.15.189:0/2446541352 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f30100557c0 0x7f3010057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:20.708 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.693+0000 7f30257fa700 1 -- 172.21.15.189:0/2446541352 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f302806e720 con 0x7f3038060310 2020-11-20T12:13:20.708 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.693+0000 7f303cb25700 1 --2- 172.21.15.189:0/2446541352 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f30100557c0 0x7f3010057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:20.709 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.693+0000 7f303cb25700 1 --2- 172.21.15.189:0/2446541352 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f30100557c0 0x7f3010057c80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f303813f4d0 tx=0x7f3034009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:20.709 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.701+0000 7f30257fa700 1 -- 172.21.15.189:0/2446541352 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3028044360 con 0x7f3038060310 2020-11-20T12:13:20.918 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.913+0000 7f303e328700 1 -- 172.21.15.189:0/2446541352 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30380446d0 con 0x7f3038060310 2020-11-20T12:13:20.918 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.913+0000 7f30257fa700 1 -- 172.21.15.189:0/2446541352 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+1785 (secure 0 0 0) 0x7f3028047980 con 0x7f3038060310 2020-11-20T12:13:20.919 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:13:20.919 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:13:20.671656+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":12,"addr":"172.21.15.47:6833/1811549340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1811549340},{"type":"v1","addr":"172.21.15.47:6833","nonce":1811549340}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:13:20.920 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.913+0000 7f303e328700 1 -- 172.21.15.189:0/2446541352 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f30100557c0 msgr2=0x7f3010057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:20.921 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.913+0000 7f303e328700 1 --2- 172.21.15.189:0/2446541352 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f30100557c0 0x7f3010057c80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f303813f4d0 tx=0x7f3034009040).stop 2020-11-20T12:13:20.921 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.913+0000 7f303e328700 1 -- 172.21.15.189:0/2446541352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3038060310 msgr2=0x7f303813e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:20.921 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.913+0000 7f303e328700 1 --2- 172.21.15.189:0/2446541352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3038060310 0x7f303813e5d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f302800c710 tx=0x7f3028007a90).stop 2020-11-20T12:13:20.921 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.913+0000 7f303e328700 1 -- 172.21.15.189:0/2446541352 shutdown_connections 2020-11-20T12:13:20.922 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.913+0000 7f303e328700 1 --2- 172.21.15.189:0/2446541352 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f30100557c0 0x7f3010057c80 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:20.922 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.913+0000 7f303e328700 1 --2- 172.21.15.189:0/2446541352 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f30380a81b0 0x7f30381476b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:20.922 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.913+0000 7f303e328700 1 --2- 172.21.15.189:0/2446541352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3038060310 0x7f303813e5d0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:20.922 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.913+0000 7f303e328700 1 --2- 172.21.15.189:0/2446541352 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f30380a7d40 0x7f303809c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:20.922 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.913+0000 7f303e328700 1 -- 172.21.15.189:0/2446541352 >> 172.21.15.189:0/2446541352 conn(0x7f3038063210 msgr2=0x7f303805ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:20.923 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.913+0000 7f303e328700 1 -- 172.21.15.189:0/2446541352 shutdown_connections 2020-11-20T12:13:20.923 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:20.913+0000 7f303e328700 1 -- 172.21.15.189:0/2446541352 wait complete. 2020-11-20T12:13:20.923 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 40 2020-11-20T12:13:20.937 INFO:tasks.mds_thrash.fs.[None]:mds.b has gained rank=0, replacing gid=5203 2020-11-20T12:13:20.937 INFO:tasks.mds_thrash.fs.[None]:waiting for 0 secs before reviving mds.a 2020-11-20T12:13:20.938 INFO:tasks.mds_thrash.fs.[None]:reviving mds.a 2020-11-20T12:13:20.938 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:13:20.938 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:13:20.942 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:13:20.943 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:21.012 INFO:tasks.ceph.mds.a.smithi189.stdout:starting mds.a at 2020-11-20T12:13:21.064 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.057+0000 7fc357fff700 1 Processor -- start 2020-11-20T12:13:21.064 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.057+0000 7fc357fff700 1 -- start start 2020-11-20T12:13:21.066 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc357fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc3580a8690 0x7fc3580a98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:21.067 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc357fff700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc35813fcb0 0x7fc3581420a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:21.067 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc357fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3580661c0 con 0x7fc358134270 2020-11-20T12:13:21.067 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc357fff700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3580671a0 con 0x7fc35813fcb0 2020-11-20T12:13:21.068 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc357fff700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3580a9530 con 0x7fc3580a8690 2020-11-20T12:13:21.068 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc356ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc3580a8690 0x7fc3580a98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:21.068 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc356ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc3580a8690 0x7fc3580a98e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50770/0 (socket says 172.21.15.189:50770) 2020-11-20T12:13:21.068 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc356ffd700 1 -- 172.21.15.189:0/3069772847 learned_addr learned my addr 172.21.15.189:0/3069772847 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:13:21.072 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc3577fe700 1 --2- 172.21.15.189:0/3069772847 >> v2:172.21.15.47:3300/0 conn(0x7fc35813fcb0 0x7fc3581420a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:21.073 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc356ffd700 1 -- 172.21.15.189:0/3069772847 >> v2:172.21.15.47:3300/0 conn(0x7fc35813fcb0 msgr2=0x7fc3581420a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:21.073 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc356ffd700 1 --2- 172.21.15.189:0/3069772847 >> v2:172.21.15.47:3300/0 conn(0x7fc35813fcb0 0x7fc3581420a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:21.073 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc356ffd700 1 -- 172.21.15.189:0/3069772847 >> v1:172.21.15.47:6789/0 conn(0x7fc358134270 legacy=0x7fc3581346d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:21.073 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc356ffd700 1 -- 172.21.15.189:0/3069772847 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc358066460 con 0x7fc3580a8690 2020-11-20T12:13:21.074 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc3577fe700 1 --2- 172.21.15.189:0/3069772847 >> v2:172.21.15.47:3300/0 conn(0x7fc35813fcb0 0x7fc3581420a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:13:21.074 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc356ffd700 1 --2- 172.21.15.189:0/3069772847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3580a8690 0x7fc3580a98e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fc3580ac970 tx=0x7fc348006eb0).ready entity=mon.1 client_cookie=61bfdfbf70f716b8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:21.074 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc3567fc700 1 -- 172.21.15.189:0/3069772847 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc348009920 con 0x7fc3580a8690 2020-11-20T12:13:21.074 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc3567fc700 1 -- 172.21.15.189:0/3069772847 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc348009a80 con 0x7fc3580a8690 2020-11-20T12:13:21.075 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc3567fc700 1 -- 172.21.15.189:0/3069772847 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc348009e00 con 0x7fc3580a8690 2020-11-20T12:13:21.075 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc357fff700 1 -- 172.21.15.189:0/3069772847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3580a8690 msgr2=0x7fc3580a98e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:21.075 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc357fff700 1 --2- 172.21.15.189:0/3069772847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3580a8690 0x7fc3580a98e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fc3580ac970 tx=0x7fc348006eb0).stop 2020-11-20T12:13:21.075 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc357fff700 1 -- 172.21.15.189:0/3069772847 shutdown_connections 2020-11-20T12:13:21.076 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc357fff700 1 --2- 172.21.15.189:0/3069772847 >> v2:172.21.15.47:3300/0 conn(0x7fc35813fcb0 0x7fc3581420a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:21.076 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc357fff700 1 --2- 172.21.15.189:0/3069772847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3580a8690 0x7fc3580a98e0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:21.076 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc357fff700 1 -- 172.21.15.189:0/3069772847 >> 172.21.15.189:0/3069772847 conn(0x7fc358062080 msgr2=0x7fc358062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:21.076 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc357fff700 1 -- 172.21.15.189:0/3069772847 shutdown_connections 2020-11-20T12:13:21.077 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc357fff700 1 -- 172.21.15.189:0/3069772847 wait complete. 2020-11-20T12:13:21.077 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc357fff700 1 Processor -- start 2020-11-20T12:13:21.077 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.061+0000 7fc357fff700 1 -- start start 2020-11-20T12:13:21.077 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.065+0000 7fc357fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3580a8690 0x7fc35814ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:21.078 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.065+0000 7fc357fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc358134270 0x7fc3581338e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:21.078 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.065+0000 7fc357fff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc35813fcb0 0x7fc3580ab9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:21.078 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.065+0000 7fc357fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35814d4b0 con 0x7fc3580a8690 2020-11-20T12:13:21.078 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.065+0000 7fc357fff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35814d750 con 0x7fc358134270 2020-11-20T12:13:21.079 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.065+0000 7fc357fff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc35814d9f0 con 0x7fc35813fcb0 2020-11-20T12:13:21.079 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.065+0000 7fc34ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc358134270 0x7fc3581338e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:21.079 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.065+0000 7fc34ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc358134270 0x7fc3581338e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50778/0 (socket says 172.21.15.189:50778) 2020-11-20T12:13:21.079 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.065+0000 7fc34ffff700 1 -- 172.21.15.189:0/1687648928 learned_addr learned my addr 172.21.15.189:0/1687648928 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:13:21.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.065+0000 7fc356ffd700 1 --2- 172.21.15.189:0/1687648928 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3580a8690 0x7fc35814ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:21.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.065+0000 7fc3577fe700 1 --2- 172.21.15.189:0/1687648928 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc35813fcb0 0x7fc3580ab9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:21.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.065+0000 7fc34ffff700 1 -- 172.21.15.189:0/1687648928 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc35813fcb0 msgr2=0x7fc3580ab9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:21.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.065+0000 7fc34ffff700 1 --2- 172.21.15.189:0/1687648928 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc35813fcb0 0x7fc3580ab9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:21.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.065+0000 7fc34ffff700 1 -- 172.21.15.189:0/1687648928 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3580a8690 msgr2=0x7fc35814ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:21.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.065+0000 7fc34ffff700 1 --2- 172.21.15.189:0/1687648928 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3580a8690 0x7fc35814ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:21.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.065+0000 7fc34ffff700 1 -- 172.21.15.189:0/1687648928 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3581498b0 con 0x7fc358134270 2020-11-20T12:13:21.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.065+0000 7fc34ffff700 1 --2- 172.21.15.189:0/1687648928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc358134270 0x7fc3581338e0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7fc34000c730 tx=0x7fc340007c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:21.082 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.065+0000 7fc34effd700 1 -- 172.21.15.189:0/1687648928 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34000e560 con 0x7fc358134270 2020-11-20T12:13:21.082 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.065+0000 7fc34effd700 1 -- 172.21.15.189:0/1687648928 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc34000e6e0 con 0x7fc358134270 2020-11-20T12:13:21.083 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.065+0000 7fc357fff700 1 -- 172.21.15.189:0/1687648928 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc358149b60 con 0x7fc358134270 2020-11-20T12:13:21.083 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.065+0000 7fc34effd700 1 -- 172.21.15.189:0/1687648928 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34000eaa0 con 0x7fc358134270 2020-11-20T12:13:21.083 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.065+0000 7fc357fff700 1 -- 172.21.15.189:0/1687648928 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc35814a030 con 0x7fc358134270 2020-11-20T12:13:21.083 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.065+0000 7fc357fff700 1 -- 172.21.15.189:0/1687648928 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3580446d0 con 0x7fc358134270 2020-11-20T12:13:21.084 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.069+0000 7fc34effd700 1 -- 172.21.15.189:0/1687648928 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fc340019030 con 0x7fc358134270 2020-11-20T12:13:21.085 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.069+0000 7fc34effd700 1 --2- 172.21.15.189:0/1687648928 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc3340557c0 0x7fc334057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:21.085 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.069+0000 7fc34effd700 1 -- 172.21.15.189:0/1687648928 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fc34006de10 con 0x7fc358134270 2020-11-20T12:13:21.085 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.069+0000 7fc356ffd700 1 --2- 172.21.15.189:0/1687648928 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc3340557c0 0x7fc334057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:21.085 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.069+0000 7fc356ffd700 1 --2- 172.21.15.189:0/1687648928 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc3340557c0 0x7fc334057c80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fc348006ef0 tx=0x7fc348015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:21.086 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.073+0000 7fc34effd700 1 -- 172.21.15.189:0/1687648928 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc340043b00 con 0x7fc358134270 2020-11-20T12:13:21.291 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.285+0000 7fc357fff700 1 -- 172.21.15.189:0/1687648928 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc35805cdd0 con 0x7fc358134270 2020-11-20T12:13:21.292 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.285+0000 7fc34effd700 1 -- 172.21.15.189:0/1687648928 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+1785 (secure 0 0 0) 0x7fc340021710 con 0x7fc358134270 2020-11-20T12:13:21.292 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:13:21.292 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:13:20.671656+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":12,"addr":"172.21.15.47:6833/1811549340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1811549340},{"type":"v1","addr":"172.21.15.47:6833","nonce":1811549340}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:13:21.294 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.285+0000 7fc357fff700 1 -- 172.21.15.189:0/1687648928 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc3340557c0 msgr2=0x7fc334057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:21.294 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.285+0000 7fc357fff700 1 --2- 172.21.15.189:0/1687648928 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc3340557c0 0x7fc334057c80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fc348006ef0 tx=0x7fc348015000).stop 2020-11-20T12:13:21.294 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.285+0000 7fc357fff700 1 -- 172.21.15.189:0/1687648928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc358134270 msgr2=0x7fc3581338e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:21.294 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.285+0000 7fc357fff700 1 --2- 172.21.15.189:0/1687648928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc358134270 0x7fc3581338e0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7fc34000c730 tx=0x7fc340007c30).stop 2020-11-20T12:13:21.294 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.285+0000 7fc357fff700 1 -- 172.21.15.189:0/1687648928 shutdown_connections 2020-11-20T12:13:21.295 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.285+0000 7fc357fff700 1 --2- 172.21.15.189:0/1687648928 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc3340557c0 0x7fc334057c80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:21.295 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.285+0000 7fc357fff700 1 --2- 172.21.15.189:0/1687648928 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc3580a8690 0x7fc35814ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:21.295 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.285+0000 7fc357fff700 1 --2- 172.21.15.189:0/1687648928 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc35813fcb0 0x7fc3580ab9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:21.295 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.285+0000 7fc357fff700 1 --2- 172.21.15.189:0/1687648928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc358134270 0x7fc3581338e0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:21.296 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.285+0000 7fc357fff700 1 -- 172.21.15.189:0/1687648928 >> 172.21.15.189:0/1687648928 conn(0x7fc358062080 msgr2=0x7fc3580a3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:21.296 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.285+0000 7fc357fff700 1 -- 172.21.15.189:0/1687648928 shutdown_connections 2020-11-20T12:13:21.296 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:21.285+0000 7fc357fff700 1 -- 172.21.15.189:0/1687648928 wait complete. 2020-11-20T12:13:21.296 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 40 2020-11-20T12:13:21.305 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2020-11-20T12:13:23.307 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:13:23.440 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.433+0000 7fba74b14700 1 Processor -- start 2020-11-20T12:13:23.440 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.433+0000 7fba74b14700 1 -- start start 2020-11-20T12:13:23.441 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.433+0000 7fba74b14700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fba7013b1d0 0x7fba700677f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:23.441 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.433+0000 7fba74b14700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fba701340e0 0x7fba70134570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:23.441 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.433+0000 7fba74b14700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba700a9c60 con 0x7fba700623c0 2020-11-20T12:13:23.442 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.433+0000 7fba74b14700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba700a8e60 con 0x7fba701340e0 2020-11-20T12:13:23.442 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.433+0000 7fba74b14700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba70067e90 con 0x7fba7013b1d0 2020-11-20T12:13:23.442 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.433+0000 7fba6ffff700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fba701340e0 0x7fba70134570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:23.442 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.433+0000 7fba6effd700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fba7013b1d0 0x7fba700677f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:23.443 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.433+0000 7fba6f7fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fba700623c0 0x7fba7013aac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.189:47076/0 (socket says 172.21.15.189:47076) 2020-11-20T12:13:23.443 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.433+0000 7fba6f7fe700 1 -- 172.21.15.189:0/3545884698 learned_addr learned my addr 172.21.15.189:0/3545884698 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:13:23.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.433+0000 7fba6ffff700 1 -- 172.21.15.189:0/3545884698 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fba7013b1d0 msgr2=0x7fba700677f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:13:23.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.437+0000 7fba6ffff700 1 --2- 172.21.15.189:0/3545884698 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fba7013b1d0 0x7fba700677f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:23.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.437+0000 7fba6ffff700 1 -- 172.21.15.189:0/3545884698 >> v1:172.21.15.47:6789/0 conn(0x7fba700623c0 legacy=0x7fba7013aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:23.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.437+0000 7fba6e7fc700 1 -- 172.21.15.189:0/3545884698 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2399078690 0 0) 0x7fba700a9c60 con 0x7fba700623c0 2020-11-20T12:13:23.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.437+0000 7fba6ffff700 1 -- 172.21.15.189:0/3545884698 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba70068130 con 0x7fba701340e0 2020-11-20T12:13:23.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.437+0000 7fba6effd700 1 --2- 172.21.15.189:0/3545884698 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fba7013b1d0 0x7fba700677f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:13:23.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.437+0000 7fba6ffff700 1 --2- 172.21.15.189:0/3545884698 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba701340e0 0x7fba70134570 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fba68009c90 tx=0x7fba680091d0).ready entity=mon.0 client_cookie=d0de63c4ad59cb09 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:23.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.437+0000 7fba6e7fc700 1 -- 172.21.15.189:0/3545884698 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6800ba70 con 0x7fba701340e0 2020-11-20T12:13:23.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.437+0000 7fba6e7fc700 1 -- 172.21.15.189:0/3545884698 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba6800bbd0 con 0x7fba701340e0 2020-11-20T12:13:23.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.437+0000 7fba6e7fc700 1 -- 172.21.15.189:0/3545884698 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba68011400 con 0x7fba701340e0 2020-11-20T12:13:23.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.437+0000 7fba74b14700 1 -- 172.21.15.189:0/3545884698 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba701340e0 msgr2=0x7fba70134570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:23.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.437+0000 7fba74b14700 1 --2- 172.21.15.189:0/3545884698 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba701340e0 0x7fba70134570 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fba68009c90 tx=0x7fba680091d0).stop 2020-11-20T12:13:23.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.437+0000 7fba74b14700 1 -- 172.21.15.189:0/3545884698 shutdown_connections 2020-11-20T12:13:23.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.437+0000 7fba74b14700 1 --2- 172.21.15.189:0/3545884698 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba701340e0 0x7fba70134570 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:23.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.437+0000 7fba74b14700 1 --2- 172.21.15.189:0/3545884698 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fba7013b1d0 0x7fba700677f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:23.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.437+0000 7fba74b14700 1 -- 172.21.15.189:0/3545884698 >> 172.21.15.189:0/3545884698 conn(0x7fba70065fe0 msgr2=0x7fba70061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:23.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.437+0000 7fba74b14700 1 -- 172.21.15.189:0/3545884698 shutdown_connections 2020-11-20T12:13:23.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.437+0000 7fba74b14700 1 -- 172.21.15.189:0/3545884698 wait complete. 2020-11-20T12:13:23.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.437+0000 7fba74b14700 1 Processor -- start 2020-11-20T12:13:23.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.437+0000 7fba74b14700 1 -- start start 2020-11-20T12:13:23.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.441+0000 7fba74b14700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba700623c0 0x7fba7014a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:23.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.441+0000 7fba74b14700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fba701340e0 0x7fba7013c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:23.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.441+0000 7fba74b14700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba7013b1d0 0x7fba700a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:23.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.441+0000 7fba74b14700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba70149960 con 0x7fba700623c0 2020-11-20T12:13:23.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.441+0000 7fba74b14700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba70149c00 con 0x7fba7013b1d0 2020-11-20T12:13:23.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.441+0000 7fba74b14700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba70149ea0 con 0x7fba701340e0 2020-11-20T12:13:23.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.441+0000 7fba6ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba7013b1d0 0x7fba700a0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:23.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.441+0000 7fba6ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba7013b1d0 0x7fba700a0a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50796/0 (socket says 172.21.15.189:50796) 2020-11-20T12:13:23.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.441+0000 7fba6ffff700 1 -- 172.21.15.189:0/879374877 learned_addr learned my addr 172.21.15.189:0/879374877 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:13:23.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.441+0000 7fba6f7fe700 1 --2- 172.21.15.189:0/879374877 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba700623c0 0x7fba7014a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:23.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.441+0000 7fba6effd700 1 --2- 172.21.15.189:0/879374877 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fba701340e0 0x7fba7013c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:23.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.441+0000 7fba6ffff700 1 -- 172.21.15.189:0/879374877 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fba701340e0 msgr2=0x7fba7013c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:23.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.441+0000 7fba6ffff700 1 --2- 172.21.15.189:0/879374877 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fba701340e0 0x7fba7013c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:23.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.441+0000 7fba6ffff700 1 -- 172.21.15.189:0/879374877 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba700623c0 msgr2=0x7fba7014a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:23.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.441+0000 7fba6ffff700 1 --2- 172.21.15.189:0/879374877 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba700623c0 0x7fba7014a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:23.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.441+0000 7fba6ffff700 1 -- 172.21.15.189:0/879374877 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba7014d2e0 con 0x7fba7013b1d0 2020-11-20T12:13:23.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.441+0000 7fba6ffff700 1 --2- 172.21.15.189:0/879374877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba7013b1d0 0x7fba700a0a90 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fba68009210 tx=0x7fba68016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:23.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.441+0000 7fba4f7fe700 1 -- 172.21.15.189:0/879374877 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6801c6c0 con 0x7fba7013b1d0 2020-11-20T12:13:23.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.441+0000 7fba4f7fe700 1 -- 172.21.15.189:0/879374877 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba6801c820 con 0x7fba7013b1d0 2020-11-20T12:13:23.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.441+0000 7fba4f7fe700 1 -- 172.21.15.189:0/879374877 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6801cba0 con 0x7fba7013b1d0 2020-11-20T12:13:23.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.441+0000 7fba74b14700 1 -- 172.21.15.189:0/879374877 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba7014d4d0 con 0x7fba7013b1d0 2020-11-20T12:13:23.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.441+0000 7fba74b14700 1 -- 172.21.15.189:0/879374877 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba7014d970 con 0x7fba7013b1d0 2020-11-20T12:13:23.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.441+0000 7fba74b14700 1 -- 172.21.15.189:0/879374877 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba7014a060 con 0x7fba7013b1d0 2020-11-20T12:13:23.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.441+0000 7fba4f7fe700 1 -- 172.21.15.189:0/879374877 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fba68017070 con 0x7fba7013b1d0 2020-11-20T12:13:23.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.445+0000 7fba4f7fe700 1 --2- 172.21.15.189:0/879374877 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fba480557c0 0x7fba48057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:13:23.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.445+0000 7fba4f7fe700 1 -- 172.21.15.189:0/879374877 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fba68070b90 con 0x7fba7013b1d0 2020-11-20T12:13:23.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.445+0000 7fba6f7fe700 1 --2- 172.21.15.189:0/879374877 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fba480557c0 0x7fba48057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:13:23.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.445+0000 7fba6f7fe700 1 --2- 172.21.15.189:0/879374877 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fba480557c0 0x7fba48057c80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fba600011a0 tx=0x7fba600079e0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:13:23.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.449+0000 7fba4f7fe700 1 -- 172.21.15.189:0/879374877 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fba68046830 con 0x7fba7013b1d0 2020-11-20T12:13:23.640 INFO:tasks.ceph.mds.b.smithi047.stderr:2020-11-20T12:13:23.634+0000 7f85b09c8700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:13:23.702 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.697+0000 7fba74b14700 1 -- 172.21.15.189:0/879374877 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba700663c0 con 0x7fba7013b1d0 2020-11-20T12:13:23.707 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.701+0000 7fba4f7fe700 1 -- 172.21.15.189:0/879374877 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2137 (secure 0 0 0) 0x7fba68015070 con 0x7fba7013b1d0 2020-11-20T12:13:23.707 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:13:23.708 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5389,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/636533719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":636533719},{"type":"v1","addr":"172.21.15.189:6835","nonce":636533719}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:13:20.671656+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":12,"addr":"172.21.15.47:6833/1811549340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1811549340},{"type":"v1","addr":"172.21.15.47:6833","nonce":1811549340}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:13:23.709 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.701+0000 7fba74b14700 1 -- 172.21.15.189:0/879374877 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fba480557c0 msgr2=0x7fba48057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:23.710 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.701+0000 7fba74b14700 1 --2- 172.21.15.189:0/879374877 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fba480557c0 0x7fba48057c80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fba600011a0 tx=0x7fba600079e0).stop 2020-11-20T12:13:23.710 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.701+0000 7fba74b14700 1 -- 172.21.15.189:0/879374877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba7013b1d0 msgr2=0x7fba700a0a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:13:23.710 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.701+0000 7fba74b14700 1 --2- 172.21.15.189:0/879374877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba7013b1d0 0x7fba700a0a90 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fba68009210 tx=0x7fba68016000).stop 2020-11-20T12:13:23.710 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.701+0000 7fba74b14700 1 -- 172.21.15.189:0/879374877 shutdown_connections 2020-11-20T12:13:23.710 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.701+0000 7fba74b14700 1 --2- 172.21.15.189:0/879374877 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fba480557c0 0x7fba48057c80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:23.711 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.701+0000 7fba74b14700 1 --2- 172.21.15.189:0/879374877 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fba700623c0 0x7fba7014a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:23.711 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.701+0000 7fba74b14700 1 --2- 172.21.15.189:0/879374877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba7013b1d0 0x7fba700a0a90 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:23.711 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.701+0000 7fba74b14700 1 --2- 172.21.15.189:0/879374877 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fba701340e0 0x7fba7013c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:13:23.711 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.701+0000 7fba74b14700 1 -- 172.21.15.189:0/879374877 >> 172.21.15.189:0/879374877 conn(0x7fba70065fe0 msgr2=0x7fba701344c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:13:23.711 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.701+0000 7fba74b14700 1 -- 172.21.15.189:0/879374877 shutdown_connections 2020-11-20T12:13:23.712 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:13:23.701+0000 7fba74b14700 1 -- 172.21.15.189:0/879374877 wait complete. 2020-11-20T12:13:23.712 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 41 2020-11-20T12:13:23.726 INFO:tasks.mds_thrash.fs.[None]:mds.a reported in up:standby state 2020-11-20T12:13:23.726 INFO:tasks.mds_thrash.fs.[None]:waiting for 104 secs before thrashing 2020-11-20T12:13:23.970 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:13:23.973 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:13:54.015 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:13:54.019 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:14:24.121 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:14:24.523 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:14:54.581 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:14:54.584 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:15:07.727 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:07.870 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.862+0000 7fec3d9a9700 1 Processor -- start 2020-11-20T12:15:07.871 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.862+0000 7fec3d9a9700 1 -- start start 2020-11-20T12:15:07.874 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec3d9a9700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fec3813b1d0 0x7fec380677f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:07.874 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec3d9a9700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec380a9c60 con 0x7fec381340e0 2020-11-20T12:15:07.875 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec3d9a9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec38067e90 con 0x7fec380623c0 2020-11-20T12:15:07.875 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec3d9a9700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec380a8e60 con 0x7fec3813b1d0 2020-11-20T12:15:07.875 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec3c9a7700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fec380623c0 0x7fec3813aac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52422/0 (socket says 172.21.15.189:52422) 2020-11-20T12:15:07.875 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec3c9a7700 1 -- 172.21.15.189:0/3475482864 learned_addr learned my addr 172.21.15.189:0/3475482864 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:15:07.876 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec37fff700 1 --2- 172.21.15.189:0/3475482864 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fec3813b1d0 0x7fec380677f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:07.876 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec377fe700 1 -- 172.21.15.189:0/3475482864 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4148618902 0 0) 0x7fec38067e90 con 0x7fec380623c0 2020-11-20T12:15:07.876 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec377fe700 1 -- 172.21.15.189:0/3475482864 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec1c003240 con 0x7fec380623c0 2020-11-20T12:15:07.876 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec377fe700 1 -- 172.21.15.189:0/3475482864 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842250169 0 0) 0x7fec380a9c60 con 0x7fec381340e0 2020-11-20T12:15:07.877 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec377fe700 1 -- 172.21.15.189:0/3475482864 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec38067e90 con 0x7fec381340e0 2020-11-20T12:15:07.877 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec377fe700 1 -- 172.21.15.189:0/3475482864 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2010416770 0 0) 0x7fec1c003240 con 0x7fec380623c0 2020-11-20T12:15:07.877 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec377fe700 1 -- 172.21.15.189:0/3475482864 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fec3813b1d0 msgr2=0x7fec380677f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:07.877 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec377fe700 1 --2- 172.21.15.189:0/3475482864 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fec3813b1d0 0x7fec380677f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:07.877 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec377fe700 1 -- 172.21.15.189:0/3475482864 >> v1:172.21.15.47:6789/0 conn(0x7fec381340e0 legacy=0x7fec38134570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:07.878 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec377fe700 1 -- 172.21.15.189:0/3475482864 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec38068130 con 0x7fec380623c0 2020-11-20T12:15:07.878 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec377fe700 1 -- 172.21.15.189:0/3475482864 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7fec240014e0 con 0x7fec380623c0 2020-11-20T12:15:07.878 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec377fe700 1 -- 172.21.15.189:0/3475482864 >> v1:172.21.15.189:6789/0 conn(0x7fec380623c0 legacy=0x7fec3813aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:07.878 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec377fe700 1 --2- 172.21.15.189:0/3475482864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec1c0038c0 0x7fec1c003cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:07.878 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec377fe700 1 --2- 172.21.15.189:0/3475482864 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fec3813b1d0 0x7fec1c0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:07.879 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec377fe700 1 --2- 172.21.15.189:0/3475482864 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec1c0049d0 0x7fec1c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:07.879 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec377fe700 1 -- 172.21.15.189:0/3475482864 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1c005340 con 0x7fec1c0049d0 2020-11-20T12:15:07.879 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec377fe700 1 -- 172.21.15.189:0/3475482864 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1c00d940 con 0x7fec1c0038c0 2020-11-20T12:15:07.879 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec377fe700 1 -- 172.21.15.189:0/3475482864 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec1c00da80 con 0x7fec3813b1d0 2020-11-20T12:15:07.879 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec3c9a7700 1 --2- 172.21.15.189:0/3475482864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec1c0038c0 0x7fec1c003cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:07.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec3d1a8700 1 --2- 172.21.15.189:0/3475482864 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec1c0049d0 0x7fec1c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:07.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec3c9a7700 1 -- 172.21.15.189:0/3475482864 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fec3813b1d0 msgr2=0x7fec1c0044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:07.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec3c9a7700 1 --2- 172.21.15.189:0/3475482864 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fec3813b1d0 0x7fec1c0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:07.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec3c9a7700 1 -- 172.21.15.189:0/3475482864 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec1c0049d0 msgr2=0x7fec1c004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:07.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec3c9a7700 1 --2- 172.21.15.189:0/3475482864 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec1c0049d0 0x7fec1c004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:07.881 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec3c9a7700 1 -- 172.21.15.189:0/3475482864 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec38068130 con 0x7fec1c0038c0 2020-11-20T12:15:07.881 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec3c9a7700 1 --2- 172.21.15.189:0/3475482864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec1c0038c0 0x7fec1c003cc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fec2400c470 tx=0x7fec240077d0).ready entity=mon.1 client_cookie=d7f620785fc67a58 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:07.884 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec377fe700 1 -- 172.21.15.189:0/3475482864 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2400ce60 con 0x7fec1c0038c0 2020-11-20T12:15:07.884 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec377fe700 1 -- 172.21.15.189:0/3475482864 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec240072e0 con 0x7fec1c0038c0 2020-11-20T12:15:07.885 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec377fe700 1 -- 172.21.15.189:0/3475482864 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec24005820 con 0x7fec1c0038c0 2020-11-20T12:15:07.885 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec3d9a9700 1 -- 172.21.15.189:0/3475482864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec1c0038c0 msgr2=0x7fec1c003cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:07.885 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec3d9a9700 1 --2- 172.21.15.189:0/3475482864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec1c0038c0 0x7fec1c003cc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fec2400c470 tx=0x7fec240077d0).stop 2020-11-20T12:15:07.885 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.866+0000 7fec3d1a8700 1 -- 172.21.15.189:0/3475482864 reap_dead start 2020-11-20T12:15:07.885 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.870+0000 7fec3d9a9700 1 -- 172.21.15.189:0/3475482864 shutdown_connections 2020-11-20T12:15:07.886 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.870+0000 7fec3d9a9700 1 -- 172.21.15.189:0/3475482864 >> 172.21.15.189:0/3475482864 conn(0x7fec38065fe0 msgr2=0x7fec38061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:07.886 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.870+0000 7fec3d9a9700 1 -- 172.21.15.189:0/3475482864 shutdown_connections 2020-11-20T12:15:07.886 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.870+0000 7fec3d9a9700 1 -- 172.21.15.189:0/3475482864 wait complete. 2020-11-20T12:15:07.886 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.870+0000 7fec3d9a9700 1 Processor -- start 2020-11-20T12:15:07.886 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.870+0000 7fec3d9a9700 1 -- start start 2020-11-20T12:15:07.887 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.870+0000 7fec3d9a9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec380a0a90 0x7fec380ae1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:07.887 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.870+0000 7fec3d9a9700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fec38139050 0x7fec3813c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:07.887 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.870+0000 7fec3d9a9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec3813ccc0 0x7fec3813d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:07.887 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.870+0000 7fec3d9a9700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec380adbf0 con 0x7fec380a0a90 2020-11-20T12:15:07.888 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.870+0000 7fec3d9a9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3813a530 con 0x7fec3813ccc0 2020-11-20T12:15:07.888 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.870+0000 7fec3d9a9700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec3813d610 con 0x7fec38139050 2020-11-20T12:15:07.888 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.870+0000 7fec3d1a8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec3813ccc0 0x7fec3813d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:07.888 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.870+0000 7fec3c9a7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec380a0a90 0x7fec380ae1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:07.889 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.870+0000 7fec3d1a8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec3813ccc0 0x7fec3813d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50816/0 (socket says 172.21.15.189:50816) 2020-11-20T12:15:07.889 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.870+0000 7fec3d1a8700 1 -- 172.21.15.189:0/4228869654 learned_addr learned my addr 172.21.15.189:0/4228869654 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:15:07.889 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.870+0000 7fec37fff700 1 --2- 172.21.15.189:0/4228869654 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fec38139050 0x7fec3813c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:07.889 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.874+0000 7fec3d1a8700 1 -- 172.21.15.189:0/4228869654 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fec38139050 msgr2=0x7fec3813c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:07.890 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.874+0000 7fec3d1a8700 1 --2- 172.21.15.189:0/4228869654 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fec38139050 0x7fec3813c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:07.890 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.874+0000 7fec3d1a8700 1 -- 172.21.15.189:0/4228869654 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec380a0a90 msgr2=0x7fec380ae1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:07.890 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.874+0000 7fec3d1a8700 1 --2- 172.21.15.189:0/4228869654 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec380a0a90 0x7fec380ae1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:07.890 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.874+0000 7fec3d1a8700 1 -- 172.21.15.189:0/4228869654 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec380ae6c0 con 0x7fec3813ccc0 2020-11-20T12:15:07.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.874+0000 7fec3c9a7700 1 --2- 172.21.15.189:0/4228869654 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec380a0a90 0x7fec380ae1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:15:07.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.874+0000 7fec3d1a8700 1 --2- 172.21.15.189:0/4228869654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec3813ccc0 0x7fec3813d120 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fec380aa090 tx=0x7fec30009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:07.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.874+0000 7fec34ff9700 1 -- 172.21.15.189:0/4228869654 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec30003320 con 0x7fec3813ccc0 2020-11-20T12:15:07.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.874+0000 7fec34ff9700 1 -- 172.21.15.189:0/4228869654 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec30003480 con 0x7fec3813ccc0 2020-11-20T12:15:07.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.874+0000 7fec3d9a9700 1 -- 172.21.15.189:0/4228869654 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec380ade20 con 0x7fec3813ccc0 2020-11-20T12:15:07.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.874+0000 7fec34ff9700 1 -- 172.21.15.189:0/4228869654 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec300018b0 con 0x7fec3813ccc0 2020-11-20T12:15:07.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.874+0000 7fec3d9a9700 1 -- 172.21.15.189:0/4228869654 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec380a0e70 con 0x7fec3813ccc0 2020-11-20T12:15:07.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.874+0000 7fec3d9a9700 1 -- 172.21.15.189:0/4228869654 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec380663c0 con 0x7fec3813ccc0 2020-11-20T12:15:07.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.874+0000 7fec34ff9700 1 -- 172.21.15.189:0/4228869654 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fec30001c60 con 0x7fec3813ccc0 2020-11-20T12:15:07.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.874+0000 7fec34ff9700 1 --2- 172.21.15.189:0/4228869654 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fec100557c0 0x7fec10057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:07.893 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.874+0000 7fec34ff9700 1 -- 172.21.15.189:0/4228869654 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fec3006e420 con 0x7fec3813ccc0 2020-11-20T12:15:07.893 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.874+0000 7fec3c9a7700 1 --2- 172.21.15.189:0/4228869654 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fec100557c0 0x7fec10057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:07.893 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.878+0000 7fec3c9a7700 1 --2- 172.21.15.189:0/4228869654 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fec100557c0 0x7fec10057c80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fec24006000 tx=0x7fec24015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:07.894 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:07.882+0000 7fec34ff9700 1 -- 172.21.15.189:0/4228869654 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fec30044060 con 0x7fec3813ccc0 2020-11-20T12:15:08.107 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.102+0000 7fec3d9a9700 1 -- 172.21.15.189:0/4228869654 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec380446d0 con 0x7fec3813ccc0 2020-11-20T12:15:08.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.102+0000 7fec34ff9700 1 -- 172.21.15.189:0/4228869654 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2137 (secure 0 0 0) 0x7fec30047680 con 0x7fec3813ccc0 2020-11-20T12:15:08.108 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:15:08.109 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5389,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/636533719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":636533719},{"type":"v1","addr":"172.21.15.189:6835","nonce":636533719}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:13:20.671656+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":12,"addr":"172.21.15.47:6833/1811549340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1811549340},{"type":"v1","addr":"172.21.15.47:6833","nonce":1811549340}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:15:08.110 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.102+0000 7fec3d9a9700 1 -- 172.21.15.189:0/4228869654 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fec100557c0 msgr2=0x7fec10057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:08.110 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.102+0000 7fec3d9a9700 1 --2- 172.21.15.189:0/4228869654 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fec100557c0 0x7fec10057c80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fec24006000 tx=0x7fec24015000).stop 2020-11-20T12:15:08.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.102+0000 7fec3d9a9700 1 -- 172.21.15.189:0/4228869654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec3813ccc0 msgr2=0x7fec3813d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:08.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.102+0000 7fec3d9a9700 1 --2- 172.21.15.189:0/4228869654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec3813ccc0 0x7fec3813d120 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fec380aa090 tx=0x7fec30009040).stop 2020-11-20T12:15:08.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.102+0000 7fec3d9a9700 1 -- 172.21.15.189:0/4228869654 shutdown_connections 2020-11-20T12:15:08.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.102+0000 7fec3d9a9700 1 --2- 172.21.15.189:0/4228869654 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fec100557c0 0x7fec10057c80 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:08.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.102+0000 7fec3d9a9700 1 --2- 172.21.15.189:0/4228869654 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec380a0a90 0x7fec380ae1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:08.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.102+0000 7fec3d9a9700 1 --2- 172.21.15.189:0/4228869654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec3813ccc0 0x7fec3813d120 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:08.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.102+0000 7fec3d9a9700 1 --2- 172.21.15.189:0/4228869654 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fec38139050 0x7fec3813c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:08.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.102+0000 7fec3d9a9700 1 -- 172.21.15.189:0/4228869654 >> 172.21.15.189:0/4228869654 conn(0x7fec38065fe0 msgr2=0x7fec380675f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:08.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.102+0000 7fec3d9a9700 1 -- 172.21.15.189:0/4228869654 shutdown_connections 2020-11-20T12:15:08.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.102+0000 7fec3d9a9700 1 -- 172.21.15.189:0/4228869654 wait complete. 2020-11-20T12:15:08.113 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 41 2020-11-20T12:15:08.123 INFO:tasks.mds_thrash.fs.[None]:kill mds.b (rank=0) 2020-11-20T12:15:08.124 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:15:08.124 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:15:08.139 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:15:08.140 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:08.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.302+0000 7fc4bffff700 1 Processor -- start 2020-11-20T12:15:08.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.302+0000 7fc4bffff700 1 -- start start 2020-11-20T12:15:08.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.306+0000 7fc4bffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc4c00a7d40 0x7fc4c00a81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:08.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.306+0000 7fc4bffff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc4c00a9cc0 0x7fc4c00aa140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:08.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.306+0000 7fc4bffff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c00a8fa0 con 0x7fc4c0060310 2020-11-20T12:15:08.315 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.306+0000 7fc4bffff700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4c00618b0 con 0x7fc4c00a7d40 2020-11-20T12:15:08.315 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.306+0000 7fc4bffff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c00a9240 con 0x7fc4c00a9cc0 2020-11-20T12:15:08.315 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.306+0000 7fc4be7fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc4c00a7d40 0x7fc4c00a81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:08.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.306+0000 7fc4be7fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc4c00a7d40 0x7fc4c00a81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50824/0 (socket says 172.21.15.189:50824) 2020-11-20T12:15:08.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.306+0000 7fc4be7fc700 1 -- 172.21.15.189:0/1229054430 learned_addr learned my addr 172.21.15.189:0/1229054430 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:15:08.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.306+0000 7fc4bf7fe700 1 --2- 172.21.15.189:0/1229054430 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc4c00a9cc0 0x7fc4c00aa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:08.317 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.306+0000 7fc4be7fc700 1 -- 172.21.15.189:0/1229054430 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc4c00a9cc0 msgr2=0x7fc4c00aa140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:08.317 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.306+0000 7fc4be7fc700 1 --2- 172.21.15.189:0/1229054430 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc4c00a9cc0 0x7fc4c00aa140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:08.317 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.306+0000 7fc4be7fc700 1 -- 172.21.15.189:0/1229054430 >> v1:172.21.15.47:6789/0 conn(0x7fc4c0060310 legacy=0x7fc4c00a1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:08.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.306+0000 7fc4be7fc700 1 -- 172.21.15.189:0/1229054430 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c00a94e0 con 0x7fc4c00a7d40 2020-11-20T12:15:08.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.306+0000 7fc4be7fc700 1 --2- 172.21.15.189:0/1229054430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4c00a7d40 0x7fc4c00a81b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fc4b0006fd0 tx=0x7fc4b0006eb0).ready entity=mon.1 client_cookie=ef2e3cdc4c8634b7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:08.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.306+0000 7fc4bdffb700 1 -- 172.21.15.189:0/1229054430 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b0009960 con 0x7fc4c00a7d40 2020-11-20T12:15:08.319 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.306+0000 7fc4bdffb700 1 -- 172.21.15.189:0/1229054430 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b0009ac0 con 0x7fc4c00a7d40 2020-11-20T12:15:08.319 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.306+0000 7fc4bdffb700 1 -- 172.21.15.189:0/1229054430 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b0009e40 con 0x7fc4c00a7d40 2020-11-20T12:15:08.323 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.306+0000 7fc4bffff700 1 -- 172.21.15.189:0/1229054430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4c00a7d40 msgr2=0x7fc4c00a81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:08.323 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.306+0000 7fc4bffff700 1 --2- 172.21.15.189:0/1229054430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4c00a7d40 0x7fc4c00a81b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fc4b0006fd0 tx=0x7fc4b0006eb0).stop 2020-11-20T12:15:08.324 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.306+0000 7fc4bffff700 1 -- 172.21.15.189:0/1229054430 shutdown_connections 2020-11-20T12:15:08.324 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.306+0000 7fc4bffff700 1 --2- 172.21.15.189:0/1229054430 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc4c00a9cc0 0x7fc4c00aa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:08.324 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.306+0000 7fc4bffff700 1 --2- 172.21.15.189:0/1229054430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4c00a7d40 0x7fc4c00a81b0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:08.325 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.306+0000 7fc4bffff700 1 -- 172.21.15.189:0/1229054430 >> 172.21.15.189:0/1229054430 conn(0x7fc4c0063210 msgr2=0x7fc4c0060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:08.325 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.306+0000 7fc4bffff700 1 -- 172.21.15.189:0/1229054430 shutdown_connections 2020-11-20T12:15:08.326 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.306+0000 7fc4bffff700 1 -- 172.21.15.189:0/1229054430 wait complete. 2020-11-20T12:15:08.326 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.310+0000 7fc4bffff700 1 Processor -- start 2020-11-20T12:15:08.326 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.310+0000 7fc4bffff700 1 -- start start 2020-11-20T12:15:08.326 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.310+0000 7fc4bffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4c0060310 0x7fc4c0067c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:08.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.310+0000 7fc4bffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4c00a7d40 0x7fc4c013a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:08.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.310+0000 7fc4bffff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc4c00a9cc0 0x7fc4c0149b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:08.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.310+0000 7fc4bffff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c0067350 con 0x7fc4c00a7d40 2020-11-20T12:15:08.328 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.310+0000 7fc4bffff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c0066900 con 0x7fc4c0060310 2020-11-20T12:15:08.328 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.310+0000 7fc4bffff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c0066ba0 con 0x7fc4c00a9cc0 2020-11-20T12:15:08.328 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.310+0000 7fc4beffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4c0060310 0x7fc4c0067c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:08.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.310+0000 7fc4beffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4c0060310 0x7fc4c0067c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50828/0 (socket says 172.21.15.189:50828) 2020-11-20T12:15:08.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.310+0000 7fc4beffd700 1 -- 172.21.15.189:0/1712309620 learned_addr learned my addr 172.21.15.189:0/1712309620 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:15:08.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.310+0000 7fc4bf7fe700 1 --2- 172.21.15.189:0/1712309620 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc4c00a9cc0 0x7fc4c0149b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:08.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.310+0000 7fc4be7fc700 1 --2- 172.21.15.189:0/1712309620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4c00a7d40 0x7fc4c013a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:08.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.310+0000 7fc4beffd700 1 -- 172.21.15.189:0/1712309620 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc4c00a9cc0 msgr2=0x7fc4c0149b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:08.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.310+0000 7fc4beffd700 1 --2- 172.21.15.189:0/1712309620 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc4c00a9cc0 0x7fc4c0149b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:08.331 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.310+0000 7fc4beffd700 1 -- 172.21.15.189:0/1712309620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4c00a7d40 msgr2=0x7fc4c013a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:08.331 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.310+0000 7fc4beffd700 1 --2- 172.21.15.189:0/1712309620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4c00a7d40 0x7fc4c013a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:08.331 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.310+0000 7fc4beffd700 1 -- 172.21.15.189:0/1712309620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c0066e40 con 0x7fc4c0060310 2020-11-20T12:15:08.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.310+0000 7fc4beffd700 1 --2- 172.21.15.189:0/1712309620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4c0060310 0x7fc4c0067c80 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fc4b400c6e0 tx=0x7fc4b4007c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:08.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.310+0000 7fc4a6ffd700 1 -- 172.21.15.189:0/1712309620 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4007020 con 0x7fc4c0060310 2020-11-20T12:15:08.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.310+0000 7fc4a6ffd700 1 -- 172.21.15.189:0/1712309620 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b4007180 con 0x7fc4c0060310 2020-11-20T12:15:08.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.310+0000 7fc4bffff700 1 -- 172.21.15.189:0/1712309620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4c0066380 con 0x7fc4c0060310 2020-11-20T12:15:08.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.310+0000 7fc4a6ffd700 1 -- 172.21.15.189:0/1712309620 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b400e560 con 0x7fc4c0060310 2020-11-20T12:15:08.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.310+0000 7fc4bffff700 1 -- 172.21.15.189:0/1712309620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c00666d0 con 0x7fc4c0060310 2020-11-20T12:15:08.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.314+0000 7fc4bffff700 1 -- 172.21.15.189:0/1712309620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c0066fd0 con 0x7fc4c0060310 2020-11-20T12:15:08.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.314+0000 7fc4a6ffd700 1 -- 172.21.15.189:0/1712309620 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fc4b4010070 con 0x7fc4c0060310 2020-11-20T12:15:08.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.314+0000 7fc4a6ffd700 1 --2- 172.21.15.189:0/1712309620 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc49c0557c0 0x7fc49c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:08.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.314+0000 7fc4a6ffd700 1 -- 172.21.15.189:0/1712309620 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fc4b406dc70 con 0x7fc4c0060310 2020-11-20T12:15:08.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.314+0000 7fc4be7fc700 1 --2- 172.21.15.189:0/1712309620 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc49c0557c0 0x7fc49c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:08.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.314+0000 7fc4be7fc700 1 --2- 172.21.15.189:0/1712309620 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc49c0557c0 0x7fc49c057c80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fc4c00a0cb0 tx=0x7fc4b0015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:08.336 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.318+0000 7fc4a6ffd700 1 -- 172.21.15.189:0/1712309620 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc4b4042fd0 con 0x7fc4c0060310 2020-11-20T12:15:08.539 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.534+0000 7fc4bffff700 1 -- 172.21.15.189:0/1712309620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4c00446d0 con 0x7fc4c0060310 2020-11-20T12:15:08.540 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.534+0000 7fc4a6ffd700 1 -- 172.21.15.189:0/1712309620 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2137 (secure 0 0 0) 0x7fc4b40438b0 con 0x7fc4c0060310 2020-11-20T12:15:08.540 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:15:08.541 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5389,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/636533719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":636533719},{"type":"v1","addr":"172.21.15.189:6835","nonce":636533719}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:13:20.671656+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":12,"addr":"172.21.15.47:6833/1811549340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1811549340},{"type":"v1","addr":"172.21.15.47:6833","nonce":1811549340}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:15:08.542 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.534+0000 7fc4bffff700 1 -- 172.21.15.189:0/1712309620 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc49c0557c0 msgr2=0x7fc49c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:08.542 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.534+0000 7fc4bffff700 1 --2- 172.21.15.189:0/1712309620 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc49c0557c0 0x7fc49c057c80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fc4c00a0cb0 tx=0x7fc4b0015000).stop 2020-11-20T12:15:08.542 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.534+0000 7fc4bffff700 1 -- 172.21.15.189:0/1712309620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4c0060310 msgr2=0x7fc4c0067c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:08.543 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.534+0000 7fc4bffff700 1 --2- 172.21.15.189:0/1712309620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4c0060310 0x7fc4c0067c80 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fc4b400c6e0 tx=0x7fc4b4007c50).stop 2020-11-20T12:15:08.543 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.534+0000 7fc4bffff700 1 -- 172.21.15.189:0/1712309620 shutdown_connections 2020-11-20T12:15:08.543 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.534+0000 7fc4bffff700 1 --2- 172.21.15.189:0/1712309620 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc4c00a9cc0 0x7fc4c0149b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:08.543 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.534+0000 7fc4bffff700 1 --2- 172.21.15.189:0/1712309620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4c0060310 0x7fc4c0067c80 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:08.543 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.534+0000 7fc4bffff700 1 --2- 172.21.15.189:0/1712309620 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc49c0557c0 0x7fc49c057c80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:08.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.534+0000 7fc4bffff700 1 --2- 172.21.15.189:0/1712309620 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc4c00a7d40 0x7fc4c013a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:08.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.534+0000 7fc4bffff700 1 -- 172.21.15.189:0/1712309620 >> 172.21.15.189:0/1712309620 conn(0x7fc4c0063210 msgr2=0x7fc4c00acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:08.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.534+0000 7fc4bffff700 1 -- 172.21.15.189:0/1712309620 shutdown_connections 2020-11-20T12:15:08.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:08.534+0000 7fc4bffff700 1 -- 172.21.15.189:0/1712309620 wait complete. 2020-11-20T12:15:08.546 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 41 2020-11-20T12:15:08.558 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:15:10.561 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:10.693 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.686+0000 7fe0a69ff700 1 Processor -- start 2020-11-20T12:15:10.694 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.686+0000 7fe0a69ff700 1 -- start start 2020-11-20T12:15:10.698 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.686+0000 7fe0a69ff700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe0a00a8840 0x7fe0a013eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:10.698 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.686+0000 7fe0a69ff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a0142eb0 con 0x7fe0a0066ed0 2020-11-20T12:15:10.698 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.686+0000 7fe0a69ff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a0066800 con 0x7fe0a013f030 2020-11-20T12:15:10.699 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.686+0000 7fe0a69ff700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0a013f4a0 con 0x7fe0a00a8840 2020-11-20T12:15:10.699 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.686+0000 7fe0a61fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe0a013f030 0x7fe0a01427a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52464/0 (socket says 172.21.15.189:52464) 2020-11-20T12:15:10.699 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.686+0000 7fe0a61fe700 1 -- 172.21.15.189:0/3246688296 learned_addr learned my addr 172.21.15.189:0/3246688296 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:15:10.699 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.686+0000 7fe0a59fd700 1 --2- 172.21.15.189:0/3246688296 >> v2:172.21.15.47:3300/0 conn(0x7fe0a00a8840 0x7fe0a013eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:10.700 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.686+0000 7fe0a49fb700 1 -- 172.21.15.189:0/3246688296 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914606202 0 0) 0x7fe0a0066800 con 0x7fe0a013f030 2020-11-20T12:15:10.700 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.686+0000 7fe0a49fb700 1 -- 172.21.15.189:0/3246688296 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe088003240 con 0x7fe0a013f030 2020-11-20T12:15:10.700 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.686+0000 7fe0a49fb700 1 -- 172.21.15.189:0/3246688296 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3585450843 0 0) 0x7fe088003240 con 0x7fe0a013f030 2020-11-20T12:15:10.700 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.686+0000 7fe0a49fb700 1 -- 172.21.15.189:0/3246688296 >> v2:172.21.15.47:3300/0 conn(0x7fe0a00a8840 msgr2=0x7fe0a013eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:10.701 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.686+0000 7fe0a49fb700 1 --2- 172.21.15.189:0/3246688296 >> v2:172.21.15.47:3300/0 conn(0x7fe0a00a8840 0x7fe0a013eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.701 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.686+0000 7fe0a49fb700 1 -- 172.21.15.189:0/3246688296 >> v1:172.21.15.47:6789/0 conn(0x7fe0a0066ed0 legacy=0x7fe0a0142090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:10.701 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.686+0000 7fe0a49fb700 1 -- 172.21.15.189:0/3246688296 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a0066aa0 con 0x7fe0a013f030 2020-11-20T12:15:10.702 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.686+0000 7fe0a49fb700 1 -- 172.21.15.189:0/3246688296 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7fe098001860 con 0x7fe0a013f030 2020-11-20T12:15:10.702 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.686+0000 7fe0a49fb700 1 -- 172.21.15.189:0/3246688296 >> v1:172.21.15.189:6789/0 conn(0x7fe0a013f030 legacy=0x7fe0a01427a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:10.702 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.686+0000 7fe0a49fb700 1 --2- 172.21.15.189:0/3246688296 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe088003730 0x7fe088003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:10.702 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.686+0000 7fe0a49fb700 1 --2- 172.21.15.189:0/3246688296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0880044e0 0x7fe088004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:10.703 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.686+0000 7fe0a49fb700 1 --2- 172.21.15.189:0/3246688296 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe088004e50 0x7fe08800b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:10.703 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.686+0000 7fe0a49fb700 1 -- 172.21.15.189:0/3246688296 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a013f4a0 con 0x7fe088003730 2020-11-20T12:15:10.703 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.686+0000 7fe0a49fb700 1 -- 172.21.15.189:0/3246688296 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe088004260 con 0x7fe0880044e0 2020-11-20T12:15:10.703 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.690+0000 7fe0a49fb700 1 -- 172.21.15.189:0/3246688296 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe08800bc20 con 0x7fe088004e50 2020-11-20T12:15:10.704 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.690+0000 7fe0a51fc700 1 --2- 172.21.15.189:0/3246688296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0880044e0 0x7fe088004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:10.704 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.690+0000 7fe0a61fe700 1 --2- 172.21.15.189:0/3246688296 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe088004e50 0x7fe08800b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:10.704 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.690+0000 7fe0a51fc700 1 -- 172.21.15.189:0/3246688296 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe088004e50 msgr2=0x7fe08800b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:10.704 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.690+0000 7fe0a51fc700 1 --2- 172.21.15.189:0/3246688296 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe088004e50 0x7fe08800b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.704 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.690+0000 7fe0a51fc700 1 -- 172.21.15.189:0/3246688296 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe088003730 msgr2=0x7fe088003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:10.705 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.690+0000 7fe0a51fc700 1 --2- 172.21.15.189:0/3246688296 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe088003730 0x7fe088003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:15:10.705 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.690+0000 7fe0a51fc700 1 -- 172.21.15.189:0/3246688296 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe088003240 con 0x7fe0880044e0 2020-11-20T12:15:10.705 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.690+0000 7fe0a61fe700 1 -- 172.21.15.189:0/3246688296 reap_dead start 2020-11-20T12:15:10.705 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.690+0000 7fe0a51fc700 1 --2- 172.21.15.189:0/3246688296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0880044e0 0x7fe088004960 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fe094001920 tx=0x7fe094007b90).ready entity=mon.1 client_cookie=67629d02d8db5c83 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:10.705 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.690+0000 7fe0a49fb700 1 -- 172.21.15.189:0/3246688296 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe094007020 con 0x7fe0880044e0 2020-11-20T12:15:10.706 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.690+0000 7fe0a49fb700 1 -- 172.21.15.189:0/3246688296 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe094007180 con 0x7fe0880044e0 2020-11-20T12:15:10.706 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.690+0000 7fe0a49fb700 1 -- 172.21.15.189:0/3246688296 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09400c620 con 0x7fe0880044e0 2020-11-20T12:15:10.706 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.690+0000 7fe0a69ff700 1 -- 172.21.15.189:0/3246688296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0880044e0 msgr2=0x7fe088004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:10.706 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.690+0000 7fe0a69ff700 1 --2- 172.21.15.189:0/3246688296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0880044e0 0x7fe088004960 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fe094001920 tx=0x7fe094007b90).stop 2020-11-20T12:15:10.707 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.690+0000 7fe0a69ff700 1 -- 172.21.15.189:0/3246688296 shutdown_connections 2020-11-20T12:15:10.707 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.690+0000 7fe0a69ff700 1 --2- 172.21.15.189:0/3246688296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0880044e0 0x7fe088004960 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.707 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.690+0000 7fe0a69ff700 1 -- 172.21.15.189:0/3246688296 >> 172.21.15.189:0/3246688296 conn(0x7fe0a0060170 msgr2=0x7fe0a0060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:10.707 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.690+0000 7fe0a69ff700 1 -- 172.21.15.189:0/3246688296 shutdown_connections 2020-11-20T12:15:10.708 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.690+0000 7fe0a69ff700 1 -- 172.21.15.189:0/3246688296 wait complete. 2020-11-20T12:15:10.711 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.694+0000 7fe0a69ff700 1 Processor -- start 2020-11-20T12:15:10.711 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.694+0000 7fe0a69ff700 1 -- start start 2020-11-20T12:15:10.712 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.694+0000 7fe0a69ff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0880044e0 0x7fe0a00a60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:10.712 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.694+0000 7fe0a69ff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe0a0114690 0x7fe0a00a21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:10.712 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.694+0000 7fe0a69ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0a00a3220 0x7fe0a00a26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:10.712 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.694+0000 7fe0a69ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a00a39a0 con 0x7fe0a00a3220 2020-11-20T12:15:10.713 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.694+0000 7fe0a69ff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a00a2bc0 con 0x7fe0880044e0 2020-11-20T12:15:10.713 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.694+0000 7fe0a69ff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a01163b0 con 0x7fe0a0114690 2020-11-20T12:15:10.713 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.694+0000 7fe0a59fd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0880044e0 0x7fe0a00a60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:10.713 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.694+0000 7fe0a59fd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0880044e0 0x7fe0a00a60a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50854/0 (socket says 172.21.15.189:50854) 2020-11-20T12:15:10.713 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.694+0000 7fe0a59fd700 1 -- 172.21.15.189:0/3736008 learned_addr learned my addr 172.21.15.189:0/3736008 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:15:10.714 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.694+0000 7fe0a61fe700 1 --2- 172.21.15.189:0/3736008 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0a00a3220 0x7fe0a00a26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:10.714 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.694+0000 7fe0a51fc700 1 --2- 172.21.15.189:0/3736008 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe0a0114690 0x7fe0a00a21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:10.714 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.694+0000 7fe0a59fd700 1 -- 172.21.15.189:0/3736008 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe0a0114690 msgr2=0x7fe0a00a21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:10.714 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.694+0000 7fe0a59fd700 1 --2- 172.21.15.189:0/3736008 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe0a0114690 0x7fe0a00a21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.714 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.694+0000 7fe0a59fd700 1 -- 172.21.15.189:0/3736008 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0a00a3220 msgr2=0x7fe0a00a26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:10.715 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.694+0000 7fe0a59fd700 1 --2- 172.21.15.189:0/3736008 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0a00a3220 0x7fe0a00a26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.715 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.694+0000 7fe0a59fd700 1 -- 172.21.15.189:0/3736008 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a00a3700 con 0x7fe0880044e0 2020-11-20T12:15:10.715 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.694+0000 7fe0a61fe700 1 --2- 172.21.15.189:0/3736008 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0a00a3220 0x7fe0a00a26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:15:10.715 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.694+0000 7fe0a59fd700 1 --2- 172.21.15.189:0/3736008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0880044e0 0x7fe0a00a60a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fe09c0067d0 tx=0x7fe09c006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:10.716 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.694+0000 7fe091ffb700 1 -- 172.21.15.189:0/3736008 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c016070 con 0x7fe0880044e0 2020-11-20T12:15:10.716 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.694+0000 7fe091ffb700 1 -- 172.21.15.189:0/3736008 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09c00c740 con 0x7fe0880044e0 2020-11-20T12:15:10.716 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.694+0000 7fe0a69ff700 1 -- 172.21.15.189:0/3736008 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a00a6c70 con 0x7fe0880044e0 2020-11-20T12:15:10.716 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.694+0000 7fe091ffb700 1 -- 172.21.15.189:0/3736008 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c00cae0 con 0x7fe0880044e0 2020-11-20T12:15:10.716 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.698+0000 7fe0a69ff700 1 -- 172.21.15.189:0/3736008 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0a0151980 con 0x7fe0880044e0 2020-11-20T12:15:10.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.698+0000 7fe091ffb700 1 -- 172.21.15.189:0/3736008 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fe09c01b410 con 0x7fe0880044e0 2020-11-20T12:15:10.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.698+0000 7fe0a69ff700 1 -- 172.21.15.189:0/3736008 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0a00a6590 con 0x7fe0880044e0 2020-11-20T12:15:10.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.698+0000 7fe091ffb700 1 --2- 172.21.15.189:0/3736008 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe0780557c0 0x7fe078057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:10.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.698+0000 7fe091ffb700 1 -- 172.21.15.189:0/3736008 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fe09c06e7d0 con 0x7fe0880044e0 2020-11-20T12:15:10.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.698+0000 7fe0a51fc700 1 --2- 172.21.15.189:0/3736008 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe0780557c0 0x7fe078057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:10.718 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.698+0000 7fe0a51fc700 1 --2- 172.21.15.189:0/3736008 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe0780557c0 0x7fe078057c80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fe094008fd0 tx=0x7fe094015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:10.718 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.706+0000 7fe091ffb700 1 -- 172.21.15.189:0/3736008 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe09c0443b0 con 0x7fe0880044e0 2020-11-20T12:15:10.912 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.906+0000 7fe0a69ff700 1 -- 172.21.15.189:0/3736008 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0a0060fb0 con 0x7fe0880044e0 2020-11-20T12:15:10.913 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.906+0000 7fe091ffb700 1 -- 172.21.15.189:0/3736008 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2137 (secure 0 0 0) 0x7fe0a0060fb0 con 0x7fe0880044e0 2020-11-20T12:15:10.913 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:15:10.913 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5389,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/636533719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":636533719},{"type":"v1","addr":"172.21.15.189:6835","nonce":636533719}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:13:20.671656+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":12,"addr":"172.21.15.47:6833/1811549340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1811549340},{"type":"v1","addr":"172.21.15.47:6833","nonce":1811549340}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:15:10.914 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.906+0000 7fe0a69ff700 1 -- 172.21.15.189:0/3736008 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe0780557c0 msgr2=0x7fe078057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:10.914 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.906+0000 7fe0a69ff700 1 --2- 172.21.15.189:0/3736008 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe0780557c0 0x7fe078057c80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fe094008fd0 tx=0x7fe094015000).stop 2020-11-20T12:15:10.915 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.906+0000 7fe0a69ff700 1 -- 172.21.15.189:0/3736008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0880044e0 msgr2=0x7fe0a00a60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:10.915 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.906+0000 7fe0a69ff700 1 --2- 172.21.15.189:0/3736008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0880044e0 0x7fe0a00a60a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fe09c0067d0 tx=0x7fe09c006df0).stop 2020-11-20T12:15:10.915 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.906+0000 7fe0a69ff700 1 -- 172.21.15.189:0/3736008 shutdown_connections 2020-11-20T12:15:10.915 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.906+0000 7fe0a69ff700 1 --2- 172.21.15.189:0/3736008 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe0780557c0 0x7fe078057c80 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.916 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.906+0000 7fe0a69ff700 1 --2- 172.21.15.189:0/3736008 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0a00a3220 0x7fe0a00a26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.916 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.906+0000 7fe0a69ff700 1 --2- 172.21.15.189:0/3736008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0880044e0 0x7fe0a00a60a0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.916 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.906+0000 7fe0a69ff700 1 --2- 172.21.15.189:0/3736008 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe0a0114690 0x7fe0a00a21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:10.916 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.906+0000 7fe0a69ff700 1 -- 172.21.15.189:0/3736008 >> 172.21.15.189:0/3736008 conn(0x7fe0a0060170 msgr2=0x7fe0a005ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:10.916 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.906+0000 7fe0a69ff700 1 -- 172.21.15.189:0/3736008 shutdown_connections 2020-11-20T12:15:10.917 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:10.906+0000 7fe0a69ff700 1 -- 172.21.15.189:0/3736008 wait complete. 2020-11-20T12:15:10.917 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 41 2020-11-20T12:15:10.927 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:15:12.929 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:13.062 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.054+0000 7f9fd8a2d700 1 Processor -- start 2020-11-20T12:15:13.062 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.054+0000 7f9fd8a2d700 1 -- start start 2020-11-20T12:15:13.063 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.054+0000 7f9fd8a2d700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f9fd40ad000 0x7f9fd413e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:13.064 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.054+0000 7f9fd8a2d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9fd413a6e0 0x7f9fd413ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:13.064 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.054+0000 7f9fd8a2d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fd4067730 con 0x7f9fd40ad3e0 2020-11-20T12:15:13.064 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.054+0000 7f9fd8a2d700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fd4138190 con 0x7f9fd40ad000 2020-11-20T12:15:13.065 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.054+0000 7f9fd8a2d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fd413dd40 con 0x7f9fd413a6e0 2020-11-20T12:15:13.065 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.054+0000 7f9fd359e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9fd413a6e0 0x7f9fd413ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:13.065 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.054+0000 7f9fd359e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9fd413a6e0 0x7f9fd413ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50868/0 (socket says 172.21.15.189:50868) 2020-11-20T12:15:13.066 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.054+0000 7f9fd359e700 1 -- 172.21.15.189:0/2952646422 learned_addr learned my addr 172.21.15.189:0/2952646422 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:15:13.066 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.054+0000 7f9fd2d9d700 1 --2- 172.21.15.189:0/2952646422 >> v2:172.21.15.47:3300/0 conn(0x7f9fd40ad000 0x7f9fd413e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:13.066 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.054+0000 7f9fd359e700 1 -- 172.21.15.189:0/2952646422 >> v2:172.21.15.47:3300/0 conn(0x7f9fd40ad000 msgr2=0x7f9fd413e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:13.069 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.054+0000 7f9fd359e700 1 --2- 172.21.15.189:0/2952646422 >> v2:172.21.15.47:3300/0 conn(0x7f9fd40ad000 0x7f9fd413e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:13.070 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.054+0000 7f9fd359e700 1 -- 172.21.15.189:0/2952646422 >> v1:172.21.15.189:6789/0 conn(0x7f9fd40ad3e0 legacy=0x7f9fd4066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:13.070 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.054+0000 7f9fd359e700 1 -- 172.21.15.189:0/2952646422 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd413dfe0 con 0x7f9fd413a6e0 2020-11-20T12:15:13.070 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.054+0000 7f9fd2d9d700 1 --2- 172.21.15.189:0/2952646422 >> v2:172.21.15.47:3300/0 conn(0x7f9fd40ad000 0x7f9fd413e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:13.070 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.054+0000 7f9fd359e700 1 --2- 172.21.15.189:0/2952646422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fd413a6e0 0x7f9fd413ad00 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f9fd4068780 tx=0x7f9fcc009290).ready entity=mon.1 client_cookie=cf01e7a5501a0b2b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:13.071 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fd259c700 1 -- 172.21.15.189:0/2952646422 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc00bbc0 con 0x7f9fd413a6e0 2020-11-20T12:15:13.071 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fd259c700 1 -- 172.21.15.189:0/2952646422 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fcc00bd20 con 0x7f9fd413a6e0 2020-11-20T12:15:13.071 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fd259c700 1 -- 172.21.15.189:0/2952646422 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc011610 con 0x7f9fd413a6e0 2020-11-20T12:15:13.071 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fd8a2d700 1 -- 172.21.15.189:0/2952646422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fd413a6e0 msgr2=0x7f9fd413ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:13.072 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fd8a2d700 1 --2- 172.21.15.189:0/2952646422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fd413a6e0 0x7f9fd413ad00 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f9fd4068780 tx=0x7f9fcc009290).stop 2020-11-20T12:15:13.072 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fd8a2d700 1 -- 172.21.15.189:0/2952646422 shutdown_connections 2020-11-20T12:15:13.072 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fd8a2d700 1 --2- 172.21.15.189:0/2952646422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fd413a6e0 0x7f9fd413ad00 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:13.072 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fd8a2d700 1 --2- 172.21.15.189:0/2952646422 >> v2:172.21.15.47:3300/0 conn(0x7f9fd40ad000 0x7f9fd413e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:13.072 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fd8a2d700 1 -- 172.21.15.189:0/2952646422 >> 172.21.15.189:0/2952646422 conn(0x7f9fd4062080 msgr2=0x7f9fd4062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:13.073 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fd8a2d700 1 -- 172.21.15.189:0/2952646422 shutdown_connections 2020-11-20T12:15:13.073 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fd8a2d700 1 -- 172.21.15.189:0/2952646422 wait complete. 2020-11-20T12:15:13.073 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fd8a2d700 1 Processor -- start 2020-11-20T12:15:13.074 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fd8a2d700 1 -- start start 2020-11-20T12:15:13.074 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fd8a2d700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9fd40ad000 0x7f9fd4140980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:13.075 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fd8a2d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fd40ad3e0 0x7f9fd40a8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:13.075 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fd8a2d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9fd4149ea0 0x7f9fd414a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:13.075 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fd8a2d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd414a910 con 0x7f9fd4149ea0 2020-11-20T12:15:13.076 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fd8a2d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd40a8340 con 0x7f9fd40ad3e0 2020-11-20T12:15:13.076 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fd8a2d700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd40a85a0 con 0x7f9fd40ad000 2020-11-20T12:15:13.077 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fcbfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fd40ad3e0 0x7f9fd40a8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:13.077 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fcbfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fd40ad3e0 0x7f9fd40a8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50872/0 (socket says 172.21.15.189:50872) 2020-11-20T12:15:13.078 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fcbfff700 1 -- 172.21.15.189:0/1528285736 learned_addr learned my addr 172.21.15.189:0/1528285736 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:15:13.078 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fd2d9d700 1 --2- 172.21.15.189:0/1528285736 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9fd40ad000 0x7f9fd4140980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:13.079 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fd359e700 1 --2- 172.21.15.189:0/1528285736 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9fd4149ea0 0x7f9fd414a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:13.079 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fcbfff700 1 -- 172.21.15.189:0/1528285736 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9fd40ad000 msgr2=0x7f9fd4140980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:13.079 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fcbfff700 1 --2- 172.21.15.189:0/1528285736 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9fd40ad000 0x7f9fd4140980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:13.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fcbfff700 1 -- 172.21.15.189:0/1528285736 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9fd4149ea0 msgr2=0x7f9fd414a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:13.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fcbfff700 1 --2- 172.21.15.189:0/1528285736 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9fd4149ea0 0x7f9fd414a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:13.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fcbfff700 1 -- 172.21.15.189:0/1528285736 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd4141720 con 0x7f9fd40ad3e0 2020-11-20T12:15:13.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fcbfff700 1 --2- 172.21.15.189:0/1528285736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fd40ad3e0 0x7f9fd40a8a40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f9fbc001e70 tx=0x7f9fbc007c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:13.082 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fcaffd700 1 -- 172.21.15.189:0/1528285736 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc007020 con 0x7f9fd40ad3e0 2020-11-20T12:15:13.082 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fcaffd700 1 -- 172.21.15.189:0/1528285736 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fbc007180 con 0x7f9fd40ad3e0 2020-11-20T12:15:13.082 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fcaffd700 1 -- 172.21.15.189:0/1528285736 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc00e560 con 0x7f9fd40ad3e0 2020-11-20T12:15:13.082 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.058+0000 7f9fd8a2d700 1 -- 172.21.15.189:0/1528285736 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fd40a96f0 con 0x7f9fd40ad3e0 2020-11-20T12:15:13.083 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.062+0000 7f9fd8a2d700 1 -- 172.21.15.189:0/1528285736 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fd4136fc0 con 0x7f9fd40ad3e0 2020-11-20T12:15:13.083 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.062+0000 7f9fd8a2d700 1 -- 172.21.15.189:0/1528285736 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fd40a8760 con 0x7f9fd40ad3e0 2020-11-20T12:15:13.083 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.062+0000 7f9fcaffd700 1 -- 172.21.15.189:0/1528285736 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f9fbc010070 con 0x7f9fd40ad3e0 2020-11-20T12:15:13.084 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.062+0000 7f9fcaffd700 1 --2- 172.21.15.189:0/1528285736 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9fb00557c0 0x7f9fb0057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:13.084 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.062+0000 7f9fd2d9d700 1 --2- 172.21.15.189:0/1528285736 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9fb00557c0 0x7f9fb0057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:13.084 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.062+0000 7f9fcaffd700 1 -- 172.21.15.189:0/1528285736 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f9fbc06dc30 con 0x7f9fd40ad3e0 2020-11-20T12:15:13.085 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.062+0000 7f9fd2d9d700 1 --2- 172.21.15.189:0/1528285736 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9fb00557c0 0x7f9fb0057c80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f9fc4006fd0 tx=0x7f9fc4008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:13.085 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.074+0000 7f9fcaffd700 1 -- 172.21.15.189:0/1528285736 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9fbc042fd0 con 0x7f9fd40ad3e0 2020-11-20T12:15:13.284 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.278+0000 7f9fd8a2d700 1 -- 172.21.15.189:0/1528285736 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9fd40acdb0 con 0x7f9fd40ad3e0 2020-11-20T12:15:13.285 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.278+0000 7f9fcaffd700 1 -- 172.21.15.189:0/1528285736 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f9fbc043870 con 0x7f9fd40ad3e0 2020-11-20T12:15:13.286 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:15:13.286 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5389,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/636533719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":636533719},{"type":"v1","addr":"172.21.15.189:6835","nonce":636533719}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:13:20.671656+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":12,"addr":"172.21.15.47:6833/1811549340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1811549340},{"type":"v1","addr":"172.21.15.47:6833","nonce":1811549340}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:15:13.288 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.282+0000 7f9fd8a2d700 1 -- 172.21.15.189:0/1528285736 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9fb00557c0 msgr2=0x7f9fb0057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:13.288 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.282+0000 7f9fd8a2d700 1 --2- 172.21.15.189:0/1528285736 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9fb00557c0 0x7f9fb0057c80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f9fc4006fd0 tx=0x7f9fc4008040).stop 2020-11-20T12:15:13.288 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.282+0000 7f9fd8a2d700 1 -- 172.21.15.189:0/1528285736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fd40ad3e0 msgr2=0x7f9fd40a8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:13.288 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.282+0000 7f9fd8a2d700 1 --2- 172.21.15.189:0/1528285736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fd40ad3e0 0x7f9fd40a8a40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f9fbc001e70 tx=0x7f9fbc007c50).stop 2020-11-20T12:15:13.289 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.282+0000 7f9fd8a2d700 1 -- 172.21.15.189:0/1528285736 shutdown_connections 2020-11-20T12:15:13.289 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.282+0000 7f9fd8a2d700 1 --2- 172.21.15.189:0/1528285736 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f9fb00557c0 0x7f9fb0057c80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:13.289 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.282+0000 7f9fd8a2d700 1 --2- 172.21.15.189:0/1528285736 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f9fd4149ea0 0x7f9fd414a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:13.289 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.282+0000 7f9fd8a2d700 1 --2- 172.21.15.189:0/1528285736 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9fd40ad000 0x7f9fd4140980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:13.290 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.282+0000 7f9fd8a2d700 1 --2- 172.21.15.189:0/1528285736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9fd40ad3e0 0x7f9fd40a8a40 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:13.290 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.282+0000 7f9fd8a2d700 1 -- 172.21.15.189:0/1528285736 >> 172.21.15.189:0/1528285736 conn(0x7f9fd4062080 msgr2=0x7f9fd405ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:13.290 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.282+0000 7f9fd8a2d700 1 -- 172.21.15.189:0/1528285736 shutdown_connections 2020-11-20T12:15:13.290 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:13.282+0000 7f9fd8a2d700 1 -- 172.21.15.189:0/1528285736 wait complete. 2020-11-20T12:15:13.291 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 41 2020-11-20T12:15:13.300 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:15:15.302 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:15.443 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.434+0000 7fa58359e700 1 Processor -- start 2020-11-20T12:15:15.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.434+0000 7fa58359e700 1 -- start start 2020-11-20T12:15:15.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.434+0000 7fa58359e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa5840aeb80 0x7fa58413a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:15.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.434+0000 7fa58359e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa58413dcb0 con 0x7fa5840aef60 2020-11-20T12:15:15.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.434+0000 7fa58359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa58413df10 con 0x7fa58413e650 2020-11-20T12:15:15.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.434+0000 7fa58359e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa58413eab0 con 0x7fa5840aeb80 2020-11-20T12:15:15.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.434+0000 7fa58259c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa5840aeb80 0x7fa58413a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:15.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.434+0000 7fa582d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa58413e650 0x7fa5840a1960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52504/0 (socket says 172.21.15.189:52504) 2020-11-20T12:15:15.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.434+0000 7fa582d9d700 1 -- 172.21.15.189:0/685547855 learned_addr learned my addr 172.21.15.189:0/685547855 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:15:15.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.438+0000 7fa58259c700 1 -- 172.21.15.189:0/685547855 >> v1:172.21.15.189:6789/0 conn(0x7fa58413e650 legacy=0x7fa5840a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:15.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.438+0000 7fa58159a700 1 -- 172.21.15.189:0/685547855 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 361451317 0 0) 0x7fa58413dcb0 con 0x7fa5840aef60 2020-11-20T12:15:15.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.438+0000 7fa58259c700 1 -- 172.21.15.189:0/685547855 >> v1:172.21.15.47:6789/0 conn(0x7fa5840aef60 legacy=0x7fa58413ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:15.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.438+0000 7fa58259c700 1 -- 172.21.15.189:0/685547855 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa58413b4f0 con 0x7fa5840aeb80 2020-11-20T12:15:15.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.438+0000 7fa58259c700 1 --2- 172.21.15.189:0/685547855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5840aeb80 0x7fa58413a640 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fa57400b920 tx=0x7fa574006df0).ready entity=mon.1 client_cookie=bbc5ad5beedcfe64 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:15.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.438+0000 7fa58159a700 1 -- 172.21.15.189:0/685547855 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa574009730 con 0x7fa5840aeb80 2020-11-20T12:15:15.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.438+0000 7fa58159a700 1 -- 172.21.15.189:0/685547855 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa574009890 con 0x7fa5840aeb80 2020-11-20T12:15:15.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.438+0000 7fa58159a700 1 -- 172.21.15.189:0/685547855 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa574009c10 con 0x7fa5840aeb80 2020-11-20T12:15:15.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.438+0000 7fa58359e700 1 -- 172.21.15.189:0/685547855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5840aeb80 msgr2=0x7fa58413a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:15.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.438+0000 7fa58359e700 1 --2- 172.21.15.189:0/685547855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5840aeb80 0x7fa58413a640 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fa57400b920 tx=0x7fa574006df0).stop 2020-11-20T12:15:15.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.438+0000 7fa58359e700 1 -- 172.21.15.189:0/685547855 shutdown_connections 2020-11-20T12:15:15.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.438+0000 7fa58359e700 1 --2- 172.21.15.189:0/685547855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5840aeb80 0x7fa58413a640 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:15.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.438+0000 7fa58359e700 1 -- 172.21.15.189:0/685547855 >> 172.21.15.189:0/685547855 conn(0x7fa584062080 msgr2=0x7fa584062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:15.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.438+0000 7fa58359e700 1 -- 172.21.15.189:0/685547855 shutdown_connections 2020-11-20T12:15:15.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.438+0000 7fa58359e700 1 -- 172.21.15.189:0/685547855 wait complete. 2020-11-20T12:15:15.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.438+0000 7fa58359e700 1 Processor -- start 2020-11-20T12:15:15.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.438+0000 7fa58359e700 1 -- start start 2020-11-20T12:15:15.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.442+0000 7fa58359e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5840aeb80 0x7fa5840a7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:15.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.442+0000 7fa58359e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5840aef60 0x7fa5840a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:15.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.442+0000 7fa58359e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa58413e650 0x7fa5840a6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:15.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.442+0000 7fa58359e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5840a7550 con 0x7fa5840aeb80 2020-11-20T12:15:15.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.442+0000 7fa58359e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5840a8630 con 0x7fa5840aef60 2020-11-20T12:15:15.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.442+0000 7fa58359e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5840a6170 con 0x7fa58413e650 2020-11-20T12:15:15.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.442+0000 7fa58259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5840aeb80 0x7fa5840a7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:15.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.442+0000 7fa581d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5840aef60 0x7fa5840a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:15.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.442+0000 7fa58259c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5840aeb80 0x7fa5840a7f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.189:57110/0 (socket says 172.21.15.189:57110) 2020-11-20T12:15:15.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.442+0000 7fa58259c700 1 -- 172.21.15.189:0/2269763565 learned_addr learned my addr 172.21.15.189:0/2269763565 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:15:15.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.442+0000 7fa581d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5840aef60 0x7fa5840a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50890/0 (socket says 172.21.15.189:50890) 2020-11-20T12:15:15.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.442+0000 7fa582d9d700 1 --2- 172.21.15.189:0/2269763565 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa58413e650 0x7fa5840a6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:15.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.442+0000 7fa581d9b700 1 -- 172.21.15.189:0/2269763565 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa58413e650 msgr2=0x7fa5840a6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:15.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.442+0000 7fa581d9b700 1 --2- 172.21.15.189:0/2269763565 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa58413e650 0x7fa5840a6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:15.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.442+0000 7fa581d9b700 1 -- 172.21.15.189:0/2269763565 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5840aeb80 msgr2=0x7fa5840a7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:15.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.442+0000 7fa581d9b700 1 --2- 172.21.15.189:0/2269763565 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5840aeb80 0x7fa5840a7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:15.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.442+0000 7fa581d9b700 1 -- 172.21.15.189:0/2269763565 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5840647a0 con 0x7fa5840aef60 2020-11-20T12:15:15.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.442+0000 7fa581d9b700 1 --2- 172.21.15.189:0/2269763565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5840aef60 0x7fa5840a8b90 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fa56c00c730 tx=0x7fa56c0079d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:15.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.442+0000 7fa57a7fc700 1 -- 172.21.15.189:0/2269763565 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c007240 con 0x7fa5840aef60 2020-11-20T12:15:15.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.442+0000 7fa57a7fc700 1 -- 172.21.15.189:0/2269763565 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56c0073c0 con 0x7fa5840aef60 2020-11-20T12:15:15.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.442+0000 7fa58359e700 1 -- 172.21.15.189:0/2269763565 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5840a6b00 con 0x7fa5840aef60 2020-11-20T12:15:15.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.442+0000 7fa57a7fc700 1 -- 172.21.15.189:0/2269763565 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c006740 con 0x7fa5840aef60 2020-11-20T12:15:15.459 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.442+0000 7fa58359e700 1 -- 172.21.15.189:0/2269763565 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa584065140 con 0x7fa5840aef60 2020-11-20T12:15:15.459 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.442+0000 7fa58359e700 1 -- 172.21.15.189:0/2269763565 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5840446d0 con 0x7fa5840aef60 2020-11-20T12:15:15.459 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.446+0000 7fa57a7fc700 1 -- 172.21.15.189:0/2269763565 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fa56c010030 con 0x7fa5840aef60 2020-11-20T12:15:15.460 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.446+0000 7fa57a7fc700 1 --2- 172.21.15.189:0/2269763565 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa5600557c0 0x7fa560057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:15.460 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.446+0000 7fa57a7fc700 1 -- 172.21.15.189:0/2269763565 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6675+0+0 (secure 0 0 0) 0x7fa56c06e380 con 0x7fa5840aef60 2020-11-20T12:15:15.460 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.446+0000 7fa58259c700 1 --2- 172.21.15.189:0/2269763565 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa5600557c0 0x7fa560057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:15.461 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.446+0000 7fa58259c700 1 --2- 172.21.15.189:0/2269763565 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa5600557c0 0x7fa560057c80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fa574009860 tx=0x7fa574015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:15.461 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.450+0000 7fa57a7fc700 1 -- 172.21.15.189:0/2269763565 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa56c043fc0 con 0x7fa5840aef60 2020-11-20T12:15:15.659 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.650+0000 7fa58359e700 1 -- 172.21.15.189:0/2269763565 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5840a6330 con 0x7fa5840aef60 2020-11-20T12:15:15.660 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.654+0000 7fa57a7fc700 1 -- 172.21.15.189:0/2269763565 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2137 (secure 0 0 0) 0x7fa56c0475e0 con 0x7fa5840aef60 2020-11-20T12:15:15.660 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:15:15.660 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5389,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/636533719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":636533719},{"type":"v1","addr":"172.21.15.189:6835","nonce":636533719}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:13:20.671656+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":12,"addr":"172.21.15.47:6833/1811549340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1811549340},{"type":"v1","addr":"172.21.15.47:6833","nonce":1811549340}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:15:15.663 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.654+0000 7fa58359e700 1 -- 172.21.15.189:0/2269763565 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa5600557c0 msgr2=0x7fa560057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:15.663 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.654+0000 7fa58359e700 1 --2- 172.21.15.189:0/2269763565 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa5600557c0 0x7fa560057c80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fa574009860 tx=0x7fa574015000).stop 2020-11-20T12:15:15.663 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.654+0000 7fa58359e700 1 -- 172.21.15.189:0/2269763565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5840aef60 msgr2=0x7fa5840a8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:15.664 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.654+0000 7fa58359e700 1 --2- 172.21.15.189:0/2269763565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5840aef60 0x7fa5840a8b90 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fa56c00c730 tx=0x7fa56c0079d0).stop 2020-11-20T12:15:15.664 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.654+0000 7fa58359e700 1 -- 172.21.15.189:0/2269763565 shutdown_connections 2020-11-20T12:15:15.664 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.654+0000 7fa58359e700 1 --2- 172.21.15.189:0/2269763565 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa5600557c0 0x7fa560057c80 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:15.664 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.654+0000 7fa58359e700 1 --2- 172.21.15.189:0/2269763565 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa5840aeb80 0x7fa5840a7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:15.664 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.654+0000 7fa58359e700 1 --2- 172.21.15.189:0/2269763565 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa58413e650 0x7fa5840a6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:15.665 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.654+0000 7fa58359e700 1 --2- 172.21.15.189:0/2269763565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5840aef60 0x7fa5840a8b90 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:15.665 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.654+0000 7fa58359e700 1 -- 172.21.15.189:0/2269763565 >> 172.21.15.189:0/2269763565 conn(0x7fa584062080 msgr2=0x7fa5840ae3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:15.665 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.654+0000 7fa58359e700 1 -- 172.21.15.189:0/2269763565 shutdown_connections 2020-11-20T12:15:15.665 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:15.654+0000 7fa58359e700 1 -- 172.21.15.189:0/2269763565 wait complete. 2020-11-20T12:15:15.666 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 41 2020-11-20T12:15:15.679 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:15:17.681 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:17.815 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.806+0000 7f051e170700 1 Processor -- start 2020-11-20T12:15:17.816 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.806+0000 7f051e170700 1 -- start start 2020-11-20T12:15:17.817 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.806+0000 7f051e170700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f05180ad000 0x7f051813e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:17.818 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.806+0000 7f051e170700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0518067730 con 0x7f051813a6e0 2020-11-20T12:15:17.818 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.806+0000 7f051e170700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051813dd40 con 0x7f05180ad3e0 2020-11-20T12:15:17.819 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.806+0000 7f051e170700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0518138190 con 0x7f05180ad000 2020-11-20T12:15:17.820 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.806+0000 7f051c96d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f05180ad3e0 0x7f0518066d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52516/0 (socket says 172.21.15.189:52516) 2020-11-20T12:15:17.820 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.806+0000 7f051c96d700 1 -- 172.21.15.189:0/2406741097 learned_addr learned my addr 172.21.15.189:0/2406741097 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:15:17.822 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.806+0000 7f051d16e700 1 --2- 172.21.15.189:0/2406741097 >> v2:172.21.15.47:3300/0 conn(0x7f05180ad000 0x7f051813e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:17.823 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.806+0000 7f050ffff700 1 -- 172.21.15.189:0/2406741097 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4156678583 0 0) 0x7f051813dd40 con 0x7f05180ad3e0 2020-11-20T12:15:17.824 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.806+0000 7f050ffff700 1 -- 172.21.15.189:0/2406741097 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04fc003240 con 0x7f05180ad3e0 2020-11-20T12:15:17.843 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.806+0000 7f050ffff700 1 -- 172.21.15.189:0/2406741097 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 976590864 0 0) 0x7f0518067730 con 0x7f051813a6e0 2020-11-20T12:15:17.848 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.806+0000 7f050ffff700 1 -- 172.21.15.189:0/2406741097 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f051813dd40 con 0x7f051813a6e0 2020-11-20T12:15:17.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f050ffff700 1 -- 172.21.15.189:0/2406741097 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3728441219 0 0) 0x7f04fc003240 con 0x7f05180ad3e0 2020-11-20T12:15:17.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f050ffff700 1 -- 172.21.15.189:0/2406741097 >> v2:172.21.15.47:3300/0 conn(0x7f05180ad000 msgr2=0x7f051813e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:17.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f050ffff700 1 --2- 172.21.15.189:0/2406741097 >> v2:172.21.15.47:3300/0 conn(0x7f05180ad000 0x7f051813e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:17.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f050ffff700 1 -- 172.21.15.189:0/2406741097 >> v1:172.21.15.47:6789/0 conn(0x7f051813a6e0 legacy=0x7f051813ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:17.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f050ffff700 1 -- 172.21.15.189:0/2406741097 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051813dfe0 con 0x7f05180ad3e0 2020-11-20T12:15:17.850 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f050ffff700 1 -- 172.21.15.189:0/2406741097 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f05140014e0 con 0x7f05180ad3e0 2020-11-20T12:15:17.850 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f050ffff700 1 -- 172.21.15.189:0/2406741097 >> v1:172.21.15.189:6789/0 conn(0x7f05180ad3e0 legacy=0x7f0518066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:17.850 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f050ffff700 1 --2- 172.21.15.189:0/2406741097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04fc0038c0 0x7f04fc003cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:17.850 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f050ffff700 1 --2- 172.21.15.189:0/2406741097 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04fc0044e0 0x7f04fc004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:17.851 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f050ffff700 1 --2- 172.21.15.189:0/2406741097 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04fc004e50 0x7f04fc00b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:17.851 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f050ffff700 1 -- 172.21.15.189:0/2406741097 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0518138190 con 0x7f04fc004e50 2020-11-20T12:15:17.851 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f050ffff700 1 -- 172.21.15.189:0/2406741097 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc0053b0 con 0x7f04fc0038c0 2020-11-20T12:15:17.851 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f050ffff700 1 -- 172.21.15.189:0/2406741097 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc00bca0 con 0x7f04fc0044e0 2020-11-20T12:15:17.851 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f051d16e700 1 --2- 172.21.15.189:0/2406741097 >> v2:172.21.15.47:3300/0 conn(0x7f05180ad000 0x7f051813e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:15:17.852 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f051d16e700 1 --2- 172.21.15.189:0/2406741097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04fc0038c0 0x7f04fc003cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:17.852 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f051c96d700 1 --2- 172.21.15.189:0/2406741097 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04fc0044e0 0x7f04fc004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:17.852 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f051d96f700 1 --2- 172.21.15.189:0/2406741097 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04fc004e50 0x7f04fc00b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:17.852 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f051d16e700 1 -- 172.21.15.189:0/2406741097 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04fc0044e0 msgr2=0x7f04fc004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:17.852 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f051d16e700 1 --2- 172.21.15.189:0/2406741097 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04fc0044e0 0x7f04fc004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:17.853 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f051d16e700 1 -- 172.21.15.189:0/2406741097 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04fc004e50 msgr2=0x7f04fc00b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:17.853 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f051d16e700 1 --2- 172.21.15.189:0/2406741097 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f04fc004e50 0x7f04fc00b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:17.853 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f051d16e700 1 -- 172.21.15.189:0/2406741097 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04fc003240 con 0x7f04fc0038c0 2020-11-20T12:15:17.853 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f051d96f700 1 -- 172.21.15.189:0/2406741097 reap_dead start 2020-11-20T12:15:17.853 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f051d16e700 1 --2- 172.21.15.189:0/2406741097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04fc0038c0 0x7f04fc003cc0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f050800be60 tx=0x7f0508006df0).ready entity=mon.1 client_cookie=a39a00ef1903d7a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:17.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f050ffff700 1 -- 172.21.15.189:0/2406741097 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050800b590 con 0x7f04fc0038c0 2020-11-20T12:15:17.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f050ffff700 1 -- 172.21.15.189:0/2406741097 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f050800ec40 con 0x7f04fc0038c0 2020-11-20T12:15:17.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f050ffff700 1 -- 172.21.15.189:0/2406741097 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05080123f0 con 0x7f04fc0038c0 2020-11-20T12:15:17.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f051e170700 1 -- 172.21.15.189:0/2406741097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04fc0038c0 msgr2=0x7f04fc003cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:17.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f051e170700 1 --2- 172.21.15.189:0/2406741097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04fc0038c0 0x7f04fc003cc0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f050800be60 tx=0x7f0508006df0).stop 2020-11-20T12:15:17.855 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f051e170700 1 -- 172.21.15.189:0/2406741097 shutdown_connections 2020-11-20T12:15:17.855 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f051e170700 1 --2- 172.21.15.189:0/2406741097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04fc0038c0 0x7f04fc003cc0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:17.855 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f051e170700 1 -- 172.21.15.189:0/2406741097 >> 172.21.15.189:0/2406741097 conn(0x7f0518062080 msgr2=0x7f0518062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:17.855 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f051e170700 1 -- 172.21.15.189:0/2406741097 shutdown_connections 2020-11-20T12:15:17.855 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.810+0000 7f051e170700 1 -- 172.21.15.189:0/2406741097 wait complete. 2020-11-20T12:15:17.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.814+0000 7f051e170700 1 Processor -- start 2020-11-20T12:15:17.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.814+0000 7f051e170700 1 -- start start 2020-11-20T12:15:17.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.814+0000 7f051e170700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04fc0038c0 0x7f05180a58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:17.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.814+0000 7f051e170700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05180a8e00 0x7f051814a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:17.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.814+0000 7f051e170700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f051814a570 0x7f0518134c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:17.857 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.814+0000 7f051e170700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05180a9280 con 0x7f051814a570 2020-11-20T12:15:17.857 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.814+0000 7f051e170700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051813b930 con 0x7f05180a8e00 2020-11-20T12:15:17.857 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.814+0000 7f051e170700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05180a9510 con 0x7f04fc0038c0 2020-11-20T12:15:17.857 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.814+0000 7f051c96d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05180a8e00 0x7f051814a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:17.858 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.814+0000 7f051c96d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05180a8e00 0x7f051814a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50908/0 (socket says 172.21.15.189:50908) 2020-11-20T12:15:17.858 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.814+0000 7f051c96d700 1 -- 172.21.15.189:0/3708820597 learned_addr learned my addr 172.21.15.189:0/3708820597 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:15:17.858 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.814+0000 7f051d16e700 1 --2- 172.21.15.189:0/3708820597 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04fc0038c0 0x7f05180a58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:17.858 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.814+0000 7f051d96f700 1 --2- 172.21.15.189:0/3708820597 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f051814a570 0x7f0518134c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:17.858 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.814+0000 7f051c96d700 1 -- 172.21.15.189:0/3708820597 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04fc0038c0 msgr2=0x7f05180a58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:17.859 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.814+0000 7f051c96d700 1 --2- 172.21.15.189:0/3708820597 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04fc0038c0 0x7f05180a58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:17.859 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.814+0000 7f051c96d700 1 -- 172.21.15.189:0/3708820597 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f051814a570 msgr2=0x7f0518134c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:17.859 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.814+0000 7f051c96d700 1 --2- 172.21.15.189:0/3708820597 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f051814a570 0x7f0518134c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:17.859 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.814+0000 7f051c96d700 1 -- 172.21.15.189:0/3708820597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05180a97b0 con 0x7f05180a8e00 2020-11-20T12:15:17.861 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.814+0000 7f051c96d700 1 --2- 172.21.15.189:0/3708820597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05180a8e00 0x7f051814a080 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f0514004b00 tx=0x7f051813e170).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:17.861 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.814+0000 7f050d7fa700 1 -- 172.21.15.189:0/3708820597 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051400f930 con 0x7f05180a8e00 2020-11-20T12:15:17.861 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.814+0000 7f050d7fa700 1 -- 172.21.15.189:0/3708820597 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f051400fa90 con 0x7f05180a8e00 2020-11-20T12:15:17.862 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.814+0000 7f050d7fa700 1 -- 172.21.15.189:0/3708820597 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051400fe10 con 0x7f05180a8e00 2020-11-20T12:15:17.862 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.814+0000 7f051e170700 1 -- 172.21.15.189:0/3708820597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0518149be0 con 0x7f05180a8e00 2020-11-20T12:15:17.862 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.814+0000 7f051e170700 1 -- 172.21.15.189:0/3708820597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05180a8240 con 0x7f05180a8e00 2020-11-20T12:15:17.862 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.818+0000 7f050d7fa700 1 -- 172.21.15.189:0/3708820597 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f0514012560 con 0x7f05180a8e00 2020-11-20T12:15:17.862 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.818+0000 7f051e170700 1 -- 172.21.15.189:0/3708820597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05180b13d0 con 0x7f05180a8e00 2020-11-20T12:15:17.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.818+0000 7f050d7fa700 1 --2- 172.21.15.189:0/3708820597 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f04f00557f0 0x7f04f0057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:17.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.818+0000 7f050d7fa700 1 -- 172.21.15.189:0/3708820597 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f051406e130 con 0x7f05180a8e00 2020-11-20T12:15:17.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.818+0000 7f051d16e700 1 --2- 172.21.15.189:0/3708820597 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f04f00557f0 0x7f04f0057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:17.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.818+0000 7f051d16e700 1 --2- 172.21.15.189:0/3708820597 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f04f00557f0 0x7f04f0057cb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f0508006d00 tx=0x7f0508019000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:17.864 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:17.822+0000 7f050d7fa700 1 -- 172.21.15.189:0/3708820597 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0514043c00 con 0x7f05180a8e00 2020-11-20T12:15:18.033 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:18.025+0000 7f051e170700 1 -- 172.21.15.189:0/3708820597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05180446d0 con 0x7f05180a8e00 2020-11-20T12:15:18.033 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:18.025+0000 7f050d7fa700 1 -- 172.21.15.189:0/3708820597 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f0514019070 con 0x7f05180a8e00 2020-11-20T12:15:18.034 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:15:18.034 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5389,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/636533719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":636533719},{"type":"v1","addr":"172.21.15.189:6835","nonce":636533719}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:13:20.671656+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":12,"addr":"172.21.15.47:6833/1811549340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1811549340},{"type":"v1","addr":"172.21.15.47:6833","nonce":1811549340}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:15:18.035 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:18.029+0000 7f051e170700 1 -- 172.21.15.189:0/3708820597 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f04f00557f0 msgr2=0x7f04f0057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:18.036 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:18.029+0000 7f051e170700 1 --2- 172.21.15.189:0/3708820597 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f04f00557f0 0x7f04f0057cb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f0508006d00 tx=0x7f0508019000).stop 2020-11-20T12:15:18.036 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:18.029+0000 7f051e170700 1 -- 172.21.15.189:0/3708820597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05180a8e00 msgr2=0x7f051814a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:18.036 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:18.029+0000 7f051e170700 1 --2- 172.21.15.189:0/3708820597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05180a8e00 0x7f051814a080 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f0514004b00 tx=0x7f051813e170).stop 2020-11-20T12:15:18.036 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:18.029+0000 7f051e170700 1 -- 172.21.15.189:0/3708820597 shutdown_connections 2020-11-20T12:15:18.037 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:18.029+0000 7f051e170700 1 --2- 172.21.15.189:0/3708820597 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f04f00557f0 0x7f04f0057cb0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:18.037 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:18.029+0000 7f051e170700 1 --2- 172.21.15.189:0/3708820597 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f051814a570 0x7f0518134c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:18.037 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:18.029+0000 7f051e170700 1 --2- 172.21.15.189:0/3708820597 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f04fc0038c0 0x7f05180a58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:18.038 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:18.029+0000 7f051e170700 1 --2- 172.21.15.189:0/3708820597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05180a8e00 0x7f051814a080 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:18.038 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:18.029+0000 7f051e170700 1 -- 172.21.15.189:0/3708820597 >> 172.21.15.189:0/3708820597 conn(0x7f0518062080 msgr2=0x7f05180abf00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:18.038 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:18.029+0000 7f051e170700 1 -- 172.21.15.189:0/3708820597 shutdown_connections 2020-11-20T12:15:18.038 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:18.029+0000 7f051e170700 1 -- 172.21.15.189:0/3708820597 wait complete. 2020-11-20T12:15:18.039 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 41 2020-11-20T12:15:18.048 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:15:20.050 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:20.180 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 Processor -- start 2020-11-20T12:15:20.180 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 -- start start 2020-11-20T12:15:20.181 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f29000a9660 0x7f290013e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:20.182 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f29000a9a40 0x7f2900060170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:20.182 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f29000a7760 0x7f2900060660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:20.182 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f290013dc10 con 0x7f29000a7760 2020-11-20T12:15:20.182 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f290013deb0 con 0x7f29000a9660 2020-11-20T12:15:20.182 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f290013e150 con 0x7f29000a9a40 2020-11-20T12:15:20.183 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28fe59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f29000a9660 0x7f290013e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:20.183 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28fe59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f29000a9660 0x7f290013e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50914/0 (socket says 172.21.15.189:50914) 2020-11-20T12:15:20.183 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28fe59c700 1 -- 172.21.15.189:0/3677772396 learned_addr learned my addr 172.21.15.189:0/3677772396 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:15:20.183 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28fed9d700 1 --2- 172.21.15.189:0/3677772396 >> v2:172.21.15.47:3300/0 conn(0x7f29000a7760 0x7f2900060660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:20.184 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28fdd9b700 1 --2- 172.21.15.189:0/3677772396 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f29000a9a40 0x7f2900060170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:20.184 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28fe59c700 1 -- 172.21.15.189:0/3677772396 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f29000a9a40 msgr2=0x7f2900060170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:20.184 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28fe59c700 1 --2- 172.21.15.189:0/3677772396 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f29000a9a40 0x7f2900060170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:20.184 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28fe59c700 1 -- 172.21.15.189:0/3677772396 >> v2:172.21.15.47:3300/0 conn(0x7f29000a7760 msgr2=0x7f2900060660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:20.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28fe59c700 1 --2- 172.21.15.189:0/3677772396 >> v2:172.21.15.47:3300/0 conn(0x7f29000a7760 0x7f2900060660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:20.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28fe59c700 1 -- 172.21.15.189:0/3677772396 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2900065a30 con 0x7f29000a9660 2020-11-20T12:15:20.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28fe59c700 1 --2- 172.21.15.189:0/3677772396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29000a9660 0x7f290013e5f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f2900068820 tx=0x7f28f0009290).ready entity=mon.1 client_cookie=d9cf9ce0b34debf3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:20.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28fd59a700 1 -- 172.21.15.189:0/3677772396 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f000bb10 con 0x7f29000a9660 2020-11-20T12:15:20.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28fd59a700 1 -- 172.21.15.189:0/3677772396 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f000bc70 con 0x7f29000a9660 2020-11-20T12:15:20.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28fd59a700 1 -- 172.21.15.189:0/3677772396 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f0011400 con 0x7f29000a9660 2020-11-20T12:15:20.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 -- 172.21.15.189:0/3677772396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29000a9660 msgr2=0x7f290013e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:20.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 --2- 172.21.15.189:0/3677772396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29000a9660 0x7f290013e5f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f2900068820 tx=0x7f28f0009290).stop 2020-11-20T12:15:20.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 -- 172.21.15.189:0/3677772396 shutdown_connections 2020-11-20T12:15:20.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 --2- 172.21.15.189:0/3677772396 >> v2:172.21.15.47:3300/0 conn(0x7f29000a7760 0x7f2900060660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:20.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 --2- 172.21.15.189:0/3677772396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29000a9660 0x7f290013e5f0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:20.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 --2- 172.21.15.189:0/3677772396 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f29000a9a40 0x7f2900060170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:20.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 -- 172.21.15.189:0/3677772396 >> 172.21.15.189:0/3677772396 conn(0x7f2900061f60 msgr2=0x7f2900062370 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:20.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 -- 172.21.15.189:0/3677772396 shutdown_connections 2020-11-20T12:15:20.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 -- 172.21.15.189:0/3677772396 wait complete. 2020-11-20T12:15:20.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 Processor -- start 2020-11-20T12:15:20.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 -- start start 2020-11-20T12:15:20.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29000a7760 0x7f29000a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:20.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29000a9660 0x7f29000a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:20.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f29000a9a40 0x7f290014cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:20.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29000a3bc0 con 0x7f29000a7760 2020-11-20T12:15:20.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29000a0b60 con 0x7f29000a9660 2020-11-20T12:15:20.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29000a0e00 con 0x7f29000a9a40 2020-11-20T12:15:20.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28fdd9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29000a9660 0x7f29000a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:20.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28fdd9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29000a9660 0x7f29000a36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50922/0 (socket says 172.21.15.189:50922) 2020-11-20T12:15:20.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28fdd9b700 1 -- 172.21.15.189:0/1944545372 learned_addr learned my addr 172.21.15.189:0/1944545372 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:15:20.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28fe59c700 1 --2- 172.21.15.189:0/1944545372 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29000a7760 0x7f29000a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:20.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28fed9d700 1 --2- 172.21.15.189:0/1944545372 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f29000a9a40 0x7f290014cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:20.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28fdd9b700 1 -- 172.21.15.189:0/1944545372 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f29000a9a40 msgr2=0x7f290014cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:20.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28fdd9b700 1 --2- 172.21.15.189:0/1944545372 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f29000a9a40 0x7f290014cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:20.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28fdd9b700 1 -- 172.21.15.189:0/1944545372 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29000a7760 msgr2=0x7f29000a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:20.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28fdd9b700 1 --2- 172.21.15.189:0/1944545372 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29000a7760 0x7f29000a1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:20.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28fdd9b700 1 -- 172.21.15.189:0/1944545372 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29000a10a0 con 0x7f29000a9660 2020-11-20T12:15:20.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28fdd9b700 1 --2- 172.21.15.189:0/1944545372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29000a9660 0x7f29000a36d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f28e800ec00 tx=0x7f28e8006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:20.194 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28f67fc700 1 -- 172.21.15.189:0/1944545372 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e800b9a0 con 0x7f29000a9660 2020-11-20T12:15:20.194 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28f67fc700 1 -- 172.21.15.189:0/1944545372 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e800bb00 con 0x7f29000a9660 2020-11-20T12:15:20.194 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 -- 172.21.15.189:0/1944545372 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29000a0850 con 0x7f29000a9660 2020-11-20T12:15:20.194 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f28f67fc700 1 -- 172.21.15.189:0/1944545372 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e800be80 con 0x7f29000a9660 2020-11-20T12:15:20.195 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.177+0000 7f2904b05700 1 -- 172.21.15.189:0/1944545372 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f290014d460 con 0x7f29000a9660 2020-11-20T12:15:20.195 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.181+0000 7f2904b05700 1 -- 172.21.15.189:0/1944545372 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29000a1230 con 0x7f29000a9660 2020-11-20T12:15:20.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.181+0000 7f28f67fc700 1 -- 172.21.15.189:0/1944545372 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f28e8012600 con 0x7f29000a9660 2020-11-20T12:15:20.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.181+0000 7f28f67fc700 1 --2- 172.21.15.189:0/1944545372 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f28dc0557c0 0x7f28dc057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:20.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.181+0000 7f28fe59c700 1 --2- 172.21.15.189:0/1944545372 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f28dc0557c0 0x7f28dc057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:20.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.181+0000 7f28f67fc700 1 -- 172.21.15.189:0/1944545372 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f28e806ed10 con 0x7f29000a9660 2020-11-20T12:15:20.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.181+0000 7f28fe59c700 1 --2- 172.21.15.189:0/1944545372 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f28dc0557c0 0x7f28dc057c80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f28f000bc40 tx=0x7f28f0016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:20.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.193+0000 7f28f67fc700 1 -- 172.21.15.189:0/1944545372 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f28e8044950 con 0x7f29000a9660 2020-11-20T12:15:20.398 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.389+0000 7f2904b05700 1 -- 172.21.15.189:0/1944545372 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2900044f60 con 0x7f29000a9660 2020-11-20T12:15:20.399 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.389+0000 7f28f67fc700 1 -- 172.21.15.189:0/1944545372 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f28e8047f70 con 0x7f29000a9660 2020-11-20T12:15:20.399 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:15:20.400 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5389,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/636533719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":636533719},{"type":"v1","addr":"172.21.15.189:6835","nonce":636533719}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:13:20.671656+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":29,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"b","rank":0,"incarnation":37,"state":"up:active","state_seq":12,"addr":"172.21.15.47:6833/1811549340","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":1811549340},{"type":"v1","addr":"172.21.15.47:6833","nonce":1811549340}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:15:20.401 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.393+0000 7f2904b05700 1 -- 172.21.15.189:0/1944545372 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f28dc0557c0 msgr2=0x7f28dc057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:20.402 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.393+0000 7f2904b05700 1 --2- 172.21.15.189:0/1944545372 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f28dc0557c0 0x7f28dc057c80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f28f000bc40 tx=0x7f28f0016000).stop 2020-11-20T12:15:20.402 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.393+0000 7f2904b05700 1 -- 172.21.15.189:0/1944545372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29000a9660 msgr2=0x7f29000a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:20.402 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.393+0000 7f2904b05700 1 --2- 172.21.15.189:0/1944545372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29000a9660 0x7f29000a36d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f28e800ec00 tx=0x7f28e8006df0).stop 2020-11-20T12:15:20.403 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.393+0000 7f2904b05700 1 -- 172.21.15.189:0/1944545372 shutdown_connections 2020-11-20T12:15:20.403 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.393+0000 7f2904b05700 1 --2- 172.21.15.189:0/1944545372 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f28dc0557c0 0x7f28dc057c80 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:20.403 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.393+0000 7f2904b05700 1 --2- 172.21.15.189:0/1944545372 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f29000a7760 0x7f29000a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:20.403 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.393+0000 7f2904b05700 1 --2- 172.21.15.189:0/1944545372 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f29000a9a40 0x7f290014cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:20.404 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.393+0000 7f2904b05700 1 --2- 172.21.15.189:0/1944545372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29000a9660 0x7f29000a36d0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:20.404 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.393+0000 7f2904b05700 1 -- 172.21.15.189:0/1944545372 >> 172.21.15.189:0/1944545372 conn(0x7f2900061f60 msgr2=0x7f29000608c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:20.404 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.393+0000 7f2904b05700 1 -- 172.21.15.189:0/1944545372 shutdown_connections 2020-11-20T12:15:20.404 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:20.393+0000 7f2904b05700 1 -- 172.21.15.189:0/1944545372 wait complete. 2020-11-20T12:15:20.405 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 41 2020-11-20T12:15:20.414 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:15:22.415 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:22.551 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.545+0000 7f946f59e700 1 Processor -- start 2020-11-20T12:15:22.551 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.545+0000 7f946f59e700 1 -- start start 2020-11-20T12:15:22.553 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946f59e700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f94700ad000 0x7f947013e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.554 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946f59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9470067730 con 0x7f94700ad3e0 2020-11-20T12:15:22.554 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946f59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f947013dd40 con 0x7f947013a6e0 2020-11-20T12:15:22.554 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946f59e700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9470138190 con 0x7f94700ad000 2020-11-20T12:15:22.554 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946ed9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f947013a6e0 0x7f947013ad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52556/0 (socket says 172.21.15.189:52556) 2020-11-20T12:15:22.554 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946ed9d700 1 -- 172.21.15.189:0/3194448153 learned_addr learned my addr 172.21.15.189:0/3194448153 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:15:22.555 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946e59c700 1 --2- 172.21.15.189:0/3194448153 >> v2:172.21.15.47:3300/0 conn(0x7f94700ad000 0x7f947013e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.555 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946d59a700 1 -- 172.21.15.189:0/3194448153 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220914792 0 0) 0x7f947013dd40 con 0x7f947013a6e0 2020-11-20T12:15:22.555 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946d59a700 1 -- 172.21.15.189:0/3194448153 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9454003240 con 0x7f947013a6e0 2020-11-20T12:15:22.555 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946d59a700 1 -- 172.21.15.189:0/3194448153 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3640544133 0 0) 0x7f9470067730 con 0x7f94700ad3e0 2020-11-20T12:15:22.555 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946d59a700 1 -- 172.21.15.189:0/3194448153 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f947013dd40 con 0x7f94700ad3e0 2020-11-20T12:15:22.556 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946d59a700 1 -- 172.21.15.189:0/3194448153 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2833068940 0 0) 0x7f9454003240 con 0x7f947013a6e0 2020-11-20T12:15:22.556 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946d59a700 1 -- 172.21.15.189:0/3194448153 >> v2:172.21.15.47:3300/0 conn(0x7f94700ad000 msgr2=0x7f947013e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:22.556 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946d59a700 1 --2- 172.21.15.189:0/3194448153 >> v2:172.21.15.47:3300/0 conn(0x7f94700ad000 0x7f947013e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.556 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946d59a700 1 -- 172.21.15.189:0/3194448153 >> v1:172.21.15.47:6789/0 conn(0x7f94700ad3e0 legacy=0x7f9470066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:22.556 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946d59a700 1 -- 172.21.15.189:0/3194448153 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f947013dfe0 con 0x7f947013a6e0 2020-11-20T12:15:22.557 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946d59a700 1 -- 172.21.15.189:0/3194448153 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f9468001860 con 0x7f947013a6e0 2020-11-20T12:15:22.557 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946d59a700 1 -- 172.21.15.189:0/3194448153 >> v1:172.21.15.189:6789/0 conn(0x7f947013a6e0 legacy=0x7f947013ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:22.557 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946d59a700 1 --2- 172.21.15.189:0/3194448153 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9454003910 0x7f9454003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.557 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946d59a700 1 --2- 172.21.15.189:0/3194448153 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94540044e0 0x7f9454004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.558 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946d59a700 1 --2- 172.21.15.189:0/3194448153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9454004e50 0x7f945400b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.558 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946d59a700 1 -- 172.21.15.189:0/3194448153 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9470138190 con 0x7f94540044e0 2020-11-20T12:15:22.558 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946d59a700 1 -- 172.21.15.189:0/3194448153 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945400bc20 con 0x7f9454004e50 2020-11-20T12:15:22.558 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946d59a700 1 -- 172.21.15.189:0/3194448153 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945400bd60 con 0x7f9454003910 2020-11-20T12:15:22.558 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946ed9d700 1 --2- 172.21.15.189:0/3194448153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9454004e50 0x7f945400b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.558 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946e59c700 1 --2- 172.21.15.189:0/3194448153 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9454003910 0x7f9454003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.559 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946dd9b700 1 --2- 172.21.15.189:0/3194448153 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94540044e0 0x7f9454004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.563 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946dd9b700 1 -- 172.21.15.189:0/3194448153 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9454003910 msgr2=0x7f9454003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:22.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946dd9b700 1 --2- 172.21.15.189:0/3194448153 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f9454003910 0x7f9454003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946dd9b700 1 -- 172.21.15.189:0/3194448153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9454004e50 msgr2=0x7f945400b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:22.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946dd9b700 1 --2- 172.21.15.189:0/3194448153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9454004e50 0x7f945400b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946dd9b700 1 -- 172.21.15.189:0/3194448153 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9454003240 con 0x7f94540044e0 2020-11-20T12:15:22.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946ed9d700 1 --2- 172.21.15.189:0/3194448153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9454004e50 0x7f945400b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:15:22.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946ed9d700 1 -- 172.21.15.189:0/3194448153 reap_dead start 2020-11-20T12:15:22.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946dd9b700 1 --2- 172.21.15.189:0/3194448153 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94540044e0 0x7f9454004960 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f9464001a80 tx=0x7f9464007b90).ready entity=mon.0 client_cookie=a33c1be7f1aa3c0b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:22.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946d59a700 1 -- 172.21.15.189:0/3194448153 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9464007020 con 0x7f94540044e0 2020-11-20T12:15:22.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946d59a700 1 -- 172.21.15.189:0/3194448153 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9464007180 con 0x7f94540044e0 2020-11-20T12:15:22.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.549+0000 7f946d59a700 1 -- 172.21.15.189:0/3194448153 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946400c620 con 0x7f94540044e0 2020-11-20T12:15:22.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946f59e700 1 -- 172.21.15.189:0/3194448153 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94540044e0 msgr2=0x7f9454004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:22.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946f59e700 1 --2- 172.21.15.189:0/3194448153 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94540044e0 0x7f9454004960 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f9464001a80 tx=0x7f9464007b90).stop 2020-11-20T12:15:22.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946f59e700 1 -- 172.21.15.189:0/3194448153 shutdown_connections 2020-11-20T12:15:22.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946f59e700 1 --2- 172.21.15.189:0/3194448153 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94540044e0 0x7f9454004960 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946f59e700 1 -- 172.21.15.189:0/3194448153 >> 172.21.15.189:0/3194448153 conn(0x7f9470062080 msgr2=0x7f9470062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:22.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946f59e700 1 -- 172.21.15.189:0/3194448153 shutdown_connections 2020-11-20T12:15:22.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946f59e700 1 -- 172.21.15.189:0/3194448153 wait complete. 2020-11-20T12:15:22.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946f59e700 1 Processor -- start 2020-11-20T12:15:22.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946f59e700 1 -- start start 2020-11-20T12:15:22.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946f59e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94540044e0 0x7f94700b0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946f59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94700a9e60 0x7f94700aa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946f59e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f94700a81c0 0x7f94700a8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946f59e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94700a6fe0 con 0x7f94540044e0 2020-11-20T12:15:22.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946f59e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94700aaf80 con 0x7f94700a9e60 2020-11-20T12:15:22.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946f59e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94700a68a0 con 0x7f94700a81c0 2020-11-20T12:15:22.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946dd9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94700a9e60 0x7f94700aa2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946dd9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94700a9e60 0x7f94700aa2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50944/0 (socket says 172.21.15.189:50944) 2020-11-20T12:15:22.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946dd9b700 1 -- 172.21.15.189:0/1438334856 learned_addr learned my addr 172.21.15.189:0/1438334856 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:15:22.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946ed9d700 1 --2- 172.21.15.189:0/1438334856 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f94700a81c0 0x7f94700a8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946e59c700 1 --2- 172.21.15.189:0/1438334856 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94540044e0 0x7f94700b0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946dd9b700 1 -- 172.21.15.189:0/1438334856 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f94700a81c0 msgr2=0x7f94700a8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:22.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946dd9b700 1 --2- 172.21.15.189:0/1438334856 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f94700a81c0 0x7f94700a8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946dd9b700 1 -- 172.21.15.189:0/1438334856 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94540044e0 msgr2=0x7f94700b0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:22.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946dd9b700 1 --2- 172.21.15.189:0/1438334856 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94540044e0 0x7f94700b0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946dd9b700 1 -- 172.21.15.189:0/1438334856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94700a6b40 con 0x7f94700a9e60 2020-11-20T12:15:22.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946ed9d700 1 --2- 172.21.15.189:0/1438334856 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f94700a81c0 0x7f94700a8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:15:22.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.553+0000 7f946e59c700 1 --2- 172.21.15.189:0/1438334856 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94540044e0 0x7f94700b0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:15:22.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.557+0000 7f946dd9b700 1 --2- 172.21.15.189:0/1438334856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94700a9e60 0x7f94700aa2e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f946400c720 tx=0x7f9464015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:22.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.557+0000 7f945e7fc700 1 -- 172.21.15.189:0/1438334856 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94640146c0 con 0x7f94700a9e60 2020-11-20T12:15:22.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.557+0000 7f945e7fc700 1 -- 172.21.15.189:0/1438334856 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9464014820 con 0x7f94700a9e60 2020-11-20T12:15:22.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.557+0000 7f946f59e700 1 -- 172.21.15.189:0/1438334856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94700a9260 con 0x7f94700a9e60 2020-11-20T12:15:22.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.557+0000 7f945e7fc700 1 -- 172.21.15.189:0/1438334856 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9464014bc0 con 0x7f94700a9e60 2020-11-20T12:15:22.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.557+0000 7f946f59e700 1 -- 172.21.15.189:0/1438334856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f947013b7d0 con 0x7f94700a9e60 2020-11-20T12:15:22.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.557+0000 7f946f59e700 1 -- 172.21.15.189:0/1438334856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94700a6cd0 con 0x7f94700a9e60 2020-11-20T12:15:22.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.557+0000 7f945e7fc700 1 -- 172.21.15.189:0/1438334856 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f9464025020 con 0x7f94700a9e60 2020-11-20T12:15:22.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.557+0000 7f945e7fc700 1 --2- 172.21.15.189:0/1438334856 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f94440557f0 0x7f9444057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.557+0000 7f945e7fc700 1 -- 172.21.15.189:0/1438334856 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6675+0+0 (secure 0 0 0) 0x7f946406f910 con 0x7f94700a9e60 2020-11-20T12:15:22.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.557+0000 7f946e59c700 1 --2- 172.21.15.189:0/1438334856 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f94440557f0 0x7f9444057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.557+0000 7f946e59c700 1 --2- 172.21.15.189:0/1438334856 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f94440557f0 0x7f9444057cb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f9460006d00 tx=0x7f9460008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:22.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.569+0000 7f945e7fc700 1 -- 172.21.15.189:0/1438334856 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9464045550 con 0x7f94700a9e60 2020-11-20T12:15:22.794 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.789+0000 7f946f59e700 1 -- 172.21.15.189:0/1438334856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94700b13d0 con 0x7f94700a9e60 2020-11-20T12:15:22.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.789+0000 7f945e7fc700 1 -- 172.21.15.189:0/1438334856 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+1784 (secure 0 0 0) 0x7f946401a090 con 0x7f94700a9e60 2020-11-20T12:15:22.795 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:15:22.796 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:15:22.691781+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5389":{"gid":5389,"name":"a","rank":0,"incarnation":42,"state":"up:replay","state_seq":1,"addr":"172.21.15.189:6835/636533719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":636533719},{"type":"v1","addr":"172.21.15.189:6835","nonce":636533719}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:15:22.797 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.793+0000 7f946f59e700 1 -- 172.21.15.189:0/1438334856 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f94440557f0 msgr2=0x7f9444057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:22.797 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.793+0000 7f946f59e700 1 --2- 172.21.15.189:0/1438334856 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f94440557f0 0x7f9444057cb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f9460006d00 tx=0x7f9460008040).stop 2020-11-20T12:15:22.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.793+0000 7f946f59e700 1 -- 172.21.15.189:0/1438334856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94700a9e60 msgr2=0x7f94700aa2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:22.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.793+0000 7f946f59e700 1 --2- 172.21.15.189:0/1438334856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94700a9e60 0x7f94700aa2e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f946400c720 tx=0x7f9464015000).stop 2020-11-20T12:15:22.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.793+0000 7f946f59e700 1 -- 172.21.15.189:0/1438334856 shutdown_connections 2020-11-20T12:15:22.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.793+0000 7f946f59e700 1 --2- 172.21.15.189:0/1438334856 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f94440557f0 0x7f9444057cb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.793+0000 7f946f59e700 1 --2- 172.21.15.189:0/1438334856 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f94540044e0 0x7f94700b0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.793+0000 7f946f59e700 1 --2- 172.21.15.189:0/1438334856 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f94700a81c0 0x7f94700a8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.793+0000 7f946f59e700 1 --2- 172.21.15.189:0/1438334856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94700a9e60 0x7f94700aa2e0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.793+0000 7f946f59e700 1 -- 172.21.15.189:0/1438334856 >> 172.21.15.189:0/1438334856 conn(0x7f9470062080 msgr2=0x7f94700aba30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:22.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.793+0000 7f946f59e700 1 -- 172.21.15.189:0/1438334856 shutdown_connections 2020-11-20T12:15:22.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.793+0000 7f946f59e700 1 -- 172.21.15.189:0/1438334856 wait complete. 2020-11-20T12:15:22.800 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 42 2020-11-20T12:15:22.809 INFO:tasks.mds_thrash.fs.[None]:mds.b down, removed from mdsmap 2020-11-20T12:15:22.809 INFO:tasks.mds_thrash.fs.[None]:waiting for mds cluster to stabilize... 2020-11-20T12:15:22.810 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:22.912 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e9680700 1 Processor -- start 2020-11-20T12:15:22.912 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e9680700 1 -- start start 2020-11-20T12:15:22.915 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e9680700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f87e40a9290 0x7f87e4134100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.916 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e9680700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e40663e0 con 0x7f87e40a9e20 2020-11-20T12:15:22.916 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e9680700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e4067b80 con 0x7f87e4068020 2020-11-20T12:15:22.916 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e9680700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e40a9970 con 0x7f87e40a9290 2020-11-20T12:15:22.916 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87dbfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f87e4068020 0x7f87e41339f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52582/0 (socket says 172.21.15.189:52582) 2020-11-20T12:15:22.916 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87dbfff700 1 -- 172.21.15.189:0/817715043 learned_addr learned my addr 172.21.15.189:0/817715043 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:15:22.917 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e37fe700 1 --2- 172.21.15.189:0/817715043 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f87e40a9290 0x7f87e4134100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.917 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e27fc700 1 -- 172.21.15.189:0/817715043 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3821132653 0 0) 0x7f87e4067b80 con 0x7f87e4068020 2020-11-20T12:15:22.917 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e27fc700 1 -- 172.21.15.189:0/817715043 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87c4003240 con 0x7f87e4068020 2020-11-20T12:15:22.917 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e27fc700 1 -- 172.21.15.189:0/817715043 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 361693680 0 0) 0x7f87c4003240 con 0x7f87e4068020 2020-11-20T12:15:22.918 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e27fc700 1 -- 172.21.15.189:0/817715043 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f87e40a9290 msgr2=0x7f87e4134100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:22.918 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e27fc700 1 --2- 172.21.15.189:0/817715043 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f87e40a9290 0x7f87e4134100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.918 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e27fc700 1 -- 172.21.15.189:0/817715043 >> v1:172.21.15.47:6789/0 conn(0x7f87e40a9e20 legacy=0x7f87e40a83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:22.918 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e27fc700 1 -- 172.21.15.189:0/817715043 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e4066680 con 0x7f87e4068020 2020-11-20T12:15:22.918 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e27fc700 1 -- 172.21.15.189:0/817715043 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f87cc0016a0 con 0x7f87e4068020 2020-11-20T12:15:22.919 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e27fc700 1 -- 172.21.15.189:0/817715043 >> v1:172.21.15.189:6789/0 conn(0x7f87e4068020 legacy=0x7f87e41339f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:22.919 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e27fc700 1 --2- 172.21.15.189:0/817715043 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87c4003730 0x7f87c4003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.919 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e27fc700 1 --2- 172.21.15.189:0/817715043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c40044e0 0x7f87c4004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.919 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e27fc700 1 --2- 172.21.15.189:0/817715043 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f87e40a9290 0x7f87c4004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.919 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e27fc700 1 -- 172.21.15.189:0/817715043 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e40a9970 con 0x7f87c4003730 2020-11-20T12:15:22.920 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e27fc700 1 -- 172.21.15.189:0/817715043 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c4004270 con 0x7f87c40044e0 2020-11-20T12:15:22.920 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e27fc700 1 -- 172.21.15.189:0/817715043 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87c4009720 con 0x7f87e40a9290 2020-11-20T12:15:22.920 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87dbfff700 1 --2- 172.21.15.189:0/817715043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c40044e0 0x7f87c4004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.920 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e37fe700 1 --2- 172.21.15.189:0/817715043 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f87e40a9290 0x7f87c4004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.920 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87dbfff700 1 -- 172.21.15.189:0/817715043 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f87e40a9290 msgr2=0x7f87c4004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:22.921 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e2ffd700 1 --2- 172.21.15.189:0/817715043 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87c4003730 0x7f87c4003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.921 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87dbfff700 1 --2- 172.21.15.189:0/817715043 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f87e40a9290 0x7f87c4004e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.921 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87dbfff700 1 -- 172.21.15.189:0/817715043 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87c4003730 msgr2=0x7f87c4003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:22.921 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87dbfff700 1 --2- 172.21.15.189:0/817715043 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87c4003730 0x7f87c4003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.921 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87dbfff700 1 -- 172.21.15.189:0/817715043 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c4003240 con 0x7f87c40044e0 2020-11-20T12:15:22.922 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e2ffd700 1 --2- 172.21.15.189:0/817715043 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87c4003730 0x7f87c4003b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:22.922 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87dbfff700 1 --2- 172.21.15.189:0/817715043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c40044e0 0x7f87c4004960 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f87cc007bd0 tx=0x7f87cc012bd0).ready entity=mon.1 client_cookie=cc9fe0b776383ded server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:22.922 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e27fc700 1 -- 172.21.15.189:0/817715043 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87cc017070 con 0x7f87c40044e0 2020-11-20T12:15:22.922 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e27fc700 1 -- 172.21.15.189:0/817715043 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87cc012c60 con 0x7f87c40044e0 2020-11-20T12:15:22.923 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.909+0000 7f87e27fc700 1 -- 172.21.15.189:0/817715043 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87cc007300 con 0x7f87c40044e0 2020-11-20T12:15:22.923 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87e9680700 1 -- 172.21.15.189:0/817715043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c40044e0 msgr2=0x7f87c4004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:22.923 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87e9680700 1 --2- 172.21.15.189:0/817715043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87c40044e0 0x7f87c4004960 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f87cc007bd0 tx=0x7f87cc012bd0).stop 2020-11-20T12:15:22.923 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87e37fe700 1 -- 172.21.15.189:0/817715043 reap_dead start 2020-11-20T12:15:22.923 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87e9680700 1 -- 172.21.15.189:0/817715043 shutdown_connections 2020-11-20T12:15:22.924 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87e9680700 1 -- 172.21.15.189:0/817715043 >> 172.21.15.189:0/817715043 conn(0x7f87e40627f0 msgr2=0x7f87e4065870 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:22.924 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87e9680700 1 -- 172.21.15.189:0/817715043 shutdown_connections 2020-11-20T12:15:22.924 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87e9680700 1 -- 172.21.15.189:0/817715043 wait complete. 2020-11-20T12:15:22.927 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87e9680700 1 Processor -- start 2020-11-20T12:15:22.927 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87e9680700 1 -- start start 2020-11-20T12:15:22.927 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87e9680700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f87e40ab8e0 0x7f87e4149190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.927 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87e9680700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e40abcc0 0x7f87e40a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.927 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87e9680700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87e413c8d0 0x7f87e40a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.928 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87e9680700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e40ac910 con 0x7f87e413c8d0 2020-11-20T12:15:22.928 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87e9680700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e40a4fa0 con 0x7f87e40abcc0 2020-11-20T12:15:22.928 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87e9680700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e4149830 con 0x7f87e40ab8e0 2020-11-20T12:15:22.928 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87dbfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e40abcc0 0x7f87e40a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.929 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87dbfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e40abcc0 0x7f87e40a45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50972/0 (socket says 172.21.15.189:50972) 2020-11-20T12:15:22.929 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87dbfff700 1 -- 172.21.15.189:0/1124227186 learned_addr learned my addr 172.21.15.189:0/1124227186 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:15:22.929 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87e2ffd700 1 --2- 172.21.15.189:0/1124227186 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f87e40ab8e0 0x7f87e4149190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.929 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87e37fe700 1 --2- 172.21.15.189:0/1124227186 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87e413c8d0 0x7f87e40a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.929 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87dbfff700 1 -- 172.21.15.189:0/1124227186 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f87e40ab8e0 msgr2=0x7f87e4149190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:22.930 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87dbfff700 1 --2- 172.21.15.189:0/1124227186 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f87e40ab8e0 0x7f87e4149190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.930 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87dbfff700 1 -- 172.21.15.189:0/1124227186 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87e413c8d0 msgr2=0x7f87e40a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:22.930 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87dbfff700 1 --2- 172.21.15.189:0/1124227186 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87e413c8d0 0x7f87e40a4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:22.930 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87dbfff700 1 -- 172.21.15.189:0/1124227186 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e4149ad0 con 0x7f87e40abcc0 2020-11-20T12:15:22.930 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87dbfff700 1 --2- 172.21.15.189:0/1124227186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e40abcc0 0x7f87e40a45c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f87cc002ea0 tx=0x7f87cc01d000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:22.931 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87daffd700 1 -- 172.21.15.189:0/1124227186 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87cc0070d0 con 0x7f87e40abcc0 2020-11-20T12:15:22.931 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87daffd700 1 -- 172.21.15.189:0/1124227186 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87cc007230 con 0x7f87e40abcc0 2020-11-20T12:15:22.931 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87e9680700 1 -- 172.21.15.189:0/1124227186 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e4149d70 con 0x7f87e40abcc0 2020-11-20T12:15:22.931 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87daffd700 1 -- 172.21.15.189:0/1124227186 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87cc014860 con 0x7f87e40abcc0 2020-11-20T12:15:22.932 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87e9680700 1 -- 172.21.15.189:0/1124227186 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e40a6c30 con 0x7f87e40abcc0 2020-11-20T12:15:22.932 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.913+0000 7f87e9680700 1 -- 172.21.15.189:0/1124227186 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e4149680 con 0x7f87e40abcc0 2020-11-20T12:15:22.932 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.917+0000 7f87daffd700 1 -- 172.21.15.189:0/1124227186 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f87cc02e030 con 0x7f87e40abcc0 2020-11-20T12:15:22.932 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.917+0000 7f87daffd700 1 --2- 172.21.15.189:0/1124227186 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f87c00557c0 0x7f87c0057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:22.932 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.917+0000 7f87daffd700 1 -- 172.21.15.189:0/1124227186 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6761+0+0 (secure 0 0 0) 0x7f87cc078020 con 0x7f87e40abcc0 2020-11-20T12:15:22.933 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.917+0000 7f87e2ffd700 1 --2- 172.21.15.189:0/1124227186 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f87c00557c0 0x7f87c0057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:22.933 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.917+0000 7f87e2ffd700 1 --2- 172.21.15.189:0/1124227186 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f87c00557c0 0x7f87c0057c80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f87d4003640 tx=0x7f87d4009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:22.933 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:22.921+0000 7f87daffd700 1 -- 172.21.15.189:0/1124227186 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f87cc04dcc0 con 0x7f87e40abcc0 2020-11-20T12:15:23.122 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:23.117+0000 7f87e9680700 1 -- 172.21.15.189:0/1124227186 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87e4149f00 con 0x7f87e40abcc0 2020-11-20T12:15:23.123 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:23.117+0000 7f87daffd700 1 -- 172.21.15.189:0/1124227186 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+1784 (secure 0 0 0) 0x7f87cc02c030 con 0x7f87e40abcc0 2020-11-20T12:15:23.123 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:15:23.123 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:15:22.691781+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5389":{"gid":5389,"name":"a","rank":0,"incarnation":42,"state":"up:replay","state_seq":1,"addr":"172.21.15.189:6835/636533719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":636533719},{"type":"v1","addr":"172.21.15.189:6835","nonce":636533719}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:15:23.125 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:23.121+0000 7f87e9680700 1 -- 172.21.15.189:0/1124227186 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f87c00557c0 msgr2=0x7f87c0057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:23.125 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:23.121+0000 7f87e9680700 1 --2- 172.21.15.189:0/1124227186 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f87c00557c0 0x7f87c0057c80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f87d4003640 tx=0x7f87d4009040).stop 2020-11-20T12:15:23.125 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:23.121+0000 7f87e9680700 1 -- 172.21.15.189:0/1124227186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e40abcc0 msgr2=0x7f87e40a45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:23.125 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:23.121+0000 7f87e9680700 1 --2- 172.21.15.189:0/1124227186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e40abcc0 0x7f87e40a45c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f87cc002ea0 tx=0x7f87cc01d000).stop 2020-11-20T12:15:23.126 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:23.121+0000 7f87e9680700 1 -- 172.21.15.189:0/1124227186 shutdown_connections 2020-11-20T12:15:23.126 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:23.121+0000 7f87e9680700 1 --2- 172.21.15.189:0/1124227186 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f87c00557c0 0x7f87c0057c80 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:23.126 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:23.121+0000 7f87e9680700 1 --2- 172.21.15.189:0/1124227186 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f87e413c8d0 0x7f87e40a4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:23.126 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:23.121+0000 7f87e9680700 1 --2- 172.21.15.189:0/1124227186 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f87e40ab8e0 0x7f87e4149190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:23.127 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:23.121+0000 7f87e9680700 1 --2- 172.21.15.189:0/1124227186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e40abcc0 0x7f87e40a45c0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:23.127 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:23.121+0000 7f87e9680700 1 -- 172.21.15.189:0/1124227186 >> 172.21.15.189:0/1124227186 conn(0x7f87e40627f0 msgr2=0x7f87e40a81d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:23.127 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:23.121+0000 7f87e9680700 1 -- 172.21.15.189:0/1124227186 shutdown_connections 2020-11-20T12:15:23.127 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:23.121+0000 7f87e9680700 1 -- 172.21.15.189:0/1124227186 wait complete. 2020-11-20T12:15:23.128 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 42 2020-11-20T12:15:23.138 INFO:tasks.mds_thrash.fs.[None]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 42, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2020-11-20T12:04:54.012110+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 42, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5389": { "addr": "172.21.15.189:6835/636533719", "addrs": { "addrvec": [ { "addr": "172.21.15.189:6834", "nonce": 636533719, "type": "v2" }, { "addr": "172.21.15.189:6835", "nonce": 636533719, "type": "v1" } ] }, "export_targets": [], "features": 4540138297136906239, "flags": 0, "gid": 5389, "incarnation": 42, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2020-11-20T12:15:22.691781+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5389 } } } ], "standbys": [] } 2020-11-20T12:15:24.640 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:15:24.648 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:15:25.139 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:25.274 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.269+0000 7f7b0d5de700 1 Processor -- start 2020-11-20T12:15:25.274 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.269+0000 7f7b0d5de700 1 -- start start 2020-11-20T12:15:25.276 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.269+0000 7f7b0d5de700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7b0813b220 0x7f7b081461d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:25.276 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.269+0000 7f7b0d5de700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b0813e320 con 0x7f7b080a8840 2020-11-20T12:15:25.276 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.269+0000 7f7b0d5de700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b08066800 con 0x7f7b0813a680 2020-11-20T12:15:25.276 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.269+0000 7f7b0d5de700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b08066aa0 con 0x7f7b0813b220 2020-11-20T12:15:25.277 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.269+0000 7f7b077fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f7b0813a680 0x7f7b0813ab10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52604/0 (socket says 172.21.15.189:52604) 2020-11-20T12:15:25.277 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.269+0000 7f7b077fe700 1 -- 172.21.15.189:0/175202969 learned_addr learned my addr 172.21.15.189:0/175202969 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:15:25.277 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b0cddd700 1 --2- 172.21.15.189:0/175202969 >> v2:172.21.15.47:3300/0 conn(0x7f7b0813b220 0x7f7b081461d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:25.277 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b06ffd700 1 -- 172.21.15.189:0/175202969 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3588535946 0 0) 0x7f7b08066800 con 0x7f7b0813a680 2020-11-20T12:15:25.278 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b06ffd700 1 -- 172.21.15.189:0/175202969 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ae8003240 con 0x7f7b0813a680 2020-11-20T12:15:25.278 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b06ffd700 1 -- 172.21.15.189:0/175202969 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 474338085 0 0) 0x7f7b0813e320 con 0x7f7b080a8840 2020-11-20T12:15:25.278 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b06ffd700 1 -- 172.21.15.189:0/175202969 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b08066800 con 0x7f7b080a8840 2020-11-20T12:15:25.278 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b06ffd700 1 -- 172.21.15.189:0/175202969 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4271896552 0 0) 0x7f7ae8003240 con 0x7f7b0813a680 2020-11-20T12:15:25.278 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b06ffd700 1 -- 172.21.15.189:0/175202969 >> v2:172.21.15.47:3300/0 conn(0x7f7b0813b220 msgr2=0x7f7b081461d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:25.279 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b06ffd700 1 --2- 172.21.15.189:0/175202969 >> v2:172.21.15.47:3300/0 conn(0x7f7b0813b220 0x7f7b081461d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:25.283 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b06ffd700 1 -- 172.21.15.189:0/175202969 >> v1:172.21.15.47:6789/0 conn(0x7f7b080a8840 legacy=0x7f7b0813dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:25.283 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b06ffd700 1 -- 172.21.15.189:0/175202969 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b080672f0 con 0x7f7b0813a680 2020-11-20T12:15:25.283 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b06ffd700 1 -- 172.21.15.189:0/175202969 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f7af80014e0 con 0x7f7b0813a680 2020-11-20T12:15:25.284 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b06ffd700 1 -- 172.21.15.189:0/175202969 >> v1:172.21.15.189:6789/0 conn(0x7f7b0813a680 legacy=0x7f7b0813ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:25.284 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b06ffd700 1 --2- 172.21.15.189:0/175202969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae80038c0 0x7f7ae8003cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:25.284 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b06ffd700 1 --2- 172.21.15.189:0/175202969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ae80044e0 0x7f7ae8004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:25.284 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b06ffd700 1 --2- 172.21.15.189:0/175202969 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7ae8004e50 0x7f7ae800b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:25.284 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b06ffd700 1 -- 172.21.15.189:0/175202969 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b08066aa0 con 0x7f7ae80044e0 2020-11-20T12:15:25.285 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b06ffd700 1 -- 172.21.15.189:0/175202969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae80053b0 con 0x7f7ae80038c0 2020-11-20T12:15:25.285 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b06ffd700 1 -- 172.21.15.189:0/175202969 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae800bca0 con 0x7f7ae8004e50 2020-11-20T12:15:25.285 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b07fff700 1 --2- 172.21.15.189:0/175202969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae80038c0 0x7f7ae8003cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:25.285 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b077fe700 1 --2- 172.21.15.189:0/175202969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ae80044e0 0x7f7ae8004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:25.285 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b0cddd700 1 --2- 172.21.15.189:0/175202969 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7ae8004e50 0x7f7ae800b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:25.286 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b07fff700 1 -- 172.21.15.189:0/175202969 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7ae8004e50 msgr2=0x7f7ae800b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:25.286 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b07fff700 1 --2- 172.21.15.189:0/175202969 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7ae8004e50 0x7f7ae800b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:25.286 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b07fff700 1 -- 172.21.15.189:0/175202969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ae80044e0 msgr2=0x7f7ae8004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:25.286 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b07fff700 1 --2- 172.21.15.189:0/175202969 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7ae80044e0 0x7f7ae8004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:25.286 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b0cddd700 1 -- 172.21.15.189:0/175202969 reap_dead start 2020-11-20T12:15:25.287 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b07fff700 1 -- 172.21.15.189:0/175202969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae8003240 con 0x7f7ae80038c0 2020-11-20T12:15:25.287 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b07fff700 1 --2- 172.21.15.189:0/175202969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae80038c0 0x7f7ae8003cc0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f7af0001a80 tx=0x7f7af0007b90).ready entity=mon.1 client_cookie=116a87290bf5605e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:25.287 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b06ffd700 1 -- 172.21.15.189:0/175202969 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af0007020 con 0x7f7ae80038c0 2020-11-20T12:15:25.287 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b06ffd700 1 -- 172.21.15.189:0/175202969 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af0007180 con 0x7f7ae80038c0 2020-11-20T12:15:25.287 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b06ffd700 1 -- 172.21.15.189:0/175202969 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af000c7f0 con 0x7f7ae80038c0 2020-11-20T12:15:25.288 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b0d5de700 1 -- 172.21.15.189:0/175202969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae80038c0 msgr2=0x7f7ae8003cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:25.288 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b0d5de700 1 --2- 172.21.15.189:0/175202969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae80038c0 0x7f7ae8003cc0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f7af0001a80 tx=0x7f7af0007b90).stop 2020-11-20T12:15:25.288 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b0d5de700 1 -- 172.21.15.189:0/175202969 shutdown_connections 2020-11-20T12:15:25.288 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b0d5de700 1 --2- 172.21.15.189:0/175202969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae80038c0 0x7f7ae8003cc0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:25.288 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b0d5de700 1 -- 172.21.15.189:0/175202969 >> 172.21.15.189:0/175202969 conn(0x7f7b08060170 msgr2=0x7f7b08060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:25.289 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b0d5de700 1 -- 172.21.15.189:0/175202969 shutdown_connections 2020-11-20T12:15:25.289 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b0d5de700 1 -- 172.21.15.189:0/175202969 wait complete. 2020-11-20T12:15:25.289 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b0d5de700 1 Processor -- start 2020-11-20T12:15:25.289 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.273+0000 7f7b0d5de700 1 -- start start 2020-11-20T12:15:25.290 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.277+0000 7f7b0d5de700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae80038c0 0x7f7b080a5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:25.290 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.277+0000 7f7b0d5de700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7b0814a860 0x7f7b0813d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:25.290 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.277+0000 7f7b0d5de700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b080a5770 0x7f7b08137690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:25.290 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.277+0000 7f7b0d5de700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b080a6d00 con 0x7f7b080a5770 2020-11-20T12:15:25.290 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.277+0000 7f7b0d5de700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b080a3e20 con 0x7f7ae80038c0 2020-11-20T12:15:25.291 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.277+0000 7f7b0d5de700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b08149e80 con 0x7f7b0814a860 2020-11-20T12:15:25.291 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.277+0000 7f7b07fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae80038c0 0x7f7b080a5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:25.291 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.277+0000 7f7b07fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae80038c0 0x7f7b080a5c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:50992/0 (socket says 172.21.15.189:50992) 2020-11-20T12:15:25.291 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.277+0000 7f7b07fff700 1 -- 172.21.15.189:0/86594107 learned_addr learned my addr 172.21.15.189:0/86594107 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:15:25.291 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.277+0000 7f7b077fe700 1 --2- 172.21.15.189:0/86594107 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7b0814a860 0x7f7b0813d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:25.292 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.277+0000 7f7b0cddd700 1 --2- 172.21.15.189:0/86594107 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b080a5770 0x7f7b08137690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:25.292 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.277+0000 7f7b07fff700 1 -- 172.21.15.189:0/86594107 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7b0814a860 msgr2=0x7f7b0813d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:25.292 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.277+0000 7f7b07fff700 1 --2- 172.21.15.189:0/86594107 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7b0814a860 0x7f7b0813d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:25.292 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.277+0000 7f7b07fff700 1 -- 172.21.15.189:0/86594107 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b080a5770 msgr2=0x7f7b08137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:25.292 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.277+0000 7f7b07fff700 1 --2- 172.21.15.189:0/86594107 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b080a5770 0x7f7b08137690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:25.293 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.277+0000 7f7b07fff700 1 -- 172.21.15.189:0/86594107 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b0814a120 con 0x7f7ae80038c0 2020-11-20T12:15:25.293 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.277+0000 7f7b077fe700 1 --2- 172.21.15.189:0/86594107 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7b0814a860 0x7f7b0813d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:15:25.293 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.277+0000 7f7b07fff700 1 --2- 172.21.15.189:0/86594107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae80038c0 0x7f7b080a5c80 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f7ae800bc70 tx=0x7f7af0016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:25.293 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.277+0000 7f7afffff700 1 -- 172.21.15.189:0/86594107 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af000c7f0 con 0x7f7ae80038c0 2020-11-20T12:15:25.293 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.277+0000 7f7afffff700 1 -- 172.21.15.189:0/86594107 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af0012c60 con 0x7f7ae80038c0 2020-11-20T12:15:25.294 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.277+0000 7f7b0d5de700 1 -- 172.21.15.189:0/86594107 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b0814a3c0 con 0x7f7ae80038c0 2020-11-20T12:15:25.294 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.277+0000 7f7afffff700 1 -- 172.21.15.189:0/86594107 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af001d440 con 0x7f7ae80038c0 2020-11-20T12:15:25.294 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.277+0000 7f7b0d5de700 1 -- 172.21.15.189:0/86594107 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b0813dc30 con 0x7f7ae80038c0 2020-11-20T12:15:25.294 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.277+0000 7f7afffff700 1 -- 172.21.15.189:0/86594107 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f7af0024030 con 0x7f7ae80038c0 2020-11-20T12:15:25.295 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.281+0000 7f7b0d5de700 1 -- 172.21.15.189:0/86594107 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b0814a550 con 0x7f7ae80038c0 2020-11-20T12:15:25.296 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.281+0000 7f7afffff700 1 --2- 172.21.15.189:0/86594107 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f7ae40557c0 0x7f7ae4057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:25.296 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.281+0000 7f7afffff700 1 -- 172.21.15.189:0/86594107 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6761+0+0 (secure 0 0 0) 0x7f7af006fa90 con 0x7f7ae80038c0 2020-11-20T12:15:25.296 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.281+0000 7f7b077fe700 1 --2- 172.21.15.189:0/86594107 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f7ae40557c0 0x7f7ae4057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:25.296 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.281+0000 7f7b077fe700 1 --2- 172.21.15.189:0/86594107 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f7ae40557c0 0x7f7ae4057c80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f7b080a3d40 tx=0x7f7af8009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:25.296 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.289+0000 7f7afffff700 1 -- 172.21.15.189:0/86594107 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7af0045770 con 0x7f7ae80038c0 2020-11-20T12:15:25.492 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.489+0000 7f7b0d5de700 1 -- 172.21.15.189:0/86594107 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b080a9230 con 0x7f7ae80038c0 2020-11-20T12:15:25.493 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.489+0000 7f7afffff700 1 -- 172.21.15.189:0/86594107 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+1785 (secure 0 0 0) 0x7f7af0022070 con 0x7f7ae80038c0 2020-11-20T12:15:25.493 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:15:25.493 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:15:24.729059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5389":{"gid":5389,"name":"a","rank":0,"incarnation":42,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.189:6835/636533719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":636533719},{"type":"v1","addr":"172.21.15.189:6835","nonce":636533719}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:15:25.494 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.489+0000 7f7b0d5de700 1 -- 172.21.15.189:0/86594107 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f7ae40557c0 msgr2=0x7f7ae4057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:25.495 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.489+0000 7f7b0d5de700 1 --2- 172.21.15.189:0/86594107 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f7ae40557c0 0x7f7ae4057c80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f7b080a3d40 tx=0x7f7af8009040).stop 2020-11-20T12:15:25.495 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.489+0000 7f7b0d5de700 1 -- 172.21.15.189:0/86594107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae80038c0 msgr2=0x7f7b080a5c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:25.495 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.489+0000 7f7b0d5de700 1 --2- 172.21.15.189:0/86594107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae80038c0 0x7f7b080a5c80 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f7ae800bc70 tx=0x7f7af0016000).stop 2020-11-20T12:15:25.495 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.489+0000 7f7b0d5de700 1 -- 172.21.15.189:0/86594107 shutdown_connections 2020-11-20T12:15:25.495 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.489+0000 7f7b0d5de700 1 --2- 172.21.15.189:0/86594107 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f7ae40557c0 0x7f7ae4057c80 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:25.496 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.489+0000 7f7b0d5de700 1 --2- 172.21.15.189:0/86594107 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7b080a5770 0x7f7b08137690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:25.496 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.489+0000 7f7b0d5de700 1 --2- 172.21.15.189:0/86594107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ae80038c0 0x7f7b080a5c80 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:25.496 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.489+0000 7f7b0d5de700 1 --2- 172.21.15.189:0/86594107 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7b0814a860 0x7f7b0813d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:25.496 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.489+0000 7f7b0d5de700 1 -- 172.21.15.189:0/86594107 >> 172.21.15.189:0/86594107 conn(0x7f7b08060170 msgr2=0x7f7b0813ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:25.497 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.489+0000 7f7b0d5de700 1 -- 172.21.15.189:0/86594107 shutdown_connections 2020-11-20T12:15:25.497 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:25.489+0000 7f7b0d5de700 1 -- 172.21.15.189:0/86594107 wait complete. 2020-11-20T12:15:25.497 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 44 2020-11-20T12:15:25.508 INFO:tasks.mds_thrash.fs.[None]:no change 2020-11-20T12:15:26.707 INFO:tasks.ceph.mds.a.smithi189.stderr:2020-11-20T12:15:26.701+0000 7f15c71de700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:15:27.509 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:27.636 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.633+0000 7fcc65e79700 1 Processor -- start 2020-11-20T12:15:27.637 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.633+0000 7fcc65e79700 1 -- start start 2020-11-20T12:15:27.638 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.633+0000 7fcc65e79700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcc600ad000 0x7fcc6013e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:27.638 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.633+0000 7fcc65e79700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc600ad3e0 0x7fcc60066d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:27.638 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.633+0000 7fcc65e79700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc60067730 con 0x7fcc6013a6e0 2020-11-20T12:15:27.639 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.633+0000 7fcc65e79700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc60138190 con 0x7fcc600ad000 2020-11-20T12:15:27.639 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.633+0000 7fcc65e79700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6013dd40 con 0x7fcc600ad3e0 2020-11-20T12:15:27.641 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.633+0000 7fcc65678700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fcc6013a6e0 0x7fcc6013ad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52628/0 (socket says 172.21.15.189:52628) 2020-11-20T12:15:27.641 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.633+0000 7fcc65678700 1 -- 172.21.15.189:0/2811425245 learned_addr learned my addr 172.21.15.189:0/2811425245 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:15:27.641 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.633+0000 7fcc64e77700 1 --2- 172.21.15.189:0/2811425245 >> v2:172.21.15.189:3300/0 conn(0x7fcc600ad000 0x7fcc6013e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:27.641 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.633+0000 7fcc5ffff700 1 --2- 172.21.15.189:0/2811425245 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc600ad3e0 0x7fcc60066d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:27.642 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.633+0000 7fcc64e77700 1 -- 172.21.15.189:0/2811425245 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc600ad3e0 msgr2=0x7fcc60066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:27.642 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.633+0000 7fcc64e77700 1 --2- 172.21.15.189:0/2811425245 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc600ad3e0 0x7fcc60066d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:27.642 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.633+0000 7fcc64e77700 1 -- 172.21.15.189:0/2811425245 >> v1:172.21.15.189:6789/0 conn(0x7fcc6013a6e0 legacy=0x7fcc6013ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:27.642 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.633+0000 7fcc5f7fe700 1 -- 172.21.15.189:0/2811425245 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1963896349 0 0) 0x7fcc60067730 con 0x7fcc6013a6e0 2020-11-20T12:15:27.643 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.633+0000 7fcc64e77700 1 -- 172.21.15.189:0/2811425245 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6013dfe0 con 0x7fcc600ad000 2020-11-20T12:15:27.643 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.633+0000 7fcc64e77700 1 --2- 172.21.15.189:0/2811425245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc600ad000 0x7fcc6013e480 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7fcc5400b9a0 tx=0x7fcc54006df0).ready entity=mon.1 client_cookie=65fa88612e36bfe3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:27.643 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.633+0000 7fcc5f7fe700 1 -- 172.21.15.189:0/2811425245 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc540097b0 con 0x7fcc600ad000 2020-11-20T12:15:27.643 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.633+0000 7fcc5f7fe700 1 -- 172.21.15.189:0/2811425245 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc54009910 con 0x7fcc600ad000 2020-11-20T12:15:27.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.633+0000 7fcc5f7fe700 1 -- 172.21.15.189:0/2811425245 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc54009c90 con 0x7fcc600ad000 2020-11-20T12:15:27.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.637+0000 7fcc65e79700 1 -- 172.21.15.189:0/2811425245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc600ad000 msgr2=0x7fcc6013e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:27.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.637+0000 7fcc65e79700 1 --2- 172.21.15.189:0/2811425245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc600ad000 0x7fcc6013e480 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7fcc5400b9a0 tx=0x7fcc54006df0).stop 2020-11-20T12:15:27.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.637+0000 7fcc65e79700 1 -- 172.21.15.189:0/2811425245 shutdown_connections 2020-11-20T12:15:27.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.637+0000 7fcc65e79700 1 --2- 172.21.15.189:0/2811425245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc600ad000 0x7fcc6013e480 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:27.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.637+0000 7fcc65e79700 1 --2- 172.21.15.189:0/2811425245 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc600ad3e0 0x7fcc60066d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:27.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.637+0000 7fcc65e79700 1 -- 172.21.15.189:0/2811425245 >> 172.21.15.189:0/2811425245 conn(0x7fcc60062080 msgr2=0x7fcc60062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:27.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.637+0000 7fcc65e79700 1 -- 172.21.15.189:0/2811425245 shutdown_connections 2020-11-20T12:15:27.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.637+0000 7fcc65e79700 1 -- 172.21.15.189:0/2811425245 wait complete. 2020-11-20T12:15:27.648 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.637+0000 7fcc65e79700 1 Processor -- start 2020-11-20T12:15:27.648 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.637+0000 7fcc65e79700 1 -- start start 2020-11-20T12:15:27.649 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.637+0000 7fcc65e79700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc600ad000 0x7fcc60134650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:27.649 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.637+0000 7fcc65e79700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc600ad3e0 0x7fcc6013ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:27.649 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.637+0000 7fcc65e79700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc6013f270 0x7fcc6013ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:27.649 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.637+0000 7fcc65e79700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6013f6b0 con 0x7fcc600ad000 2020-11-20T12:15:27.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.637+0000 7fcc65e79700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc600af3d0 con 0x7fcc6013f270 2020-11-20T12:15:27.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.637+0000 7fcc65e79700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc600af630 con 0x7fcc600ad3e0 2020-11-20T12:15:27.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.637+0000 7fcc65678700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc6013f270 0x7fcc6013ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:27.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.637+0000 7fcc65678700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc6013f270 0x7fcc6013ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51012/0 (socket says 172.21.15.189:51012) 2020-11-20T12:15:27.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.637+0000 7fcc65678700 1 -- 172.21.15.189:0/1592134493 learned_addr learned my addr 172.21.15.189:0/1592134493 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:15:27.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.637+0000 7fcc5ffff700 1 --2- 172.21.15.189:0/1592134493 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc600ad3e0 0x7fcc6013ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:27.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.637+0000 7fcc64e77700 1 --2- 172.21.15.189:0/1592134493 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc600ad000 0x7fcc60134650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:27.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.641+0000 7fcc65678700 1 -- 172.21.15.189:0/1592134493 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc600ad3e0 msgr2=0x7fcc6013ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:27.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.641+0000 7fcc65678700 1 --2- 172.21.15.189:0/1592134493 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc600ad3e0 0x7fcc6013ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:27.652 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.641+0000 7fcc65678700 1 -- 172.21.15.189:0/1592134493 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc600ad000 msgr2=0x7fcc60134650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:27.652 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.641+0000 7fcc65678700 1 --2- 172.21.15.189:0/1592134493 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc600ad000 0x7fcc60134650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:27.652 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.641+0000 7fcc65678700 1 -- 172.21.15.189:0/1592134493 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc600af8d0 con 0x7fcc6013f270 2020-11-20T12:15:27.652 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.641+0000 7fcc65678700 1 --2- 172.21.15.189:0/1592134493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc6013f270 0x7fcc6013ba60 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7fcc5800cae0 tx=0x7fcc58007d40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:27.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.641+0000 7fcc5cff9700 1 -- 172.21.15.189:0/1592134493 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc58005b80 con 0x7fcc6013f270 2020-11-20T12:15:27.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.641+0000 7fcc5cff9700 1 -- 172.21.15.189:0/1592134493 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc58005ce0 con 0x7fcc6013f270 2020-11-20T12:15:27.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.641+0000 7fcc65e79700 1 -- 172.21.15.189:0/1592134493 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc60138420 con 0x7fcc6013f270 2020-11-20T12:15:27.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.641+0000 7fcc5cff9700 1 -- 172.21.15.189:0/1592134493 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc58007320 con 0x7fcc6013f270 2020-11-20T12:15:27.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.641+0000 7fcc65e79700 1 -- 172.21.15.189:0/1592134493 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc60135240 con 0x7fcc6013f270 2020-11-20T12:15:27.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.641+0000 7fcc65e79700 1 -- 172.21.15.189:0/1592134493 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc600aff90 con 0x7fcc6013f270 2020-11-20T12:15:27.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.641+0000 7fcc5cff9700 1 -- 172.21.15.189:0/1592134493 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fcc58010030 con 0x7fcc6013f270 2020-11-20T12:15:27.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.641+0000 7fcc5cff9700 1 --2- 172.21.15.189:0/1592134493 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcc380557f0 0x7fcc38057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:27.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.641+0000 7fcc5cff9700 1 -- 172.21.15.189:0/1592134493 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6761+0+0 (secure 0 0 0) 0x7fcc5806e940 con 0x7fcc6013f270 2020-11-20T12:15:27.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.641+0000 7fcc64e77700 1 --2- 172.21.15.189:0/1592134493 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcc380557f0 0x7fcc38057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:27.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.645+0000 7fcc64e77700 1 --2- 172.21.15.189:0/1592134493 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcc380557f0 0x7fcc38057cb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fcc54006e30 tx=0x7fcc54015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:27.656 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.649+0000 7fcc5cff9700 1 -- 172.21.15.189:0/1592134493 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcc58044530 con 0x7fcc6013f270 2020-11-20T12:15:27.886 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.881+0000 7fcc65e79700 1 -- 172.21.15.189:0/1592134493 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc600b13f0 con 0x7fcc6013f270 2020-11-20T12:15:27.887 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.885+0000 7fcc5cff9700 1 -- 172.21.15.189:0/1592134493 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+1785 (secure 0 0 0) 0x7fcc5806ce60 con 0x7fcc6013f270 2020-11-20T12:15:27.888 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:15:27.888 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:15:25.740722+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5389":{"gid":5389,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":34,"addr":"172.21.15.189:6835/636533719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":636533719},{"type":"v1","addr":"172.21.15.189:6835","nonce":636533719}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:15:27.889 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.885+0000 7fcc65e79700 1 -- 172.21.15.189:0/1592134493 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcc380557f0 msgr2=0x7fcc38057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:27.890 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.885+0000 7fcc65e79700 1 --2- 172.21.15.189:0/1592134493 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcc380557f0 0x7fcc38057cb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fcc54006e30 tx=0x7fcc54015000).stop 2020-11-20T12:15:27.890 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.885+0000 7fcc65e79700 1 -- 172.21.15.189:0/1592134493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc6013f270 msgr2=0x7fcc6013ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:27.890 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.885+0000 7fcc65e79700 1 --2- 172.21.15.189:0/1592134493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc6013f270 0x7fcc6013ba60 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7fcc5800cae0 tx=0x7fcc58007d40).stop 2020-11-20T12:15:27.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.885+0000 7fcc65e79700 1 -- 172.21.15.189:0/1592134493 shutdown_connections 2020-11-20T12:15:27.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.885+0000 7fcc65e79700 1 --2- 172.21.15.189:0/1592134493 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcc380557f0 0x7fcc38057cb0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:27.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.885+0000 7fcc65e79700 1 --2- 172.21.15.189:0/1592134493 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcc600ad000 0x7fcc60134650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:27.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.885+0000 7fcc65e79700 1 --2- 172.21.15.189:0/1592134493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc6013f270 0x7fcc6013ba60 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:27.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.885+0000 7fcc65e79700 1 --2- 172.21.15.189:0/1592134493 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcc600ad3e0 0x7fcc6013ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:27.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.885+0000 7fcc65e79700 1 -- 172.21.15.189:0/1592134493 >> 172.21.15.189:0/1592134493 conn(0x7fcc60062080 msgr2=0x7fcc60066c70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:27.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.885+0000 7fcc65e79700 1 -- 172.21.15.189:0/1592134493 shutdown_connections 2020-11-20T12:15:27.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:27.885+0000 7fcc65e79700 1 -- 172.21.15.189:0/1592134493 wait complete. 2020-11-20T12:15:27.893 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 45 2020-11-20T12:15:27.907 INFO:tasks.mds_thrash.fs.[None]:mds.a has gained rank=0, replacing gid=4937 2020-11-20T12:15:27.907 INFO:tasks.mds_thrash.fs.[None]:waiting for 0 secs before reviving mds.b 2020-11-20T12:15:27.908 INFO:tasks.mds_thrash.fs.[None]:reviving mds.b 2020-11-20T12:15:27.908 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:15:27.908 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:15:27.911 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:15:27.911 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:27.990 INFO:tasks.ceph.mds.b.smithi047.stdout:starting mds.b at 2020-11-20T12:15:28.032 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f3522ad7700 1 Processor -- start 2020-11-20T12:15:28.032 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f3522ad7700 1 -- start start 2020-11-20T12:15:28.035 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f3522ad7700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f351c1431c0 0x7f351c1455b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:28.035 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f3522ad7700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f351c066ee0 con 0x7f351c13dc20 2020-11-20T12:15:28.035 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f3522ad7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f351c067180 con 0x7f351c0a0a90 2020-11-20T12:15:28.036 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f3522ad7700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f351c0a96e0 con 0x7f351c1431c0 2020-11-20T12:15:28.036 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f3521ad5700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f351c0a0a90 0x7f351c0a8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52638/0 (socket says 172.21.15.189:52638) 2020-11-20T12:15:28.036 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f3521ad5700 1 -- 172.21.15.189:0/3759850713 learned_addr learned my addr 172.21.15.189:0/3759850713 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:15:28.036 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f35222d6700 1 --2- 172.21.15.189:0/3759850713 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f351c1431c0 0x7f351c1455b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:28.036 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f3520ad3700 1 -- 172.21.15.189:0/3759850713 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 672095849 0 0) 0x7f351c067180 con 0x7f351c0a0a90 2020-11-20T12:15:28.037 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f3520ad3700 1 -- 172.21.15.189:0/3759850713 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3504003240 con 0x7f351c0a0a90 2020-11-20T12:15:28.037 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f3520ad3700 1 -- 172.21.15.189:0/3759850713 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2817955791 0 0) 0x7f3504003240 con 0x7f351c0a0a90 2020-11-20T12:15:28.037 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f3520ad3700 1 -- 172.21.15.189:0/3759850713 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f351c1431c0 msgr2=0x7f351c1455b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:28.037 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f3520ad3700 1 --2- 172.21.15.189:0/3759850713 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f351c1431c0 0x7f351c1455b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:28.037 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f3520ad3700 1 -- 172.21.15.189:0/3759850713 >> v1:172.21.15.47:6789/0 conn(0x7f351c13dc20 legacy=0x7f351c061180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:28.038 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f3520ad3700 1 -- 172.21.15.189:0/3759850713 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351c0a0e70 con 0x7f351c0a0a90 2020-11-20T12:15:28.038 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f3520ad3700 1 -- 172.21.15.189:0/3759850713 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f35180014e0 con 0x7f351c0a0a90 2020-11-20T12:15:28.038 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f3520ad3700 1 -- 172.21.15.189:0/3759850713 >> v1:172.21.15.189:6789/0 conn(0x7f351c0a0a90 legacy=0x7f351c0a8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:28.038 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f3520ad3700 1 --2- 172.21.15.189:0/3759850713 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f351c1431c0 0x7f3504003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:28.039 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f3520ad3700 1 --2- 172.21.15.189:0/3759850713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3504003c20 0x7f35040044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:28.039 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f3520ad3700 1 --2- 172.21.15.189:0/3759850713 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35040049d0 0x7f3504004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:28.039 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f3520ad3700 1 -- 172.21.15.189:0/3759850713 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c0a96e0 con 0x7f35040049d0 2020-11-20T12:15:28.039 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f3520ad3700 1 -- 172.21.15.189:0/3759850713 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3504004140 con 0x7f3504003c20 2020-11-20T12:15:28.039 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f3520ad3700 1 -- 172.21.15.189:0/3759850713 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3504004300 con 0x7f351c1431c0 2020-11-20T12:15:28.040 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f3521ad5700 1 --2- 172.21.15.189:0/3759850713 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f351c1431c0 0x7f3504003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:28.040 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f35212d4700 1 --2- 172.21.15.189:0/3759850713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3504003c20 0x7f35040044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:28.040 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f35222d6700 1 --2- 172.21.15.189:0/3759850713 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35040049d0 0x7f3504004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:28.040 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f35212d4700 1 -- 172.21.15.189:0/3759850713 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f351c1431c0 msgr2=0x7f3504003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:28.040 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f35212d4700 1 --2- 172.21.15.189:0/3759850713 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f351c1431c0 0x7f3504003730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:28.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f35212d4700 1 -- 172.21.15.189:0/3759850713 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35040049d0 msgr2=0x7f3504004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:28.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f35212d4700 1 --2- 172.21.15.189:0/3759850713 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35040049d0 0x7f3504004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:28.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f35212d4700 1 -- 172.21.15.189:0/3759850713 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3504003240 con 0x7f3504003c20 2020-11-20T12:15:28.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.029+0000 7f35222d6700 1 --2- 172.21.15.189:0/3759850713 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35040049d0 0x7f3504004e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:15:28.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.033+0000 7f35212d4700 1 --2- 172.21.15.189:0/3759850713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3504003c20 0x7f35040044e0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f350c001a00 tx=0x7f350c007b70).ready entity=mon.1 client_cookie=df38b76f0c468a31 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:28.044 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.033+0000 7f3520ad3700 1 -- 172.21.15.189:0/3759850713 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c00c830 con 0x7f3504003c20 2020-11-20T12:15:28.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.033+0000 7f3520ad3700 1 -- 172.21.15.189:0/3759850713 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350c00c990 con 0x7f3504003c20 2020-11-20T12:15:28.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.033+0000 7f3520ad3700 1 -- 172.21.15.189:0/3759850713 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c00cd50 con 0x7f3504003c20 2020-11-20T12:15:28.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.033+0000 7f3522ad7700 1 -- 172.21.15.189:0/3759850713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3504003c20 msgr2=0x7f35040044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:28.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.033+0000 7f3522ad7700 1 --2- 172.21.15.189:0/3759850713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3504003c20 0x7f35040044e0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f350c001a00 tx=0x7f350c007b70).stop 2020-11-20T12:15:28.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.033+0000 7f35222d6700 1 -- 172.21.15.189:0/3759850713 reap_dead start 2020-11-20T12:15:28.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.033+0000 7f3522ad7700 1 -- 172.21.15.189:0/3759850713 shutdown_connections 2020-11-20T12:15:28.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.033+0000 7f3522ad7700 1 -- 172.21.15.189:0/3759850713 >> 172.21.15.189:0/3759850713 conn(0x7f351c063210 msgr2=0x7f351c060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:28.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.033+0000 7f3522ad7700 1 -- 172.21.15.189:0/3759850713 shutdown_connections 2020-11-20T12:15:28.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.033+0000 7f3522ad7700 1 -- 172.21.15.189:0/3759850713 wait complete. 2020-11-20T12:15:28.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.033+0000 7f3522ad7700 1 Processor -- start 2020-11-20T12:15:28.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.033+0000 7f3522ad7700 1 -- start start 2020-11-20T12:15:28.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.033+0000 7f3522ad7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3504003c20 0x7f351c0a6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:28.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.033+0000 7f3522ad7700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f351c14d7e0 0x7f351c0abd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:28.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.033+0000 7f3522ad7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f351c0a59d0 0x7f351c0a5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:28.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.033+0000 7f3522ad7700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c14d0a0 con 0x7f3504003c20 2020-11-20T12:15:28.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.033+0000 7f3522ad7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c14d340 con 0x7f351c0a59d0 2020-11-20T12:15:28.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.033+0000 7f3522ad7700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f351c142f30 con 0x7f351c14d7e0 2020-11-20T12:15:28.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.033+0000 7f35222d6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f351c0a59d0 0x7f351c0a5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:28.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.033+0000 7f35222d6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f351c0a59d0 0x7f351c0a5e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51032/0 (socket says 172.21.15.189:51032) 2020-11-20T12:15:28.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.033+0000 7f35222d6700 1 -- 172.21.15.189:0/814259476 learned_addr learned my addr 172.21.15.189:0/814259476 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:15:28.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.033+0000 7f35212d4700 1 --2- 172.21.15.189:0/814259476 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f351c14d7e0 0x7f351c0abd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:28.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.033+0000 7f3521ad5700 1 --2- 172.21.15.189:0/814259476 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3504003c20 0x7f351c0a6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:28.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.037+0000 7f35222d6700 1 -- 172.21.15.189:0/814259476 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f351c14d7e0 msgr2=0x7f351c0abd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:28.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.037+0000 7f35222d6700 1 --2- 172.21.15.189:0/814259476 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f351c14d7e0 0x7f351c0abd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:28.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.037+0000 7f35222d6700 1 -- 172.21.15.189:0/814259476 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3504003c20 msgr2=0x7f351c0a6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:28.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.037+0000 7f35222d6700 1 --2- 172.21.15.189:0/814259476 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3504003c20 0x7f351c0a6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:28.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.037+0000 7f35222d6700 1 -- 172.21.15.189:0/814259476 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351c0a6340 con 0x7f351c0a59d0 2020-11-20T12:15:28.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.037+0000 7f35222d6700 1 --2- 172.21.15.189:0/814259476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f351c0a59d0 0x7f351c0a5e50 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f351c0a9290 tx=0x7f3514008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:28.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.037+0000 7f3509ffb700 1 -- 172.21.15.189:0/814259476 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351400e960 con 0x7f351c0a59d0 2020-11-20T12:15:28.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.037+0000 7f3509ffb700 1 -- 172.21.15.189:0/814259476 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f351400eac0 con 0x7f351c0a59d0 2020-11-20T12:15:28.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.037+0000 7f3509ffb700 1 -- 172.21.15.189:0/814259476 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351400ee80 con 0x7f351c0a59d0 2020-11-20T12:15:28.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.037+0000 7f3522ad7700 1 -- 172.21.15.189:0/814259476 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f351c0ac270 con 0x7f351c0a59d0 2020-11-20T12:15:28.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.037+0000 7f3522ad7700 1 -- 172.21.15.189:0/814259476 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f351c14dc60 con 0x7f351c0a59d0 2020-11-20T12:15:28.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.037+0000 7f3522ad7700 1 -- 172.21.15.189:0/814259476 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f351c14de20 con 0x7f351c0a59d0 2020-11-20T12:15:28.052 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.037+0000 7f3509ffb700 1 -- 172.21.15.189:0/814259476 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f3514014640 con 0x7f351c0a59d0 2020-11-20T12:15:28.052 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.037+0000 7f3509ffb700 1 --2- 172.21.15.189:0/814259476 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f34f40557c0 0x7f34f4057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:28.052 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.037+0000 7f3509ffb700 1 -- 172.21.15.189:0/814259476 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6761+0+0 (secure 0 0 0) 0x7f351406efc0 con 0x7f351c0a59d0 2020-11-20T12:15:28.052 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.041+0000 7f3521ad5700 1 --2- 172.21.15.189:0/814259476 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f34f40557c0 0x7f34f4057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:28.052 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.041+0000 7f3521ad5700 1 --2- 172.21.15.189:0/814259476 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f34f40557c0 0x7f34f4057c80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f3518007130 tx=0x7f3518009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:28.053 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.045+0000 7f3509ffb700 1 -- 172.21.15.189:0/814259476 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f351c14de20 con 0x7f351c0a59d0 2020-11-20T12:15:28.270 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.265+0000 7f3522ad7700 1 -- 172.21.15.189:0/814259476 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f351c14de20 con 0x7f351c0a59d0 2020-11-20T12:15:28.271 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.265+0000 7f3509ffb700 1 -- 172.21.15.189:0/814259476 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+1785 (secure 0 0 0) 0x7f351c14de20 con 0x7f351c0a59d0 2020-11-20T12:15:28.271 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:15:28.271 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:15:25.740722+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5389":{"gid":5389,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":34,"addr":"172.21.15.189:6835/636533719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":636533719},{"type":"v1","addr":"172.21.15.189:6835","nonce":636533719}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:15:28.273 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.269+0000 7f3522ad7700 1 -- 172.21.15.189:0/814259476 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f34f40557c0 msgr2=0x7f34f4057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:28.273 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.269+0000 7f3522ad7700 1 --2- 172.21.15.189:0/814259476 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f34f40557c0 0x7f34f4057c80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f3518007130 tx=0x7f3518009040).stop 2020-11-20T12:15:28.273 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.269+0000 7f3522ad7700 1 -- 172.21.15.189:0/814259476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f351c0a59d0 msgr2=0x7f351c0a5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:28.274 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.269+0000 7f3522ad7700 1 --2- 172.21.15.189:0/814259476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f351c0a59d0 0x7f351c0a5e50 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f351c0a9290 tx=0x7f3514008040).stop 2020-11-20T12:15:28.274 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.269+0000 7f3522ad7700 1 -- 172.21.15.189:0/814259476 shutdown_connections 2020-11-20T12:15:28.274 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.269+0000 7f3522ad7700 1 --2- 172.21.15.189:0/814259476 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f34f40557c0 0x7f34f4057c80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:28.274 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.269+0000 7f3522ad7700 1 --2- 172.21.15.189:0/814259476 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f3504003c20 0x7f351c0a6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:28.274 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.269+0000 7f3522ad7700 1 --2- 172.21.15.189:0/814259476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f351c0a59d0 0x7f351c0a5e50 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:28.275 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.269+0000 7f3522ad7700 1 --2- 172.21.15.189:0/814259476 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f351c14d7e0 0x7f351c0abd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:28.275 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.269+0000 7f3522ad7700 1 -- 172.21.15.189:0/814259476 >> 172.21.15.189:0/814259476 conn(0x7f351c063210 msgr2=0x7f351c13a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:28.275 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.269+0000 7f3522ad7700 1 -- 172.21.15.189:0/814259476 shutdown_connections 2020-11-20T12:15:28.275 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:28.269+0000 7f3522ad7700 1 -- 172.21.15.189:0/814259476 wait complete. 2020-11-20T12:15:28.276 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 45 2020-11-20T12:15:28.286 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2020-11-20T12:15:30.286 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:15:30.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.429+0000 7f61d6f4b700 1 Processor -- start 2020-11-20T12:15:30.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.429+0000 7f61d6f4b700 1 -- start start 2020-11-20T12:15:30.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61d6f4b700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f61d013b0e0 0x7f61d00682e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:30.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61d6f4b700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61d00687d0 con 0x7f61d0063e30 2020-11-20T12:15:30.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61d6f4b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61d00a8550 con 0x7f61d0063a50 2020-11-20T12:15:30.436 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61d6f4b700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61d00a87f0 con 0x7f61d013b0e0 2020-11-20T12:15:30.436 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61d4ce7700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f61d0063a50 0x7f61d013a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52658/0 (socket says 172.21.15.189:52658) 2020-11-20T12:15:30.436 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61d4ce7700 1 -- 172.21.15.189:0/2150576604 learned_addr learned my addr 172.21.15.189:0/2150576604 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:15:30.437 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61d54e8700 1 --2- 172.21.15.189:0/2150576604 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f61d013b0e0 0x7f61d00682e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:30.437 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61cf7fe700 1 -- 172.21.15.189:0/2150576604 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568278405 0 0) 0x7f61d00a8550 con 0x7f61d0063a50 2020-11-20T12:15:30.437 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61cf7fe700 1 -- 172.21.15.189:0/2150576604 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61b4003240 con 0x7f61d0063a50 2020-11-20T12:15:30.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61cf7fe700 1 -- 172.21.15.189:0/2150576604 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2460056628 0 0) 0x7f61b4003240 con 0x7f61d0063a50 2020-11-20T12:15:30.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61cf7fe700 1 -- 172.21.15.189:0/2150576604 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f61d013b0e0 msgr2=0x7f61d00682e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:30.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61cf7fe700 1 --2- 172.21.15.189:0/2150576604 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f61d013b0e0 0x7f61d00682e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:30.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61cf7fe700 1 -- 172.21.15.189:0/2150576604 >> v1:172.21.15.47:6789/0 conn(0x7f61d0063e30 legacy=0x7f61d00a7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:30.442 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61cf7fe700 1 -- 172.21.15.189:0/2150576604 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d00a8280 con 0x7f61d0063a50 2020-11-20T12:15:30.442 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61cf7fe700 1 -- 172.21.15.189:0/2150576604 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f61c4001520 con 0x7f61d0063a50 2020-11-20T12:15:30.442 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61cf7fe700 1 -- 172.21.15.189:0/2150576604 >> v1:172.21.15.189:6789/0 conn(0x7f61d0063a50 legacy=0x7f61d013a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:30.442 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61cf7fe700 1 --2- 172.21.15.189:0/2150576604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b4003730 0x7f61b4003b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:30.442 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61cf7fe700 1 --2- 172.21.15.189:0/2150576604 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61b4004070 0x7f61b4004540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:30.443 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61cf7fe700 1 --2- 172.21.15.189:0/2150576604 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f61d013b0e0 0x7f61b4004ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:30.443 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61cf7fe700 1 -- 172.21.15.189:0/2150576604 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d00a87f0 con 0x7f61b4004070 2020-11-20T12:15:30.443 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61cf7fe700 1 -- 172.21.15.189:0/2150576604 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b40050e0 con 0x7f61b4003730 2020-11-20T12:15:30.443 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61cf7fe700 1 -- 172.21.15.189:0/2150576604 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b40052a0 con 0x7f61d013b0e0 2020-11-20T12:15:30.443 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61d4ce7700 1 --2- 172.21.15.189:0/2150576604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b4003730 0x7f61b4003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:30.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61d54e8700 1 --2- 172.21.15.189:0/2150576604 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f61d013b0e0 0x7f61b4004ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:30.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61cffff700 1 --2- 172.21.15.189:0/2150576604 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61b4004070 0x7f61b4004540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:30.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61d4ce7700 1 -- 172.21.15.189:0/2150576604 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f61d013b0e0 msgr2=0x7f61b4004ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:30.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61d4ce7700 1 --2- 172.21.15.189:0/2150576604 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f61d013b0e0 0x7f61b4004ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:30.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61d4ce7700 1 -- 172.21.15.189:0/2150576604 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61b4004070 msgr2=0x7f61b4004540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:15:30.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61d4ce7700 1 --2- 172.21.15.189:0/2150576604 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61b4004070 0x7f61b4004540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:30.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61d4ce7700 1 -- 172.21.15.189:0/2150576604 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b4003240 con 0x7f61b4003730 2020-11-20T12:15:30.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61cffff700 1 --2- 172.21.15.189:0/2150576604 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61b4004070 0x7f61b4004540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:15:30.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61d4ce7700 1 --2- 172.21.15.189:0/2150576604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b4003730 0x7f61b4003b10 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f61c400c230 tx=0x7f61d01380b0).ready entity=mon.1 client_cookie=c67a4901f2b9000 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:30.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61cf7fe700 1 -- 172.21.15.189:0/2150576604 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c40070d0 con 0x7f61b4003730 2020-11-20T12:15:30.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61cf7fe700 1 -- 172.21.15.189:0/2150576604 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61c4007230 con 0x7f61b4003730 2020-11-20T12:15:30.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61cf7fe700 1 -- 172.21.15.189:0/2150576604 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c40055f0 con 0x7f61b4003730 2020-11-20T12:15:30.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61d6f4b700 1 -- 172.21.15.189:0/2150576604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b4003730 msgr2=0x7f61b4003b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:30.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61d6f4b700 1 --2- 172.21.15.189:0/2150576604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b4003730 0x7f61b4003b10 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f61c400c230 tx=0x7f61d01380b0).stop 2020-11-20T12:15:30.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61d54e8700 1 -- 172.21.15.189:0/2150576604 reap_dead start 2020-11-20T12:15:30.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61d6f4b700 1 -- 172.21.15.189:0/2150576604 shutdown_connections 2020-11-20T12:15:30.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61d6f4b700 1 -- 172.21.15.189:0/2150576604 >> 172.21.15.189:0/2150576604 conn(0x7f61d00ad8e0 msgr2=0x7f61d00adcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:30.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61d6f4b700 1 -- 172.21.15.189:0/2150576604 shutdown_connections 2020-11-20T12:15:30.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.433+0000 7f61d6f4b700 1 -- 172.21.15.189:0/2150576604 wait complete. 2020-11-20T12:15:30.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61d6f4b700 1 Processor -- start 2020-11-20T12:15:30.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61d6f4b700 1 -- start start 2020-11-20T12:15:30.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61d6f4b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61d00ace10 0x7f61d00a4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:30.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61d6f4b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61d009c9f0 0x7f61d009dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:30.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61d6f4b700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f61d0136c70 0x7f61d0133b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:30.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61d6f4b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d009e830 con 0x7f61d00ace10 2020-11-20T12:15:30.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61d6f4b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d01370f0 con 0x7f61d009c9f0 2020-11-20T12:15:30.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61d6f4b700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61d009e590 con 0x7f61d0136c70 2020-11-20T12:15:30.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61cffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61d009c9f0 0x7f61d009dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:30.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61cffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61d009c9f0 0x7f61d009dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51050/0 (socket says 172.21.15.189:51050) 2020-11-20T12:15:30.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61cffff700 1 -- 172.21.15.189:0/4208469254 learned_addr learned my addr 172.21.15.189:0/4208469254 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:15:30.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61d4ce7700 1 --2- 172.21.15.189:0/4208469254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61d00ace10 0x7f61d00a4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:30.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61d54e8700 1 --2- 172.21.15.189:0/4208469254 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f61d0136c70 0x7f61d0133b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:30.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61cffff700 1 -- 172.21.15.189:0/4208469254 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f61d0136c70 msgr2=0x7f61d0133b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:30.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61cffff700 1 --2- 172.21.15.189:0/4208469254 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f61d0136c70 0x7f61d0133b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:30.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61cffff700 1 -- 172.21.15.189:0/4208469254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61d00ace10 msgr2=0x7f61d00a4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:30.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61cffff700 1 --2- 172.21.15.189:0/4208469254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61d00ace10 0x7f61d00a4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:30.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61cffff700 1 -- 172.21.15.189:0/4208469254 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d00a3fb0 con 0x7f61d009c9f0 2020-11-20T12:15:30.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61d4ce7700 1 --2- 172.21.15.189:0/4208469254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61d00ace10 0x7f61d00a4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:15:30.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61cffff700 1 --2- 172.21.15.189:0/4208469254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61d009c9f0 0x7f61d009dfa0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f61bc00db80 tx=0x7f61bc009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:30.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61ccff9700 1 -- 172.21.15.189:0/4208469254 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61bc00b860 con 0x7f61d009c9f0 2020-11-20T12:15:30.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61ccff9700 1 -- 172.21.15.189:0/4208469254 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61bc00b9c0 con 0x7f61d009c9f0 2020-11-20T12:15:30.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61d6f4b700 1 -- 172.21.15.189:0/4208469254 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61d00ad1f0 con 0x7f61d009c9f0 2020-11-20T12:15:30.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61ccff9700 1 -- 172.21.15.189:0/4208469254 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61bc00bd40 con 0x7f61d009c9f0 2020-11-20T12:15:30.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61d6f4b700 1 -- 172.21.15.189:0/4208469254 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61d0134040 con 0x7f61d009c9f0 2020-11-20T12:15:30.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61d6f4b700 1 -- 172.21.15.189:0/4208469254 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61d00a4140 con 0x7f61d009c9f0 2020-11-20T12:15:30.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.437+0000 7f61ccff9700 1 -- 172.21.15.189:0/4208469254 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f61bc012620 con 0x7f61d009c9f0 2020-11-20T12:15:30.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.441+0000 7f61ccff9700 1 --2- 172.21.15.189:0/4208469254 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f61a80557c0 0x7f61a8057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:15:30.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.441+0000 7f61ccff9700 1 -- 172.21.15.189:0/4208469254 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6761+0+0 (secure 0 0 0) 0x7f61bc06ec00 con 0x7f61d009c9f0 2020-11-20T12:15:30.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.441+0000 7f61d4ce7700 1 --2- 172.21.15.189:0/4208469254 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f61a80557c0 0x7f61a8057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:15:30.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.441+0000 7f61d4ce7700 1 --2- 172.21.15.189:0/4208469254 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f61a80557c0 0x7f61a8057c80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f61d0064fd0 tx=0x7f61c4015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:15:30.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.445+0000 7f61ccff9700 1 -- 172.21.15.189:0/4208469254 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f61bc074020 con 0x7f61d009c9f0 2020-11-20T12:15:30.709 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.705+0000 7f61d6f4b700 1 -- 172.21.15.189:0/4208469254 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61d00446d0 con 0x7f61d009c9f0 2020-11-20T12:15:30.709 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.705+0000 7f61ccff9700 1 -- 172.21.15.189:0/4208469254 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f61bc0447f0 con 0x7f61d009c9f0 2020-11-20T12:15:30.710 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:15:30.710 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5075,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/563711894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":563711894},{"type":"v1","addr":"172.21.15.47:6833","nonce":563711894}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:15:25.740722+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5389":{"gid":5389,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":34,"addr":"172.21.15.189:6835/636533719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":636533719},{"type":"v1","addr":"172.21.15.189:6835","nonce":636533719}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:15:30.712 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.709+0000 7f61d6f4b700 1 -- 172.21.15.189:0/4208469254 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f61a80557c0 msgr2=0x7f61a8057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:30.712 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.709+0000 7f61d6f4b700 1 --2- 172.21.15.189:0/4208469254 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f61a80557c0 0x7f61a8057c80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f61d0064fd0 tx=0x7f61c4015000).stop 2020-11-20T12:15:30.712 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.709+0000 7f61d6f4b700 1 -- 172.21.15.189:0/4208469254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61d009c9f0 msgr2=0x7f61d009dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:15:30.712 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.709+0000 7f61d6f4b700 1 --2- 172.21.15.189:0/4208469254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61d009c9f0 0x7f61d009dfa0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f61bc00db80 tx=0x7f61bc009040).stop 2020-11-20T12:15:30.713 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.709+0000 7f61d6f4b700 1 -- 172.21.15.189:0/4208469254 shutdown_connections 2020-11-20T12:15:30.713 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.709+0000 7f61d6f4b700 1 --2- 172.21.15.189:0/4208469254 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f61a80557c0 0x7f61a8057c80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:30.713 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.709+0000 7f61d6f4b700 1 --2- 172.21.15.189:0/4208469254 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f61d00ace10 0x7f61d00a4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:30.713 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.709+0000 7f61d6f4b700 1 --2- 172.21.15.189:0/4208469254 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f61d0136c70 0x7f61d0133b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:30.713 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.709+0000 7f61d6f4b700 1 --2- 172.21.15.189:0/4208469254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61d009c9f0 0x7f61d009dfa0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:15:30.714 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.709+0000 7f61d6f4b700 1 -- 172.21.15.189:0/4208469254 >> 172.21.15.189:0/4208469254 conn(0x7f61d00ad8e0 msgr2=0x7f61d013a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:15:30.714 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.709+0000 7f61d6f4b700 1 -- 172.21.15.189:0/4208469254 shutdown_connections 2020-11-20T12:15:30.714 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:15:30.709+0000 7f61d6f4b700 1 -- 172.21.15.189:0/4208469254 wait complete. 2020-11-20T12:15:30.715 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 46 2020-11-20T12:15:30.726 INFO:tasks.mds_thrash.fs.[None]:mds.b reported in up:standby state 2020-11-20T12:15:30.727 INFO:tasks.mds_thrash.fs.[None]:waiting for 48 secs before thrashing 2020-11-20T12:15:54.996 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:15:55.000 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:16:18.727 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:16:18.999 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.992+0000 7f1f8c535700 1 Processor -- start 2020-11-20T12:16:18.999 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.992+0000 7f1f8c535700 1 -- start start 2020-11-20T12:16:19.000 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f8c535700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f1f8413ab80 0x7f1f8413e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:19.001 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f8c535700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1f8413e6c0 0x7f1f8413cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:19.001 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f8c535700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8413b300 con 0x7f1f84066de0 2020-11-20T12:16:19.001 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f8c535700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f8413dd30 con 0x7f1f8413ab80 2020-11-20T12:16:19.001 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f8c535700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f8413aff0 con 0x7f1f8413e6c0 2020-11-20T12:16:19.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f8a2d1700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1f84066de0 0x7f1f840671c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52678/0 (socket says 172.21.15.189:52678) 2020-11-20T12:16:19.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f8a2d1700 1 -- 172.21.15.189:0/1007946655 learned_addr learned my addr 172.21.15.189:0/1007946655 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:16:19.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f8aad2700 1 --2- 172.21.15.189:0/1007946655 >> v2:172.21.15.189:3300/0 conn(0x7f1f8413e6c0 0x7f1f8413cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:19.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f89ad0700 1 --2- 172.21.15.189:0/1007946655 >> v2:172.21.15.47:3300/0 conn(0x7f1f8413ab80 0x7f1f8413e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:19.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f892cf700 1 -- 172.21.15.189:0/1007946655 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454230944 0 0) 0x7f1f8413b300 con 0x7f1f84066de0 2020-11-20T12:16:19.003 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f892cf700 1 -- 172.21.15.189:0/1007946655 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f6c003240 con 0x7f1f84066de0 2020-11-20T12:16:19.004 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f892cf700 1 -- 172.21.15.189:0/1007946655 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1058238232 0 0) 0x7f1f6c003240 con 0x7f1f84066de0 2020-11-20T12:16:19.005 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f8aad2700 1 -- 172.21.15.189:0/1007946655 >> v2:172.21.15.47:3300/0 conn(0x7f1f8413ab80 msgr2=0x7f1f8413e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:19.005 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f8aad2700 1 --2- 172.21.15.189:0/1007946655 >> v2:172.21.15.47:3300/0 conn(0x7f1f8413ab80 0x7f1f8413e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:19.005 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f8aad2700 1 -- 172.21.15.189:0/1007946655 >> v1:172.21.15.189:6789/0 conn(0x7f1f84066de0 legacy=0x7f1f840671c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:19.005 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f8aad2700 1 -- 172.21.15.189:0/1007946655 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8413d390 con 0x7f1f8413e6c0 2020-11-20T12:16:19.005 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f89ad0700 1 --2- 172.21.15.189:0/1007946655 >> v2:172.21.15.47:3300/0 conn(0x7f1f8413ab80 0x7f1f8413e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:16:19.006 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f8aad2700 1 --2- 172.21.15.189:0/1007946655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f8413e6c0 0x7f1f8413cea0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f1f7c009c90 tx=0x7f1f7c0091d0).ready entity=mon.1 client_cookie=7d5080feeb498f7f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:19.006 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f892cf700 1 -- 172.21.15.189:0/1007946655 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f800014e0 con 0x7f1f8413e6c0 2020-11-20T12:16:19.006 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f892cf700 1 -- 172.21.15.189:0/1007946655 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f7c00b6b0 con 0x7f1f8413e6c0 2020-11-20T12:16:19.006 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f892cf700 1 -- 172.21.15.189:0/1007946655 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c00ba30 con 0x7f1f8413e6c0 2020-11-20T12:16:19.007 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f8c535700 1 -- 172.21.15.189:0/1007946655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f8413e6c0 msgr2=0x7f1f8413cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:19.007 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f8c535700 1 --2- 172.21.15.189:0/1007946655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f8413e6c0 0x7f1f8413cea0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f1f7c009c90 tx=0x7f1f7c0091d0).stop 2020-11-20T12:16:19.007 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f8c535700 1 -- 172.21.15.189:0/1007946655 shutdown_connections 2020-11-20T12:16:19.007 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f8c535700 1 --2- 172.21.15.189:0/1007946655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f8413e6c0 0x7f1f8413cea0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:19.007 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f8c535700 1 --2- 172.21.15.189:0/1007946655 >> v2:172.21.15.47:3300/0 conn(0x7f1f8413ab80 0x7f1f8413e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:19.008 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f8c535700 1 -- 172.21.15.189:0/1007946655 >> 172.21.15.189:0/1007946655 conn(0x7f1f84061f60 msgr2=0x7f1f84062340 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:19.008 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f8c535700 1 -- 172.21.15.189:0/1007946655 shutdown_connections 2020-11-20T12:16:19.008 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:18.996+0000 7f1f8c535700 1 -- 172.21.15.189:0/1007946655 wait complete. 2020-11-20T12:16:19.008 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.000+0000 7f1f8c535700 1 Processor -- start 2020-11-20T12:16:19.008 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.000+0000 7f1f8c535700 1 -- start start 2020-11-20T12:16:19.011 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.000+0000 7f1f8c535700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f84066de0 0x7f1f84140850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:19.011 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.000+0000 7f1f8c535700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1f8413ab80 0x7f1f840a90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:19.011 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.000+0000 7f1f8c535700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f8413e6c0 0x7f1f840a95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:19.011 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.000+0000 7f1f8c535700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f840aa1f0 con 0x7f1f8413e6c0 2020-11-20T12:16:19.012 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.000+0000 7f1f8c535700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f840aa490 con 0x7f1f84066de0 2020-11-20T12:16:19.012 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.000+0000 7f1f8c535700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f840aa730 con 0x7f1f8413ab80 2020-11-20T12:16:19.012 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.000+0000 7f1f8a2d1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f84066de0 0x7f1f84140850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:19.012 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.000+0000 7f1f8a2d1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f84066de0 0x7f1f84140850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51062/0 (socket says 172.21.15.189:51062) 2020-11-20T12:16:19.013 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.000+0000 7f1f8a2d1700 1 -- 172.21.15.189:0/442882210 learned_addr learned my addr 172.21.15.189:0/442882210 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:16:19.013 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.000+0000 7f1f8aad2700 1 --2- 172.21.15.189:0/442882210 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f8413e6c0 0x7f1f840a95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:19.013 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.000+0000 7f1f89ad0700 1 --2- 172.21.15.189:0/442882210 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1f8413ab80 0x7f1f840a90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:19.013 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.000+0000 7f1f8a2d1700 1 -- 172.21.15.189:0/442882210 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1f8413ab80 msgr2=0x7f1f840a90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:19.014 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.000+0000 7f1f8a2d1700 1 --2- 172.21.15.189:0/442882210 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1f8413ab80 0x7f1f840a90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:19.014 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.000+0000 7f1f8a2d1700 1 -- 172.21.15.189:0/442882210 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f8413e6c0 msgr2=0x7f1f840a95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:19.014 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.000+0000 7f1f8a2d1700 1 --2- 172.21.15.189:0/442882210 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f8413e6c0 0x7f1f840a95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:19.014 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.000+0000 7f1f8a2d1700 1 -- 172.21.15.189:0/442882210 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f840aa9d0 con 0x7f1f84066de0 2020-11-20T12:16:19.014 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.000+0000 7f1f8a2d1700 1 --2- 172.21.15.189:0/442882210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f84066de0 0x7f1f84140850 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f1f8000cab0 tx=0x7f1f800011f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:19.015 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.000+0000 7f1f727fc700 1 -- 172.21.15.189:0/442882210 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f80007e20 con 0x7f1f84066de0 2020-11-20T12:16:19.015 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.000+0000 7f1f727fc700 1 -- 172.21.15.189:0/442882210 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f80007170 con 0x7f1f84066de0 2020-11-20T12:16:19.015 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.000+0000 7f1f8c535700 1 -- 172.21.15.189:0/442882210 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f840a18c0 con 0x7f1f84066de0 2020-11-20T12:16:19.015 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.000+0000 7f1f727fc700 1 -- 172.21.15.189:0/442882210 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8000a870 con 0x7f1f84066de0 2020-11-20T12:16:19.015 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.000+0000 7f1f8c535700 1 -- 172.21.15.189:0/442882210 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f840a77d0 con 0x7f1f84066de0 2020-11-20T12:16:19.016 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.004+0000 7f1f8c535700 1 -- 172.21.15.189:0/442882210 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f840aad90 con 0x7f1f84066de0 2020-11-20T12:16:19.016 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.004+0000 7f1f727fc700 1 -- 172.21.15.189:0/442882210 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f1f8001b060 con 0x7f1f84066de0 2020-11-20T12:16:19.016 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.004+0000 7f1f727fc700 1 --2- 172.21.15.189:0/442882210 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1f5c0557f0 0x7f1f5c057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:19.016 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.004+0000 7f1f727fc700 1 -- 172.21.15.189:0/442882210 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6761+0+0 (secure 0 0 0) 0x7f1f8006e4e0 con 0x7f1f84066de0 2020-11-20T12:16:19.017 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.004+0000 7f1f89ad0700 1 --2- 172.21.15.189:0/442882210 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1f5c0557f0 0x7f1f5c057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:19.017 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.004+0000 7f1f89ad0700 1 --2- 172.21.15.189:0/442882210 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1f5c0557f0 0x7f1f5c057cb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f1f840a9060 tx=0x7f1f74008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:19.017 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.012+0000 7f1f727fc700 1 -- 172.21.15.189:0/442882210 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1f800440d0 con 0x7f1f84066de0 2020-11-20T12:16:19.241 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.236+0000 7f1f8c535700 1 -- 172.21.15.189:0/442882210 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f84044f60 con 0x7f1f84066de0 2020-11-20T12:16:19.242 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.236+0000 7f1f727fc700 1 -- 172.21.15.189:0/442882210 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f1f80019030 con 0x7f1f84066de0 2020-11-20T12:16:19.242 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:16:19.242 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5075,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/563711894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":563711894},{"type":"v1","addr":"172.21.15.47:6833","nonce":563711894}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:15:25.740722+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5389":{"gid":5389,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":34,"addr":"172.21.15.189:6835/636533719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":636533719},{"type":"v1","addr":"172.21.15.189:6835","nonce":636533719}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:16:19.244 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.240+0000 7f1f8c535700 1 -- 172.21.15.189:0/442882210 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1f5c0557f0 msgr2=0x7f1f5c057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:19.244 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.240+0000 7f1f8c535700 1 --2- 172.21.15.189:0/442882210 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1f5c0557f0 0x7f1f5c057cb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f1f840a9060 tx=0x7f1f74008040).stop 2020-11-20T12:16:19.244 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.240+0000 7f1f8c535700 1 -- 172.21.15.189:0/442882210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f84066de0 msgr2=0x7f1f84140850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:19.244 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.240+0000 7f1f8c535700 1 --2- 172.21.15.189:0/442882210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f84066de0 0x7f1f84140850 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f1f8000cab0 tx=0x7f1f800011f0).stop 2020-11-20T12:16:19.245 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.240+0000 7f1f8c535700 1 -- 172.21.15.189:0/442882210 shutdown_connections 2020-11-20T12:16:19.245 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.240+0000 7f1f8c535700 1 --2- 172.21.15.189:0/442882210 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1f5c0557f0 0x7f1f5c057cb0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:19.245 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.240+0000 7f1f8c535700 1 --2- 172.21.15.189:0/442882210 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f8413e6c0 0x7f1f840a95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:19.245 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.240+0000 7f1f8c535700 1 --2- 172.21.15.189:0/442882210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f84066de0 0x7f1f84140850 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:19.245 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.240+0000 7f1f8c535700 1 --2- 172.21.15.189:0/442882210 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1f8413ab80 0x7f1f840a90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:19.246 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.240+0000 7f1f8c535700 1 -- 172.21.15.189:0/442882210 >> 172.21.15.189:0/442882210 conn(0x7f1f84061f60 msgr2=0x7f1f840addf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:19.246 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.240+0000 7f1f8c535700 1 -- 172.21.15.189:0/442882210 shutdown_connections 2020-11-20T12:16:19.246 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.240+0000 7f1f8c535700 1 -- 172.21.15.189:0/442882210 wait complete. 2020-11-20T12:16:19.246 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 46 2020-11-20T12:16:19.256 INFO:tasks.mds_thrash.fs.[None]:kill mds.a (rank=0) 2020-11-20T12:16:19.256 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T12:16:19.257 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:16:19.269 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T12:16:19.270 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:16:19.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.444+0000 7f5116219700 1 Processor -- start 2020-11-20T12:16:19.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.444+0000 7f5116219700 1 -- start start 2020-11-20T12:16:19.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.444+0000 7f5116219700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f51100aa680 0x7f51100a70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:19.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.444+0000 7f5116219700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f51100a75c0 0x7f51100a9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:19.452 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.444+0000 7f5116219700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51100a90c0 con 0x7f51100a9b60 2020-11-20T12:16:19.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.444+0000 7f5116219700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51100ab630 con 0x7f51100a75c0 2020-11-20T12:16:19.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.444+0000 7f5116219700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51100a9ff0 con 0x7f51100aa680 2020-11-20T12:16:19.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.444+0000 7f510ffff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f51100a9b60 0x7f5110060170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52698/0 (socket says 172.21.15.189:52698) 2020-11-20T12:16:19.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.444+0000 7f510ffff700 1 -- 172.21.15.189:0/1933438976 learned_addr learned my addr 172.21.15.189:0/1933438976 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:16:19.453 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.444+0000 7f510effd700 1 --2- 172.21.15.189:0/1933438976 >> v2:172.21.15.47:3300/0 conn(0x7f51100a75c0 0x7f51100a9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:19.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.444+0000 7f510f7fe700 1 --2- 172.21.15.189:0/1933438976 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f51100aa680 0x7f51100a70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:19.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.444+0000 7f510e7fc700 1 -- 172.21.15.189:0/1933438976 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869373602 0 0) 0x7f51100a90c0 con 0x7f51100a9b60 2020-11-20T12:16:19.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.444+0000 7f510e7fc700 1 -- 172.21.15.189:0/1933438976 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50fc003240 con 0x7f51100a9b60 2020-11-20T12:16:19.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.444+0000 7f510e7fc700 1 -- 172.21.15.189:0/1933438976 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4167898696 0 0) 0x7f50fc003240 con 0x7f51100a9b60 2020-11-20T12:16:19.454 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.444+0000 7f510e7fc700 1 -- 172.21.15.189:0/1933438976 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f51100aa680 msgr2=0x7f51100a70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:19.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.444+0000 7f510e7fc700 1 --2- 172.21.15.189:0/1933438976 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f51100aa680 0x7f51100a70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:19.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.444+0000 7f510e7fc700 1 -- 172.21.15.189:0/1933438976 >> v2:172.21.15.47:3300/0 conn(0x7f51100a75c0 msgr2=0x7f51100a9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:19.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.444+0000 7f510e7fc700 1 --2- 172.21.15.189:0/1933438976 >> v2:172.21.15.47:3300/0 conn(0x7f51100a75c0 0x7f51100a9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:19.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.444+0000 7f510e7fc700 1 -- 172.21.15.189:0/1933438976 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51100ab890 con 0x7f51100a9b60 2020-11-20T12:16:19.455 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.444+0000 7f510e7fc700 1 -- 172.21.15.189:0/1933438976 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f5108001860 con 0x7f51100a9b60 2020-11-20T12:16:19.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f510e7fc700 1 -- 172.21.15.189:0/1933438976 >> v1:172.21.15.189:6789/0 conn(0x7f51100a9b60 legacy=0x7f5110060170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:19.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f510e7fc700 1 --2- 172.21.15.189:0/1933438976 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50fc0037b0 0x7f50fc003bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:19.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f510e7fc700 1 --2- 172.21.15.189:0/1933438976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50fc0044e0 0x7f50fc004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:19.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f510e7fc700 1 --2- 172.21.15.189:0/1933438976 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f51100aa680 0x7f50fc004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:19.456 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f510e7fc700 1 -- 172.21.15.189:0/1933438976 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51100ab630 con 0x7f50fc0037b0 2020-11-20T12:16:19.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f510e7fc700 1 -- 172.21.15.189:0/1933438976 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51100a9ff0 con 0x7f50fc0044e0 2020-11-20T12:16:19.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f510e7fc700 1 -- 172.21.15.189:0/1933438976 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc005340 con 0x7f51100aa680 2020-11-20T12:16:19.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f510effd700 1 --2- 172.21.15.189:0/1933438976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50fc0044e0 0x7f50fc004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:19.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f510f7fe700 1 --2- 172.21.15.189:0/1933438976 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50fc0037b0 0x7f50fc003bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:19.457 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f510ffff700 1 --2- 172.21.15.189:0/1933438976 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f51100aa680 0x7f50fc004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:19.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f510effd700 1 -- 172.21.15.189:0/1933438976 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f51100aa680 msgr2=0x7f50fc004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:19.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f510effd700 1 --2- 172.21.15.189:0/1933438976 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f51100aa680 0x7f50fc004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:19.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f510effd700 1 -- 172.21.15.189:0/1933438976 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50fc0037b0 msgr2=0x7f50fc003bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:19.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f510effd700 1 --2- 172.21.15.189:0/1933438976 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50fc0037b0 0x7f50fc003bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:19.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f510effd700 1 -- 172.21.15.189:0/1933438976 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50fc003240 con 0x7f50fc0044e0 2020-11-20T12:16:19.459 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f510f7fe700 1 --2- 172.21.15.189:0/1933438976 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50fc0037b0 0x7f50fc003bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:16:19.459 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f510effd700 1 --2- 172.21.15.189:0/1933438976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50fc0044e0 0x7f50fc004960 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f5110062f70 tx=0x7f5104006df0).ready entity=mon.1 client_cookie=aff4c5051d95441f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:19.459 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f510e7fc700 1 -- 172.21.15.189:0/1933438976 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510400b330 con 0x7f50fc0044e0 2020-11-20T12:16:19.459 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f510e7fc700 1 -- 172.21.15.189:0/1933438976 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f510400eae0 con 0x7f50fc0044e0 2020-11-20T12:16:19.459 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f510e7fc700 1 -- 172.21.15.189:0/1933438976 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510400ee60 con 0x7f50fc0044e0 2020-11-20T12:16:19.462 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f5116219700 1 -- 172.21.15.189:0/1933438976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50fc0044e0 msgr2=0x7f50fc004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:19.463 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f5116219700 1 --2- 172.21.15.189:0/1933438976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50fc0044e0 0x7f50fc004960 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f5110062f70 tx=0x7f5104006df0).stop 2020-11-20T12:16:19.463 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f510ffff700 1 -- 172.21.15.189:0/1933438976 reap_dead start 2020-11-20T12:16:19.463 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f5116219700 1 -- 172.21.15.189:0/1933438976 shutdown_connections 2020-11-20T12:16:19.463 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f5116219700 1 -- 172.21.15.189:0/1933438976 >> 172.21.15.189:0/1933438976 conn(0x7f5110061f60 msgr2=0x7f5110062340 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:19.463 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f5116219700 1 -- 172.21.15.189:0/1933438976 shutdown_connections 2020-11-20T12:16:19.464 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f5116219700 1 -- 172.21.15.189:0/1933438976 wait complete. 2020-11-20T12:16:19.464 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f5116219700 1 Processor -- start 2020-11-20T12:16:19.464 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f5116219700 1 -- start start 2020-11-20T12:16:19.464 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f5116219700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50fc0044e0 0x7f511013b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:19.464 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f5116219700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f51101397c0 0x7f511014ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:19.465 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f5116219700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f511014b270 0x7f51100a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:19.465 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f5116219700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51100a5880 con 0x7f50fc0044e0 2020-11-20T12:16:19.465 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f5116219700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511013e750 con 0x7f511014b270 2020-11-20T12:16:19.465 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.448+0000 7f5116219700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51100a4c70 con 0x7f51101397c0 2020-11-20T12:16:19.465 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.452+0000 7f510ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f511014b270 0x7f51100a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:19.466 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.452+0000 7f510ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f511014b270 0x7f51100a3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51088/0 (socket says 172.21.15.189:51088) 2020-11-20T12:16:19.466 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.452+0000 7f510ffff700 1 -- 172.21.15.189:0/2444049324 learned_addr learned my addr 172.21.15.189:0/2444049324 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:16:19.466 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.452+0000 7f510effd700 1 --2- 172.21.15.189:0/2444049324 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f51101397c0 0x7f511014ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:19.466 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.452+0000 7f510f7fe700 1 --2- 172.21.15.189:0/2444049324 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50fc0044e0 0x7f511013b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:19.466 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.452+0000 7f510ffff700 1 -- 172.21.15.189:0/2444049324 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f51101397c0 msgr2=0x7f511014ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:19.467 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.452+0000 7f510ffff700 1 --2- 172.21.15.189:0/2444049324 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f51101397c0 0x7f511014ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:19.467 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.452+0000 7f510ffff700 1 -- 172.21.15.189:0/2444049324 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50fc0044e0 msgr2=0x7f511013b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:19.467 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.452+0000 7f510ffff700 1 --2- 172.21.15.189:0/2444049324 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50fc0044e0 0x7f511013b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:19.467 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.452+0000 7f510ffff700 1 -- 172.21.15.189:0/2444049324 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51100a5d90 con 0x7f511014b270 2020-11-20T12:16:19.467 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.452+0000 7f510ffff700 1 --2- 172.21.15.189:0/2444049324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f511014b270 0x7f51100a3c00 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f51100afbc0 tx=0x7f5110138440).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:19.468 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.452+0000 7f50f77fe700 1 -- 172.21.15.189:0/2444049324 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510800ee60 con 0x7f511014b270 2020-11-20T12:16:19.468 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.452+0000 7f50f77fe700 1 -- 172.21.15.189:0/2444049324 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f510800cb70 con 0x7f511014b270 2020-11-20T12:16:19.468 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.452+0000 7f50f77fe700 1 -- 172.21.15.189:0/2444049324 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5108014430 con 0x7f511014b270 2020-11-20T12:16:19.468 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.452+0000 7f5116219700 1 -- 172.21.15.189:0/2444049324 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51100a4ea0 con 0x7f511014b270 2020-11-20T12:16:19.468 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.452+0000 7f5116219700 1 -- 172.21.15.189:0/2444049324 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51100a53e0 con 0x7f511014b270 2020-11-20T12:16:19.469 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.452+0000 7f5116219700 1 -- 172.21.15.189:0/2444049324 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51100a5570 con 0x7f511014b270 2020-11-20T12:16:19.469 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.452+0000 7f50f77fe700 1 -- 172.21.15.189:0/2444049324 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f5108014860 con 0x7f511014b270 2020-11-20T12:16:19.469 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.452+0000 7f50f77fe700 1 --2- 172.21.15.189:0/2444049324 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f50e4059c00 0x7f50e405c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:19.469 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.452+0000 7f50f77fe700 1 -- 172.21.15.189:0/2444049324 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6761+0+0 (secure 0 0 0) 0x7f510806de10 con 0x7f511014b270 2020-11-20T12:16:19.469 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.452+0000 7f510f7fe700 1 --2- 172.21.15.189:0/2444049324 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f50e4059c00 0x7f50e405c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:19.470 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.456+0000 7f510f7fe700 1 --2- 172.21.15.189:0/2444049324 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f50e4059c00 0x7f50e405c0c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f5100006d00 tx=0x7f5100008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:19.470 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.460+0000 7f50f77fe700 1 -- 172.21.15.189:0/2444049324 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5108043a00 con 0x7f511014b270 2020-11-20T12:16:19.714 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.708+0000 7f5116219700 1 -- 172.21.15.189:0/2444049324 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5110044f60 con 0x7f511014b270 2020-11-20T12:16:19.714 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.708+0000 7f50f77fe700 1 -- 172.21.15.189:0/2444049324 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f5108047020 con 0x7f511014b270 2020-11-20T12:16:19.715 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:16:19.715 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5075,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/563711894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":563711894},{"type":"v1","addr":"172.21.15.47:6833","nonce":563711894}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:15:25.740722+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5389":{"gid":5389,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":34,"addr":"172.21.15.189:6835/636533719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":636533719},{"type":"v1","addr":"172.21.15.189:6835","nonce":636533719}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:16:19.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.712+0000 7f5116219700 1 -- 172.21.15.189:0/2444049324 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f50e4059c00 msgr2=0x7f50e405c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:19.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.712+0000 7f5116219700 1 --2- 172.21.15.189:0/2444049324 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f50e4059c00 0x7f50e405c0c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f5100006d00 tx=0x7f5100008040).stop 2020-11-20T12:16:19.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.712+0000 7f5116219700 1 -- 172.21.15.189:0/2444049324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f511014b270 msgr2=0x7f51100a3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:19.718 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.712+0000 7f5116219700 1 --2- 172.21.15.189:0/2444049324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f511014b270 0x7f51100a3c00 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f51100afbc0 tx=0x7f5110138440).stop 2020-11-20T12:16:19.718 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.712+0000 7f5116219700 1 -- 172.21.15.189:0/2444049324 shutdown_connections 2020-11-20T12:16:19.718 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.712+0000 7f5116219700 1 --2- 172.21.15.189:0/2444049324 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f50e4059c00 0x7f50e405c0c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:19.718 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.712+0000 7f5116219700 1 --2- 172.21.15.189:0/2444049324 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f50fc0044e0 0x7f511013b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:19.719 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.712+0000 7f5116219700 1 --2- 172.21.15.189:0/2444049324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f511014b270 0x7f51100a3c00 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:19.719 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.712+0000 7f5116219700 1 --2- 172.21.15.189:0/2444049324 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f51101397c0 0x7f511014ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:19.719 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.712+0000 7f5116219700 1 -- 172.21.15.189:0/2444049324 >> 172.21.15.189:0/2444049324 conn(0x7f5110061f60 msgr2=0x7f51100a8310 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:19.719 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.712+0000 7f5116219700 1 -- 172.21.15.189:0/2444049324 shutdown_connections 2020-11-20T12:16:19.720 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:19.712+0000 7f5116219700 1 -- 172.21.15.189:0/2444049324 wait complete. 2020-11-20T12:16:19.720 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 46 2020-11-20T12:16:19.729 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:16:21.730 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:16:21.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.856+0000 7f68b2bc9700 1 Processor -- start 2020-11-20T12:16:21.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.856+0000 7f68b2bc9700 1 -- start start 2020-11-20T12:16:21.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.856+0000 7f68b2bc9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f68ac13a680 0x7f68ac13ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:21.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.856+0000 7f68b2bc9700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68ac13e320 con 0x7f68ac13b000 2020-11-20T12:16:21.864 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.856+0000 7f68b2bc9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68ac066800 con 0x7f68ac0a8840 2020-11-20T12:16:21.864 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.856+0000 7f68b2bc9700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68ac066aa0 con 0x7f68ac13a680 2020-11-20T12:16:21.864 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.856+0000 7f68b1bc7700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f68ac0a8840 0x7f68ac13dc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52724/0 (socket says 172.21.15.189:52724) 2020-11-20T12:16:21.864 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.856+0000 7f68b13c6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f68ac13a680 0x7f68ac13ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:21.864 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.856+0000 7f68b1bc7700 1 -- 172.21.15.189:0/694359736 learned_addr learned my addr 172.21.15.189:0/694359736 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:16:21.865 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b13c6700 1 -- 172.21.15.189:0/694359736 >> v1:172.21.15.189:6789/0 conn(0x7f68ac0a8840 legacy=0x7f68ac13dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:21.866 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b0bc5700 1 -- 172.21.15.189:0/694359736 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2890945880 0 0) 0x7f68ac066800 con 0x7f68ac0a8840 2020-11-20T12:16:21.866 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b13c6700 1 -- 172.21.15.189:0/694359736 >> v1:172.21.15.47:6789/0 conn(0x7f68ac13b000 legacy=0x7f68ac1451c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:21.866 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b13c6700 1 -- 172.21.15.189:0/694359736 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ac0672f0 con 0x7f68ac13a680 2020-11-20T12:16:21.867 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b13c6700 1 --2- 172.21.15.189:0/694359736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68ac13a680 0x7f68ac13ab10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f68a000b9d0 tx=0x7f68a0006df0).ready entity=mon.1 client_cookie=21cbde6b97bff38a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:21.867 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b0bc5700 1 -- 172.21.15.189:0/694359736 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a0009830 con 0x7f68ac13a680 2020-11-20T12:16:21.868 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b0bc5700 1 -- 172.21.15.189:0/694359736 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a0009990 con 0x7f68ac13a680 2020-11-20T12:16:21.870 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b0bc5700 1 -- 172.21.15.189:0/694359736 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a0009d10 con 0x7f68ac13a680 2020-11-20T12:16:21.871 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b2bc9700 1 -- 172.21.15.189:0/694359736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68ac13a680 msgr2=0x7f68ac13ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:21.871 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b2bc9700 1 --2- 172.21.15.189:0/694359736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68ac13a680 0x7f68ac13ab10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f68a000b9d0 tx=0x7f68a0006df0).stop 2020-11-20T12:16:21.871 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b2bc9700 1 -- 172.21.15.189:0/694359736 shutdown_connections 2020-11-20T12:16:21.871 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b2bc9700 1 --2- 172.21.15.189:0/694359736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68ac13a680 0x7f68ac13ab10 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:21.872 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b2bc9700 1 -- 172.21.15.189:0/694359736 >> 172.21.15.189:0/694359736 conn(0x7f68ac060170 msgr2=0x7f68ac060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:21.872 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b2bc9700 1 -- 172.21.15.189:0/694359736 shutdown_connections 2020-11-20T12:16:21.872 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b2bc9700 1 -- 172.21.15.189:0/694359736 wait complete. 2020-11-20T12:16:21.872 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b2bc9700 1 Processor -- start 2020-11-20T12:16:21.872 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b2bc9700 1 -- start start 2020-11-20T12:16:21.873 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b2bc9700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f68ac0a8840 0x7f68ac138c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:21.873 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b2bc9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68ac13a680 0x7f68ac0abca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:21.873 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b2bc9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68ac13b000 0x7f68ac14d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:21.873 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b2bc9700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac0ac190 con 0x7f68ac13a680 2020-11-20T12:16:21.873 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b2bc9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac0ac420 con 0x7f68ac13b000 2020-11-20T12:16:21.874 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b2bc9700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac14d260 con 0x7f68ac0a8840 2020-11-20T12:16:21.874 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b23c8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68ac13b000 0x7f68ac14d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:21.874 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b23c8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68ac13b000 0x7f68ac14d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51106/0 (socket says 172.21.15.189:51106) 2020-11-20T12:16:21.874 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b23c8700 1 -- 172.21.15.189:0/1038276112 learned_addr learned my addr 172.21.15.189:0/1038276112 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:16:21.874 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b13c6700 1 --2- 172.21.15.189:0/1038276112 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68ac13a680 0x7f68ac0abca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:21.875 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b1bc7700 1 --2- 172.21.15.189:0/1038276112 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f68ac0a8840 0x7f68ac138c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:21.875 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b23c8700 1 -- 172.21.15.189:0/1038276112 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f68ac0a8840 msgr2=0x7f68ac138c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:21.875 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b23c8700 1 --2- 172.21.15.189:0/1038276112 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f68ac0a8840 0x7f68ac138c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:21.875 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b23c8700 1 -- 172.21.15.189:0/1038276112 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68ac13a680 msgr2=0x7f68ac0abca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:21.875 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b23c8700 1 --2- 172.21.15.189:0/1038276112 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68ac13a680 0x7f68ac0abca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:21.876 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b23c8700 1 -- 172.21.15.189:0/1038276112 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ac0ac6c0 con 0x7f68ac13b000 2020-11-20T12:16:21.876 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b13c6700 1 --2- 172.21.15.189:0/1038276112 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68ac13a680 0x7f68ac0abca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:16:21.876 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.860+0000 7f68b1bc7700 1 --2- 172.21.15.189:0/1038276112 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f68ac0a8840 0x7f68ac138c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:16:21.876 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.864+0000 7f68b23c8700 1 --2- 172.21.15.189:0/1038276112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68ac13b000 0x7f68ac14d700 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7f689c00ca90 tx=0x7f689c001360).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:21.877 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.864+0000 7f68a9ffb700 1 -- 172.21.15.189:0/1038276112 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c00a730 con 0x7f68ac13b000 2020-11-20T12:16:21.877 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.864+0000 7f68a9ffb700 1 -- 172.21.15.189:0/1038276112 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689c00a890 con 0x7f68ac13b000 2020-11-20T12:16:21.878 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.864+0000 7f68b2bc9700 1 -- 172.21.15.189:0/1038276112 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68ac0a45f0 con 0x7f68ac13b000 2020-11-20T12:16:21.878 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.864+0000 7f68a9ffb700 1 -- 172.21.15.189:0/1038276112 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c00ac30 con 0x7f68ac13b000 2020-11-20T12:16:21.878 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.864+0000 7f68b2bc9700 1 -- 172.21.15.189:0/1038276112 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68ac14dd90 con 0x7f68ac13b000 2020-11-20T12:16:21.878 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.864+0000 7f68a9ffb700 1 -- 172.21.15.189:0/1038276112 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f689c010070 con 0x7f68ac13b000 2020-11-20T12:16:21.878 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.864+0000 7f68b2bc9700 1 -- 172.21.15.189:0/1038276112 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68ac14d420 con 0x7f68ac13b000 2020-11-20T12:16:21.879 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.864+0000 7f68a9ffb700 1 --2- 172.21.15.189:0/1038276112 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f68840557f0 0x7f6884057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:21.879 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.864+0000 7f68b1bc7700 1 --2- 172.21.15.189:0/1038276112 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f68840557f0 0x7f6884057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:21.879 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.864+0000 7f68a9ffb700 1 -- 172.21.15.189:0/1038276112 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6761+0+0 (secure 0 0 0) 0x7f689c06e660 con 0x7f68ac13b000 2020-11-20T12:16:21.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.864+0000 7f68b1bc7700 1 --2- 172.21.15.189:0/1038276112 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f68840557f0 0x7f6884057cb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f68a4000bc0 tx=0x7f68a4009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:21.881 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:21.872+0000 7f68a9ffb700 1 -- 172.21.15.189:0/1038276112 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f689c044250 con 0x7f68ac13b000 2020-11-20T12:16:22.077 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:22.072+0000 7f68b2bc9700 1 -- 172.21.15.189:0/1038276112 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68ac14dfe0 con 0x7f68ac13b000 2020-11-20T12:16:22.078 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:22.072+0000 7f68a9ffb700 1 -- 172.21.15.189:0/1038276112 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f689c047870 con 0x7f68ac13b000 2020-11-20T12:16:22.078 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:16:22.078 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5075,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/563711894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":563711894},{"type":"v1","addr":"172.21.15.47:6833","nonce":563711894}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:15:25.740722+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5389":{"gid":5389,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":34,"addr":"172.21.15.189:6835/636533719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":636533719},{"type":"v1","addr":"172.21.15.189:6835","nonce":636533719}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:16:22.079 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:22.072+0000 7f68b2bc9700 1 -- 172.21.15.189:0/1038276112 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f68840557f0 msgr2=0x7f6884057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:22.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:22.072+0000 7f68b2bc9700 1 --2- 172.21.15.189:0/1038276112 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f68840557f0 0x7f6884057cb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f68a4000bc0 tx=0x7f68a4009040).stop 2020-11-20T12:16:22.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:22.072+0000 7f68b2bc9700 1 -- 172.21.15.189:0/1038276112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68ac13b000 msgr2=0x7f68ac14d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:22.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:22.072+0000 7f68b2bc9700 1 --2- 172.21.15.189:0/1038276112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68ac13b000 0x7f68ac14d700 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7f689c00ca90 tx=0x7f689c001360).stop 2020-11-20T12:16:22.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:22.072+0000 7f68b2bc9700 1 -- 172.21.15.189:0/1038276112 shutdown_connections 2020-11-20T12:16:22.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:22.072+0000 7f68b2bc9700 1 --2- 172.21.15.189:0/1038276112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68ac13b000 0x7f68ac14d700 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:22.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:22.072+0000 7f68b2bc9700 1 --2- 172.21.15.189:0/1038276112 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f68ac0a8840 0x7f68ac138c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:22.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:22.072+0000 7f68b2bc9700 1 --2- 172.21.15.189:0/1038276112 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f68840557f0 0x7f6884057cb0 secure :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0x7f68a4000bc0 tx=0x7f68a4009040).stop 2020-11-20T12:16:22.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:22.072+0000 7f68b2bc9700 1 --2- 172.21.15.189:0/1038276112 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f68ac13a680 0x7f68ac0abca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:22.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:22.072+0000 7f68b2bc9700 1 -- 172.21.15.189:0/1038276112 >> 172.21.15.189:0/1038276112 conn(0x7f68ac060170 msgr2=0x7f68ac13e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:22.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:22.076+0000 7f68b2bc9700 1 -- 172.21.15.189:0/1038276112 shutdown_connections 2020-11-20T12:16:22.082 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:22.076+0000 7f68b2bc9700 1 -- 172.21.15.189:0/1038276112 wait complete. 2020-11-20T12:16:22.082 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 46 2020-11-20T12:16:22.093 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:16:24.095 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:16:24.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.620+0000 7f7556ac6700 1 Processor -- start 2020-11-20T12:16:24.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.620+0000 7f7556ac6700 1 -- start start 2020-11-20T12:16:24.626 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.620+0000 7f7556ac6700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f75500a8840 0x7f755013dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:24.627 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.620+0000 7f7556ac6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f75501378c0 0x7f75501451c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:24.627 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.620+0000 7f7556ac6700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7550066800 con 0x7f755013e100 2020-11-20T12:16:24.627 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.620+0000 7f7556ac6700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7550066aa0 con 0x7f75500a8840 2020-11-20T12:16:24.627 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.620+0000 7f7556ac6700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75500672f0 con 0x7f75501378c0 2020-11-20T12:16:24.628 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.620+0000 7f75562c5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f75501378c0 0x7f75501451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:24.628 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.620+0000 7f75552c3700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f755013e100 0x7f75501371b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52738/0 (socket says 172.21.15.189:52738) 2020-11-20T12:16:24.628 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.620+0000 7f75562c5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f75501378c0 0x7f75501451c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51116/0 (socket says 172.21.15.189:51116) 2020-11-20T12:16:24.628 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.620+0000 7f75552c3700 1 -- 172.21.15.189:0/30630181 learned_addr learned my addr 172.21.15.189:0/30630181 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:16:24.628 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.620+0000 7f7555ac4700 1 --2- 172.21.15.189:0/30630181 >> v2:172.21.15.47:3300/0 conn(0x7f75500a8840 0x7f755013dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:24.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.620+0000 7f75562c5700 1 -- 172.21.15.189:0/30630181 >> v2:172.21.15.47:3300/0 conn(0x7f75500a8840 msgr2=0x7f755013dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:24.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.620+0000 7f75562c5700 1 --2- 172.21.15.189:0/30630181 >> v2:172.21.15.47:3300/0 conn(0x7f75500a8840 0x7f755013dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:24.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.620+0000 7f75562c5700 1 -- 172.21.15.189:0/30630181 >> v1:172.21.15.189:6789/0 conn(0x7f755013e100 legacy=0x7f75501371b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:24.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.620+0000 7f7554ac2700 1 -- 172.21.15.189:0/30630181 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2095127139 0 0) 0x7f7550066800 con 0x7f755013e100 2020-11-20T12:16:24.629 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.620+0000 7f75562c5700 1 -- 172.21.15.189:0/30630181 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7550067590 con 0x7f75501378c0 2020-11-20T12:16:24.635 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.620+0000 7f75562c5700 1 --2- 172.21.15.189:0/30630181 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75501378c0 0x7f75501451c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f7550066db0 tx=0x7f75480091d0).ready entity=mon.1 client_cookie=4eca97b0723081f0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:24.635 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.620+0000 7f7554ac2700 1 -- 172.21.15.189:0/30630181 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754800bbc0 con 0x7f75501378c0 2020-11-20T12:16:24.635 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.620+0000 7f7554ac2700 1 -- 172.21.15.189:0/30630181 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f754800bd20 con 0x7f75501378c0 2020-11-20T12:16:24.637 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.620+0000 7f7554ac2700 1 -- 172.21.15.189:0/30630181 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7548011610 con 0x7f75501378c0 2020-11-20T12:16:24.637 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f7556ac6700 1 -- 172.21.15.189:0/30630181 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75501378c0 msgr2=0x7f75501451c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:24.638 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f7556ac6700 1 --2- 172.21.15.189:0/30630181 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75501378c0 0x7f75501451c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f7550066db0 tx=0x7f75480091d0).stop 2020-11-20T12:16:24.638 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f7556ac6700 1 -- 172.21.15.189:0/30630181 shutdown_connections 2020-11-20T12:16:24.638 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f7556ac6700 1 --2- 172.21.15.189:0/30630181 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75501378c0 0x7f75501451c0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:24.638 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f7556ac6700 1 --2- 172.21.15.189:0/30630181 >> v2:172.21.15.47:3300/0 conn(0x7f75500a8840 0x7f755013dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:24.639 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f7556ac6700 1 -- 172.21.15.189:0/30630181 >> 172.21.15.189:0/30630181 conn(0x7f7550060170 msgr2=0x7f7550060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:24.639 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f7556ac6700 1 -- 172.21.15.189:0/30630181 shutdown_connections 2020-11-20T12:16:24.639 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f7556ac6700 1 -- 172.21.15.189:0/30630181 wait complete. 2020-11-20T12:16:24.639 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f7556ac6700 1 Processor -- start 2020-11-20T12:16:24.639 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f7556ac6700 1 -- start start 2020-11-20T12:16:24.640 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f7556ac6700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75500a8840 0x7f75500abb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:24.640 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f7556ac6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75501378c0 0x7f755013d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:24.640 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f7556ac6700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f755013e100 0x7f755013ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:24.640 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f7556ac6700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755013a770 con 0x7f755013e100 2020-11-20T12:16:24.640 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f7556ac6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7550138890 con 0x7f75501378c0 2020-11-20T12:16:24.641 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f7556ac6700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f755013a960 con 0x7f75500a8840 2020-11-20T12:16:24.641 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f75552c3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75501378c0 0x7f755013d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:24.641 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f75552c3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75501378c0 0x7f755013d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51122/0 (socket says 172.21.15.189:51122) 2020-11-20T12:16:24.641 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f75552c3700 1 -- 172.21.15.189:0/440068106 learned_addr learned my addr 172.21.15.189:0/440068106 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:16:24.641 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f7555ac4700 1 --2- 172.21.15.189:0/440068106 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75500a8840 0x7f75500abb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:24.642 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f75562c5700 1 --2- 172.21.15.189:0/440068106 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f755013e100 0x7f755013ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:24.642 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f75552c3700 1 -- 172.21.15.189:0/440068106 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75500a8840 msgr2=0x7f75500abb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:24.642 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f75552c3700 1 --2- 172.21.15.189:0/440068106 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75500a8840 0x7f75500abb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:24.642 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f75552c3700 1 -- 172.21.15.189:0/440068106 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f755013e100 msgr2=0x7f755013ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:24.642 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f75552c3700 1 --2- 172.21.15.189:0/440068106 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f755013e100 0x7f755013ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:24.643 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f75552c3700 1 -- 172.21.15.189:0/440068106 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755013d6c0 con 0x7f75501378c0 2020-11-20T12:16:24.643 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f75552c3700 1 --2- 172.21.15.189:0/440068106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75501378c0 0x7f755013d1d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f754400c720 tx=0x7f7544007ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:24.643 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f7541ffb700 1 -- 172.21.15.189:0/440068106 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7544007320 con 0x7f75501378c0 2020-11-20T12:16:24.643 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f7541ffb700 1 -- 172.21.15.189:0/440068106 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7544007480 con 0x7f75501378c0 2020-11-20T12:16:24.643 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f7541ffb700 1 -- 172.21.15.189:0/440068106 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7544005800 con 0x7f75501378c0 2020-11-20T12:16:24.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f7556ac6700 1 -- 172.21.15.189:0/440068106 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75501392d0 con 0x7f75501378c0 2020-11-20T12:16:24.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.624+0000 7f7556ac6700 1 -- 172.21.15.189:0/440068106 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7550138310 con 0x7f75501378c0 2020-11-20T12:16:24.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.628+0000 7f7556ac6700 1 -- 172.21.15.189:0/440068106 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f755013ab20 con 0x7f75501378c0 2020-11-20T12:16:24.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.628+0000 7f7541ffb700 1 -- 172.21.15.189:0/440068106 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f7544010030 con 0x7f75501378c0 2020-11-20T12:16:24.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.628+0000 7f7541ffb700 1 --2- 172.21.15.189:0/440068106 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f75280557c0 0x7f7528057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:24.646 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.628+0000 7f7541ffb700 1 -- 172.21.15.189:0/440068106 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6761+0+0 (secure 0 0 0) 0x7f754406e730 con 0x7f75501378c0 2020-11-20T12:16:24.646 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.628+0000 7f7555ac4700 1 --2- 172.21.15.189:0/440068106 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f75280557c0 0x7f7528057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:24.646 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.628+0000 7f7555ac4700 1 --2- 172.21.15.189:0/440068106 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f75280557c0 0x7f7528057c80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f754c00d8f0 tx=0x7f754c008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:24.646 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.640+0000 7f7541ffb700 1 -- 172.21.15.189:0/440068106 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7544044210 con 0x7f75501378c0 2020-11-20T12:16:24.844 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.840+0000 7f7556ac6700 1 -- 172.21.15.189:0/440068106 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7550139460 con 0x7f75501378c0 2020-11-20T12:16:24.845 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.840+0000 7f7541ffb700 1 -- 172.21.15.189:0/440068106 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f7544047830 con 0x7f75501378c0 2020-11-20T12:16:24.845 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:16:24.846 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5075,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/563711894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":563711894},{"type":"v1","addr":"172.21.15.47:6833","nonce":563711894}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:15:25.740722+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5389":{"gid":5389,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":34,"addr":"172.21.15.189:6835/636533719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":636533719},{"type":"v1","addr":"172.21.15.189:6835","nonce":636533719}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:16:24.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.840+0000 7f7556ac6700 1 -- 172.21.15.189:0/440068106 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f75280557c0 msgr2=0x7f7528057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:24.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.840+0000 7f7556ac6700 1 --2- 172.21.15.189:0/440068106 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f75280557c0 0x7f7528057c80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f754c00d8f0 tx=0x7f754c008040).stop 2020-11-20T12:16:24.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.840+0000 7f7556ac6700 1 -- 172.21.15.189:0/440068106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75501378c0 msgr2=0x7f755013d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:24.847 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.840+0000 7f7556ac6700 1 --2- 172.21.15.189:0/440068106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75501378c0 0x7f755013d1d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f754400c720 tx=0x7f7544007ac0).stop 2020-11-20T12:16:24.848 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.840+0000 7f7556ac6700 1 -- 172.21.15.189:0/440068106 shutdown_connections 2020-11-20T12:16:24.848 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.840+0000 7f7556ac6700 1 --2- 172.21.15.189:0/440068106 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f75280557c0 0x7f7528057c80 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:24.848 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.840+0000 7f7556ac6700 1 --2- 172.21.15.189:0/440068106 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f755013e100 0x7f755013ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:24.848 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.840+0000 7f7556ac6700 1 --2- 172.21.15.189:0/440068106 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75500a8840 0x7f75500abb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:24.848 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.840+0000 7f7556ac6700 1 --2- 172.21.15.189:0/440068106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75501378c0 0x7f755013d1d0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:24.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.840+0000 7f7556ac6700 1 -- 172.21.15.189:0/440068106 >> 172.21.15.189:0/440068106 conn(0x7f7550060170 msgr2=0x7f7550060550 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:24.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.840+0000 7f7556ac6700 1 -- 172.21.15.189:0/440068106 shutdown_connections 2020-11-20T12:16:24.849 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:24.840+0000 7f7556ac6700 1 -- 172.21.15.189:0/440068106 wait complete. 2020-11-20T12:16:24.850 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 46 2020-11-20T12:16:24.859 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:16:25.037 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:16:25.042 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:16:26.861 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:16:26.989 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.984+0000 7f20943fa700 1 Processor -- start 2020-11-20T12:16:26.989 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.984+0000 7f20943fa700 1 -- start start 2020-11-20T12:16:26.994 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.984+0000 7f20943fa700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f208c066e70 0x7f208c13ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:26.995 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.984+0000 7f20943fa700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f208c13b100 0x7f208c141ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:26.995 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.984+0000 7f20943fa700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f208c0a4850 con 0x7f208c066a90 2020-11-20T12:16:26.995 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.984+0000 7f20943fa700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f208c0a9270 con 0x7f208c066e70 2020-11-20T12:16:26.996 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.984+0000 7f20943fa700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f208c0a5190 con 0x7f208c13b100 2020-11-20T12:16:26.996 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.984+0000 7f2091995700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f208c066e70 0x7f208c13ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:26.997 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.984+0000 7f2091995700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f208c066e70 0x7f208c13ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.189:57356/0 (socket says 172.21.15.189:57356) 2020-11-20T12:16:26.997 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.984+0000 7f2092997700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f208c13b100 0x7f208c141ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:26.998 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.984+0000 7f2091995700 1 -- 172.21.15.189:0/4223048930 learned_addr learned my addr 172.21.15.189:0/4223048930 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:16:26.998 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.984+0000 7f2092196700 1 --1- 172.21.15.189:0/4223048930 >> v1:172.21.15.47:6789/0 conn(0x7f208c066a90 0x7f208c064640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.189:47416/0 (socket says 172.21.15.189:47416) 2020-11-20T12:16:26.998 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.988+0000 7f2091995700 1 -- 172.21.15.189:0/4223048930 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f208c13b100 msgr2=0x7f208c141ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:26.999 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.988+0000 7f2091995700 1 --2- 172.21.15.189:0/4223048930 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f208c13b100 0x7f208c141ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:26.999 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.988+0000 7f2091995700 1 -- 172.21.15.189:0/4223048930 >> v1:172.21.15.47:6789/0 conn(0x7f208c066a90 legacy=0x7f208c064640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:26.999 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.988+0000 7f2091995700 1 -- 172.21.15.189:0/4223048930 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208c0a86c0 con 0x7f208c066e70 2020-11-20T12:16:27.000 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.988+0000 7f2091995700 1 --2- 172.21.15.189:0/4223048930 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f208c066e70 0x7f208c13ac10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f207c006fd0 tx=0x7f207c006eb0).ready entity=mon.0 client_cookie=1d88565a4d99a59b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:27.000 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.988+0000 7f2091194700 1 -- 172.21.15.189:0/4223048930 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207c009810 con 0x7f208c066e70 2020-11-20T12:16:27.001 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.988+0000 7f2091194700 1 -- 172.21.15.189:0/4223048930 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f207c009970 con 0x7f208c066e70 2020-11-20T12:16:27.001 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.988+0000 7f2091194700 1 -- 172.21.15.189:0/4223048930 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207c009cf0 con 0x7f208c066e70 2020-11-20T12:16:27.001 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.988+0000 7f20943fa700 1 -- 172.21.15.189:0/4223048930 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f208c066e70 msgr2=0x7f208c13ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:27.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.988+0000 7f20943fa700 1 --2- 172.21.15.189:0/4223048930 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f208c066e70 0x7f208c13ac10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f207c006fd0 tx=0x7f207c006eb0).stop 2020-11-20T12:16:27.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.988+0000 7f20943fa700 1 -- 172.21.15.189:0/4223048930 shutdown_connections 2020-11-20T12:16:27.003 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.988+0000 7f20943fa700 1 --2- 172.21.15.189:0/4223048930 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f208c13b100 0x7f208c141ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:27.003 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.988+0000 7f20943fa700 1 --2- 172.21.15.189:0/4223048930 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f208c066e70 0x7f208c13ac10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:27.003 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.988+0000 7f20943fa700 1 -- 172.21.15.189:0/4223048930 >> 172.21.15.189:0/4223048930 conn(0x7f208c062530 msgr2=0x7f208c0ad9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:27.004 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.988+0000 7f20943fa700 1 -- 172.21.15.189:0/4223048930 shutdown_connections 2020-11-20T12:16:27.004 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.988+0000 7f20943fa700 1 -- 172.21.15.189:0/4223048930 wait complete. 2020-11-20T12:16:27.010 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f20943fa700 1 Processor -- start 2020-11-20T12:16:27.010 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f20943fa700 1 -- start start 2020-11-20T12:16:27.011 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f20943fa700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f208c066a90 0x7f208c137a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:27.011 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f20943fa700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f208c066e70 0x7f208c147460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:27.012 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f20943fa700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f208c13b100 0x7f208c147950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:27.012 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f20943fa700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208c14afb0 con 0x7f208c066e70 2020-11-20T12:16:27.012 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f20943fa700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208c14b250 con 0x7f208c13b100 2020-11-20T12:16:27.013 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f20943fa700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f208c14b4f0 con 0x7f208c066a90 2020-11-20T12:16:27.013 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f2092997700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f208c13b100 0x7f208c147950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:27.014 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f2092997700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f208c13b100 0x7f208c147950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51136/0 (socket says 172.21.15.189:51136) 2020-11-20T12:16:27.014 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f2092997700 1 -- 172.21.15.189:0/2429285805 learned_addr learned my addr 172.21.15.189:0/2429285805 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:16:27.014 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f2091995700 1 --2- 172.21.15.189:0/2429285805 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f208c066e70 0x7f208c147460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:27.015 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f2092196700 1 --2- 172.21.15.189:0/2429285805 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f208c066a90 0x7f208c137a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:27.015 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f2092997700 1 -- 172.21.15.189:0/2429285805 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f208c066a90 msgr2=0x7f208c137a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:27.016 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f2092997700 1 --2- 172.21.15.189:0/2429285805 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f208c066a90 0x7f208c137a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:27.016 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f2092997700 1 -- 172.21.15.189:0/2429285805 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f208c066e70 msgr2=0x7f208c147460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:27.016 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f2092997700 1 --2- 172.21.15.189:0/2429285805 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f208c066e70 0x7f208c147460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:27.017 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f2092997700 1 -- 172.21.15.189:0/2429285805 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f208c13db40 con 0x7f208c13b100 2020-11-20T12:16:27.017 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f2091995700 1 --2- 172.21.15.189:0/2429285805 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f208c066e70 0x7f208c147460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:16:27.018 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f2092997700 1 --2- 172.21.15.189:0/2429285805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f208c13b100 0x7f208c147950 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f208400ed20 tx=0x7f20840091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:27.018 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f207a7fc700 1 -- 172.21.15.189:0/2429285805 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208400cbb0 con 0x7f208c13b100 2020-11-20T12:16:27.018 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f20943fa700 1 -- 172.21.15.189:0/2429285805 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f208c09f5f0 con 0x7f208c13b100 2020-11-20T12:16:27.019 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f20943fa700 1 -- 172.21.15.189:0/2429285805 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f208c14aa30 con 0x7f208c13b100 2020-11-20T12:16:27.019 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f207a7fc700 1 -- 172.21.15.189:0/2429285805 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f208401b400 con 0x7f208c13b100 2020-11-20T12:16:27.020 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f207a7fc700 1 -- 172.21.15.189:0/2429285805 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208401b570 con 0x7f208c13b100 2020-11-20T12:16:27.020 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f20943fa700 1 -- 172.21.15.189:0/2429285805 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f208c14b6b0 con 0x7f208c13b100 2020-11-20T12:16:27.020 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.992+0000 7f207a7fc700 1 -- 172.21.15.189:0/2429285805 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f2084022070 con 0x7f208c13b100 2020-11-20T12:16:27.021 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.996+0000 7f207a7fc700 1 --2- 172.21.15.189:0/2429285805 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f20640557c0 0x7f2064057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:27.021 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.996+0000 7f207a7fc700 1 -- 172.21.15.189:0/2429285805 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6761+0+0 (secure 0 0 0) 0x7f208406f060 con 0x7f208c13b100 2020-11-20T12:16:27.022 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.996+0000 7f2092196700 1 --2- 172.21.15.189:0/2429285805 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f20640557c0 0x7f2064057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:27.022 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:26.996+0000 7f2092196700 1 --2- 172.21.15.189:0/2429285805 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f20640557c0 0x7f2064057c80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f2088007f70 tx=0x7f2088007180).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:27.022 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:27.004+0000 7f207a7fc700 1 -- 172.21.15.189:0/2429285805 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2084044bf0 con 0x7f208c13b100 2020-11-20T12:16:27.211 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:27.208+0000 7f20943fa700 1 -- 172.21.15.189:0/2429285805 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f208c14b8b0 con 0x7f208c13b100 2020-11-20T12:16:27.212 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:27.208+0000 7f207a7fc700 1 -- 172.21.15.189:0/2429285805 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f2084016070 con 0x7f208c13b100 2020-11-20T12:16:27.212 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:16:27.213 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5075,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/563711894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":563711894},{"type":"v1","addr":"172.21.15.47:6833","nonce":563711894}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:15:25.740722+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5389":{"gid":5389,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":34,"addr":"172.21.15.189:6835/636533719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":636533719},{"type":"v1","addr":"172.21.15.189:6835","nonce":636533719}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:16:27.214 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:27.208+0000 7f20943fa700 1 -- 172.21.15.189:0/2429285805 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f20640557c0 msgr2=0x7f2064057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:27.215 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:27.208+0000 7f20943fa700 1 --2- 172.21.15.189:0/2429285805 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f20640557c0 0x7f2064057c80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f2088007f70 tx=0x7f2088007180).stop 2020-11-20T12:16:27.215 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:27.208+0000 7f20943fa700 1 -- 172.21.15.189:0/2429285805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f208c13b100 msgr2=0x7f208c147950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:27.215 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:27.208+0000 7f20943fa700 1 --2- 172.21.15.189:0/2429285805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f208c13b100 0x7f208c147950 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f208400ed20 tx=0x7f20840091d0).stop 2020-11-20T12:16:27.215 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:27.208+0000 7f20943fa700 1 -- 172.21.15.189:0/2429285805 shutdown_connections 2020-11-20T12:16:27.215 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:27.208+0000 7f20943fa700 1 --2- 172.21.15.189:0/2429285805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f208c13b100 0x7f208c147950 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:27.216 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:27.208+0000 7f20943fa700 1 --2- 172.21.15.189:0/2429285805 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f208c066a90 0x7f208c137a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:27.216 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:27.208+0000 7f20943fa700 1 --2- 172.21.15.189:0/2429285805 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f20640557c0 0x7f2064057c80 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:27.216 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:27.208+0000 7f20943fa700 1 --2- 172.21.15.189:0/2429285805 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f208c066e70 0x7f208c147460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:27.216 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:27.208+0000 7f20943fa700 1 -- 172.21.15.189:0/2429285805 >> 172.21.15.189:0/2429285805 conn(0x7f208c062530 msgr2=0x7f208c138650 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:27.217 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:27.208+0000 7f20943fa700 1 -- 172.21.15.189:0/2429285805 shutdown_connections 2020-11-20T12:16:27.217 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:27.208+0000 7f20943fa700 1 -- 172.21.15.189:0/2429285805 wait complete. 2020-11-20T12:16:27.217 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 46 2020-11-20T12:16:27.232 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:16:29.234 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:16:29.376 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.372+0000 7efc21d5c700 1 Processor -- start 2020-11-20T12:16:29.377 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.372+0000 7efc21d5c700 1 -- start start 2020-11-20T12:16:29.382 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.372+0000 7efc21d5c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efc1c13a6e0 0x7efc1c13ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:29.382 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.372+0000 7efc21d5c700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1c067730 con 0x7efc1c0ad3e0 2020-11-20T12:16:29.383 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.372+0000 7efc21d5c700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1c13dd40 con 0x7efc1c0ad000 2020-11-20T12:16:29.383 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.372+0000 7efc21d5c700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc1c066050 con 0x7efc1c13a6e0 2020-11-20T12:16:29.383 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.372+0000 7efc2155b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efc1c13a6e0 0x7efc1c13ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:29.383 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.372+0000 7efc20d5a700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7efc1c0ad000 0x7efc1c13e480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52764/0 (socket says 172.21.15.189:52764) 2020-11-20T12:16:29.383 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.372+0000 7efc20d5a700 1 -- 172.21.15.189:0/367989990 learned_addr learned my addr 172.21.15.189:0/367989990 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:16:29.384 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.372+0000 7efc2155b700 1 -- 172.21.15.189:0/367989990 >> v1:172.21.15.189:6789/0 conn(0x7efc1c0ad000 legacy=0x7efc1c13e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:29.384 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.372+0000 7efc2155b700 1 -- 172.21.15.189:0/367989990 >> v1:172.21.15.47:6789/0 conn(0x7efc1c0ad3e0 legacy=0x7efc1c066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:29.384 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.372+0000 7efc2155b700 1 -- 172.21.15.189:0/367989990 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc1c13dfe0 con 0x7efc1c13a6e0 2020-11-20T12:16:29.384 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.372+0000 7efc2155b700 1 --2- 172.21.15.189:0/367989990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc1c13a6e0 0x7efc1c13ad00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7efc0c009c90 tx=0x7efc0c0091d0).ready entity=mon.1 client_cookie=e2463a0a58039f20 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:29.384 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.372+0000 7efc1b7fe700 1 -- 172.21.15.189:0/367989990 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0c00bbc0 con 0x7efc1c13a6e0 2020-11-20T12:16:29.385 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.372+0000 7efc1b7fe700 1 -- 172.21.15.189:0/367989990 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0c00bd20 con 0x7efc1c13a6e0 2020-11-20T12:16:29.385 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.372+0000 7efc1b7fe700 1 -- 172.21.15.189:0/367989990 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0c011610 con 0x7efc1c13a6e0 2020-11-20T12:16:29.385 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.376+0000 7efc21d5c700 1 -- 172.21.15.189:0/367989990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc1c13a6e0 msgr2=0x7efc1c13ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:29.385 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.376+0000 7efc21d5c700 1 --2- 172.21.15.189:0/367989990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc1c13a6e0 0x7efc1c13ad00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7efc0c009c90 tx=0x7efc0c0091d0).stop 2020-11-20T12:16:29.386 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.376+0000 7efc21d5c700 1 -- 172.21.15.189:0/367989990 shutdown_connections 2020-11-20T12:16:29.388 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.376+0000 7efc21d5c700 1 --2- 172.21.15.189:0/367989990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc1c13a6e0 0x7efc1c13ad00 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:29.388 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.376+0000 7efc21d5c700 1 -- 172.21.15.189:0/367989990 >> 172.21.15.189:0/367989990 conn(0x7efc1c062080 msgr2=0x7efc1c062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:29.388 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.376+0000 7efc21d5c700 1 -- 172.21.15.189:0/367989990 shutdown_connections 2020-11-20T12:16:29.389 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.376+0000 7efc21d5c700 1 -- 172.21.15.189:0/367989990 wait complete. 2020-11-20T12:16:29.389 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.376+0000 7efc21d5c700 1 Processor -- start 2020-11-20T12:16:29.389 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.376+0000 7efc21d5c700 1 -- start start 2020-11-20T12:16:29.389 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.380+0000 7efc21d5c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc1c0ad000 0x7efc1c13d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:29.389 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.380+0000 7efc21d5c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc1c0ad3e0 0x7efc1c0a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:29.390 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.380+0000 7efc21d5c700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efc1c0a9410 0x7efc1c0aa520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:29.390 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.380+0000 7efc21d5c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c0a9b20 con 0x7efc1c0ad3e0 2020-11-20T12:16:29.390 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.380+0000 7efc21d5c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c0a9dc0 con 0x7efc1c0ad000 2020-11-20T12:16:29.390 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.380+0000 7efc21d5c700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c0aa060 con 0x7efc1c0a9410 2020-11-20T12:16:29.390 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.380+0000 7efc20d5a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc1c0ad000 0x7efc1c13d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:29.391 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.380+0000 7efc20d5a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc1c0ad000 0x7efc1c13d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51148/0 (socket says 172.21.15.189:51148) 2020-11-20T12:16:29.391 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.380+0000 7efc20d5a700 1 -- 172.21.15.189:0/816092207 learned_addr learned my addr 172.21.15.189:0/816092207 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:16:29.391 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.380+0000 7efc2155b700 1 --2- 172.21.15.189:0/816092207 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efc1c0a9410 0x7efc1c0aa520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:29.391 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.380+0000 7efc1bfff700 1 --2- 172.21.15.189:0/816092207 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc1c0ad3e0 0x7efc1c0a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:29.391 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.380+0000 7efc2155b700 1 -- 172.21.15.189:0/816092207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc1c0ad000 msgr2=0x7efc1c13d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:29.392 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.380+0000 7efc2155b700 1 --2- 172.21.15.189:0/816092207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc1c0ad000 0x7efc1c13d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:29.392 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.380+0000 7efc2155b700 1 -- 172.21.15.189:0/816092207 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc1c0ad3e0 msgr2=0x7efc1c0a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:29.392 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.380+0000 7efc2155b700 1 --2- 172.21.15.189:0/816092207 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc1c0ad3e0 0x7efc1c0a5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:29.392 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.380+0000 7efc2155b700 1 -- 172.21.15.189:0/816092207 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc1c0a9870 con 0x7efc1c0a9410 2020-11-20T12:16:29.392 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.380+0000 7efc2155b700 1 --2- 172.21.15.189:0/816092207 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efc1c0a9410 0x7efc1c0aa520 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7efc0c009c90 tx=0x7efc0c016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:29.393 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.380+0000 7efc18ff9700 1 -- 172.21.15.189:0/816092207 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0c00be90 con 0x7efc1c0a9410 2020-11-20T12:16:29.393 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.380+0000 7efc18ff9700 1 -- 172.21.15.189:0/816092207 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0c010ae0 con 0x7efc1c0a9410 2020-11-20T12:16:29.393 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.380+0000 7efc21d5c700 1 -- 172.21.15.189:0/816092207 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc1c0aaa10 con 0x7efc1c0a9410 2020-11-20T12:16:29.393 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.380+0000 7efc18ff9700 1 -- 172.21.15.189:0/816092207 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0c010e60 con 0x7efc1c0a9410 2020-11-20T12:16:29.393 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.380+0000 7efc21d5c700 1 -- 172.21.15.189:0/816092207 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc1c137120 con 0x7efc1c0a9410 2020-11-20T12:16:29.394 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.384+0000 7efc21d5c700 1 -- 172.21.15.189:0/816092207 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1c0446d0 con 0x7efc1c0a9410 2020-11-20T12:16:29.395 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.384+0000 7efc18ff9700 1 -- 172.21.15.189:0/816092207 <== mon.2 v2:172.21.15.47:3301/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7efc0c017070 con 0x7efc1c0a9410 2020-11-20T12:16:29.395 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.384+0000 7efc18ff9700 1 --2- 172.21.15.189:0/816092207 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7efbf80557c0 0x7efbf8057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:29.395 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.384+0000 7efc20d5a700 1 --2- 172.21.15.189:0/816092207 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7efbf80557c0 0x7efbf8057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:29.395 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.384+0000 7efc18ff9700 1 -- 172.21.15.189:0/816092207 <== mon.2 v2:172.21.15.47:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6761+0+0 (secure 0 0 0) 0x7efc0c070ff0 con 0x7efc1c0a9410 2020-11-20T12:16:29.395 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.384+0000 7efc20d5a700 1 --2- 172.21.15.189:0/816092207 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7efbf80557c0 0x7efbf8057c80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7efc140046b0 tx=0x7efc14007c40).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:29.398 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.392+0000 7efc18ff9700 1 -- 172.21.15.189:0/816092207 <== mon.2 v2:172.21.15.47:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7efc0c046af0 con 0x7efc1c0a9410 2020-11-20T12:16:29.603 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.600+0000 7efc21d5c700 1 -- 172.21.15.189:0/816092207 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc1c05cdd0 con 0x7efc1c0a9410 2020-11-20T12:16:29.605 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.600+0000 7efc18ff9700 1 -- 172.21.15.189:0/816092207 <== mon.2 v2:172.21.15.47:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+2134 (secure 0 0 0) 0x7efc0c015070 con 0x7efc1c0a9410 2020-11-20T12:16:29.607 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:16:29.607 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5075,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/563711894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":563711894},{"type":"v1","addr":"172.21.15.47:6833","nonce":563711894}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:15:25.740722+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5389":{"gid":5389,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":34,"addr":"172.21.15.189:6835/636533719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":636533719},{"type":"v1","addr":"172.21.15.189:6835","nonce":636533719}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:16:29.609 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.600+0000 7efc21d5c700 1 -- 172.21.15.189:0/816092207 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7efbf80557c0 msgr2=0x7efbf8057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:29.609 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.600+0000 7efc21d5c700 1 --2- 172.21.15.189:0/816092207 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7efbf80557c0 0x7efbf8057c80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7efc140046b0 tx=0x7efc14007c40).stop 2020-11-20T12:16:29.609 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.600+0000 7efc21d5c700 1 -- 172.21.15.189:0/816092207 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efc1c0a9410 msgr2=0x7efc1c0aa520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:29.609 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.600+0000 7efc21d5c700 1 --2- 172.21.15.189:0/816092207 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efc1c0a9410 0x7efc1c0aa520 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7efc0c009c90 tx=0x7efc0c016000).stop 2020-11-20T12:16:29.610 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.600+0000 7efc21d5c700 1 -- 172.21.15.189:0/816092207 shutdown_connections 2020-11-20T12:16:29.610 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.600+0000 7efc21d5c700 1 --2- 172.21.15.189:0/816092207 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efc1c0a9410 0x7efc1c0aa520 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:29.610 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.600+0000 7efc21d5c700 1 --2- 172.21.15.189:0/816092207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc1c0ad000 0x7efc1c13d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:29.610 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.600+0000 7efc21d5c700 1 --2- 172.21.15.189:0/816092207 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7efbf80557c0 0x7efbf8057c80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:29.610 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.600+0000 7efc21d5c700 1 --2- 172.21.15.189:0/816092207 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc1c0ad3e0 0x7efc1c0a5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:29.611 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.600+0000 7efc21d5c700 1 -- 172.21.15.189:0/816092207 >> 172.21.15.189:0/816092207 conn(0x7efc1c062080 msgr2=0x7efc1c13aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:29.611 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.600+0000 7efc21d5c700 1 -- 172.21.15.189:0/816092207 shutdown_connections 2020-11-20T12:16:29.611 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:29.600+0000 7efc21d5c700 1 -- 172.21.15.189:0/816092207 wait complete. 2020-11-20T12:16:29.611 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 46 2020-11-20T12:16:29.619 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:16:31.620 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:16:31.755 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1c180d700 1 Processor -- start 2020-11-20T12:16:31.756 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1c180d700 1 -- start start 2020-11-20T12:16:31.758 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1c180d700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd1bc0a7bf0 0x7fd1bc0a8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:31.758 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1c180d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd1bc13bcf0 0x7fd1bc13c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:31.759 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1c180d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1bc0a9080 con 0x7fd1bc0a9e20 2020-11-20T12:16:31.759 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1c180d700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1bc066770 con 0x7fd1bc0a7bf0 2020-11-20T12:16:31.759 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1c180d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1bc0a8560 con 0x7fd1bc13bcf0 2020-11-20T12:16:31.759 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1baffd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd1bc0a9e20 0x7fd1bc067220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52778/0 (socket says 172.21.15.189:52778) 2020-11-20T12:16:31.760 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1baffd700 1 -- 172.21.15.189:0/1858611661 learned_addr learned my addr 172.21.15.189:0/1858611661 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:16:31.760 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1bb7fe700 1 --2- 172.21.15.189:0/1858611661 >> v2:172.21.15.189:3300/0 conn(0x7fd1bc13bcf0 0x7fd1bc13c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:31.760 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1ba7fc700 1 --2- 172.21.15.189:0/1858611661 >> v2:172.21.15.47:3300/0 conn(0x7fd1bc0a7bf0 0x7fd1bc0a8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:31.760 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1b9ffb700 1 -- 172.21.15.189:0/1858611661 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630638382 0 0) 0x7fd1bc0a9080 con 0x7fd1bc0a9e20 2020-11-20T12:16:31.760 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1b9ffb700 1 -- 172.21.15.189:0/1858611661 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd19c003240 con 0x7fd1bc0a9e20 2020-11-20T12:16:31.761 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1b9ffb700 1 -- 172.21.15.189:0/1858611661 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2919184826 0 0) 0x7fd19c003240 con 0x7fd1bc0a9e20 2020-11-20T12:16:31.761 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1bb7fe700 1 -- 172.21.15.189:0/1858611661 >> v2:172.21.15.47:3300/0 conn(0x7fd1bc0a7bf0 msgr2=0x7fd1bc0a8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:31.761 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1bb7fe700 1 --2- 172.21.15.189:0/1858611661 >> v2:172.21.15.47:3300/0 conn(0x7fd1bc0a7bf0 0x7fd1bc0a8070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:31.761 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1bb7fe700 1 -- 172.21.15.189:0/1858611661 >> v1:172.21.15.189:6789/0 conn(0x7fd1bc0a9e20 legacy=0x7fd1bc067220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:31.761 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1bb7fe700 1 -- 172.21.15.189:0/1858611661 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1bc068480 con 0x7fd1bc13bcf0 2020-11-20T12:16:31.762 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1bb7fe700 1 --2- 172.21.15.189:0/1858611661 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1bc13bcf0 0x7fd1bc13c130 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7fd1bc0656f0 tx=0x7fd1b4006df0).ready entity=mon.1 client_cookie=9616d9e7941ed8ea server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:31.762 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1b9ffb700 1 -- 172.21.15.189:0/1858611661 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac001860 con 0x7fd1bc13bcf0 2020-11-20T12:16:31.762 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1b9ffb700 1 -- 172.21.15.189:0/1858611661 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b4011400 con 0x7fd1bc13bcf0 2020-11-20T12:16:31.762 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1b9ffb700 1 -- 172.21.15.189:0/1858611661 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4011780 con 0x7fd1bc13bcf0 2020-11-20T12:16:31.762 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1c180d700 1 -- 172.21.15.189:0/1858611661 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1bc13bcf0 msgr2=0x7fd1bc13c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:31.763 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1c180d700 1 --2- 172.21.15.189:0/1858611661 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1bc13bcf0 0x7fd1bc13c130 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7fd1bc0656f0 tx=0x7fd1b4006df0).stop 2020-11-20T12:16:31.763 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1c180d700 1 -- 172.21.15.189:0/1858611661 shutdown_connections 2020-11-20T12:16:31.763 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1c180d700 1 --2- 172.21.15.189:0/1858611661 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1bc13bcf0 0x7fd1bc13c130 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:31.763 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1c180d700 1 --2- 172.21.15.189:0/1858611661 >> v2:172.21.15.47:3300/0 conn(0x7fd1bc0a7bf0 0x7fd1bc0a8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:31.763 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1c180d700 1 -- 172.21.15.189:0/1858611661 >> 172.21.15.189:0/1858611661 conn(0x7fd1bc0627f0 msgr2=0x7fd1bc065870 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:31.764 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1c180d700 1 -- 172.21.15.189:0/1858611661 shutdown_connections 2020-11-20T12:16:31.764 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.752+0000 7fd1c180d700 1 -- 172.21.15.189:0/1858611661 wait complete. 2020-11-20T12:16:31.766 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.756+0000 7fd1c180d700 1 Processor -- start 2020-11-20T12:16:31.766 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.756+0000 7fd1c180d700 1 -- start start 2020-11-20T12:16:31.766 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.756+0000 7fd1c180d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1bc0a7bf0 0x7fd1bc0a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:31.767 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.756+0000 7fd1c180d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1bc0a9e20 0x7fd1bc0a22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:31.767 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.756+0000 7fd1c180d700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd1bc13bcf0 0x7fd1bc14e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:31.767 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.756+0000 7fd1c180d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc0a7180 con 0x7fd1bc0a7bf0 2020-11-20T12:16:31.767 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.756+0000 7fd1c180d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc0a2cf0 con 0x7fd1bc0a9e20 2020-11-20T12:16:31.768 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.756+0000 7fd1c180d700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc0a2f90 con 0x7fd1bc13bcf0 2020-11-20T12:16:31.768 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.756+0000 7fd1ba7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1bc0a9e20 0x7fd1bc0a22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:31.768 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.760+0000 7fd1ba7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1bc0a9e20 0x7fd1bc0a22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51164/0 (socket says 172.21.15.189:51164) 2020-11-20T12:16:31.768 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.760+0000 7fd1ba7fc700 1 -- 172.21.15.189:0/932309053 learned_addr learned my addr 172.21.15.189:0/932309053 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:16:31.768 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.760+0000 7fd1baffd700 1 --2- 172.21.15.189:0/932309053 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1bc0a7bf0 0x7fd1bc0a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:31.769 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.760+0000 7fd1bb7fe700 1 --2- 172.21.15.189:0/932309053 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd1bc13bcf0 0x7fd1bc14e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:31.769 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.760+0000 7fd1ba7fc700 1 -- 172.21.15.189:0/932309053 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd1bc13bcf0 msgr2=0x7fd1bc14e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:31.769 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.760+0000 7fd1ba7fc700 1 --2- 172.21.15.189:0/932309053 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd1bc13bcf0 0x7fd1bc14e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:31.769 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.760+0000 7fd1ba7fc700 1 -- 172.21.15.189:0/932309053 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1bc0a7bf0 msgr2=0x7fd1bc0a15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:31.769 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.760+0000 7fd1ba7fc700 1 --2- 172.21.15.189:0/932309053 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1bc0a7bf0 0x7fd1bc0a15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:31.770 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.760+0000 7fd1ba7fc700 1 -- 172.21.15.189:0/932309053 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1bc14db80 con 0x7fd1bc0a9e20 2020-11-20T12:16:31.770 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.760+0000 7fd1ba7fc700 1 --2- 172.21.15.189:0/932309053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1bc0a9e20 0x7fd1bc0a22c0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7fd1a400eba0 tx=0x7fd1a4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:31.770 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.760+0000 7fd1b2ffd700 1 -- 172.21.15.189:0/932309053 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a400b280 con 0x7fd1bc0a9e20 2020-11-20T12:16:31.771 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.760+0000 7fd1b2ffd700 1 -- 172.21.15.189:0/932309053 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a400c8e0 con 0x7fd1bc0a9e20 2020-11-20T12:16:31.772 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.760+0000 7fd1c180d700 1 -- 172.21.15.189:0/932309053 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1bc0a27b0 con 0x7fd1bc0a9e20 2020-11-20T12:16:31.772 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.760+0000 7fd1b2ffd700 1 -- 172.21.15.189:0/932309053 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a400cc80 con 0x7fd1bc0a9e20 2020-11-20T12:16:31.772 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.760+0000 7fd1c180d700 1 -- 172.21.15.189:0/932309053 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1bc14e760 con 0x7fd1bc0a9e20 2020-11-20T12:16:31.772 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.760+0000 7fd1c180d700 1 -- 172.21.15.189:0/932309053 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1bc14e9e0 con 0x7fd1bc0a9e20 2020-11-20T12:16:31.773 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.760+0000 7fd1b2ffd700 1 -- 172.21.15.189:0/932309053 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fd1a4022070 con 0x7fd1bc0a9e20 2020-11-20T12:16:31.773 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.764+0000 7fd1b2ffd700 1 --2- 172.21.15.189:0/932309053 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd1980557c0 0x7fd198057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:31.773 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.764+0000 7fd1b2ffd700 1 -- 172.21.15.189:0/932309053 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6761+0+0 (secure 0 0 0) 0x7fd1a406ed40 con 0x7fd1bc0a9e20 2020-11-20T12:16:31.773 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.764+0000 7fd1baffd700 1 --2- 172.21.15.189:0/932309053 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd1980557c0 0x7fd198057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:31.773 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.764+0000 7fd1baffd700 1 --2- 172.21.15.189:0/932309053 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd1980557c0 0x7fd198057c80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fd1ac001520 tx=0x7fd1ac002110).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:31.774 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.768+0000 7fd1b2ffd700 1 -- 172.21.15.189:0/932309053 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd1a4044820 con 0x7fd1bc0a9e20 2020-11-20T12:16:31.977 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.972+0000 7fd1c180d700 1 -- 172.21.15.189:0/932309053 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1bc0a2b00 con 0x7fd1bc0a9e20 2020-11-20T12:16:31.978 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.972+0000 7fd1b2ffd700 1 -- 172.21.15.189:0/932309053 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+2134 (secure 0 0 0) 0x7fd1a4047e40 con 0x7fd1bc0a9e20 2020-11-20T12:16:31.978 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:16:31.978 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5075,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/563711894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":563711894},{"type":"v1","addr":"172.21.15.47:6833","nonce":563711894}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:15:25.740722+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5389":{"gid":5389,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":34,"addr":"172.21.15.189:6835/636533719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":636533719},{"type":"v1","addr":"172.21.15.189:6835","nonce":636533719}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:16:31.980 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.976+0000 7fd1c180d700 1 -- 172.21.15.189:0/932309053 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd1980557c0 msgr2=0x7fd198057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:31.980 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.976+0000 7fd1c180d700 1 --2- 172.21.15.189:0/932309053 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd1980557c0 0x7fd198057c80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fd1ac001520 tx=0x7fd1ac002110).stop 2020-11-20T12:16:31.980 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.976+0000 7fd1c180d700 1 -- 172.21.15.189:0/932309053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1bc0a9e20 msgr2=0x7fd1bc0a22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:31.980 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.976+0000 7fd1c180d700 1 --2- 172.21.15.189:0/932309053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1bc0a9e20 0x7fd1bc0a22c0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7fd1a400eba0 tx=0x7fd1a4008040).stop 2020-11-20T12:16:31.981 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.976+0000 7fd1c180d700 1 -- 172.21.15.189:0/932309053 shutdown_connections 2020-11-20T12:16:31.981 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.976+0000 7fd1c180d700 1 --2- 172.21.15.189:0/932309053 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd1980557c0 0x7fd198057c80 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:31.981 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.976+0000 7fd1c180d700 1 --2- 172.21.15.189:0/932309053 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd1bc0a7bf0 0x7fd1bc0a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:31.981 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.976+0000 7fd1c180d700 1 --2- 172.21.15.189:0/932309053 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd1bc13bcf0 0x7fd1bc14e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:31.981 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.976+0000 7fd1c180d700 1 --2- 172.21.15.189:0/932309053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1bc0a9e20 0x7fd1bc0a22c0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:31.982 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.976+0000 7fd1c180d700 1 -- 172.21.15.189:0/932309053 >> 172.21.15.189:0/932309053 conn(0x7fd1bc0627f0 msgr2=0x7fd1bc067170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:31.982 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.976+0000 7fd1c180d700 1 -- 172.21.15.189:0/932309053 shutdown_connections 2020-11-20T12:16:31.982 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:31.976+0000 7fd1c180d700 1 -- 172.21.15.189:0/932309053 wait complete. 2020-11-20T12:16:31.983 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 46 2020-11-20T12:16:31.993 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:16:33.994 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:16:34.125 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.120+0000 7f1f76ab8700 1 Processor -- start 2020-11-20T12:16:34.126 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.120+0000 7f1f76ab8700 1 -- start start 2020-11-20T12:16:34.127 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.120+0000 7f1f76ab8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1f70064e80 0x7f1f700aadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:34.127 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.120+0000 7f1f76ab8700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1f700aeb80 0x7f1f700a9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:34.127 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.120+0000 7f1f76ab8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f700a9b80 con 0x7f1f70064830 2020-11-20T12:16:34.128 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.120+0000 7f1f76ab8700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f700ab2c0 con 0x7f1f70064e80 2020-11-20T12:16:34.128 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.120+0000 7f1f76ab8700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f700af090 con 0x7f1f700aeb80 2020-11-20T12:16:34.128 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.120+0000 7f1f75ab6700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1f70064e80 0x7f1f700aadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:34.128 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.120+0000 7f1f762b7700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1f70064830 0x7f1f700ab670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52800/0 (socket says 172.21.15.189:52800) 2020-11-20T12:16:34.128 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.120+0000 7f1f762b7700 1 -- 172.21.15.189:0/620828451 learned_addr learned my addr 172.21.15.189:0/620828451 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:16:34.129 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.120+0000 7f1f752b5700 1 --2- 172.21.15.189:0/620828451 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1f700aeb80 0x7f1f700a9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:34.129 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.124+0000 7f1f75ab6700 1 -- 172.21.15.189:0/620828451 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1f700aeb80 msgr2=0x7f1f700a9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:34.129 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.124+0000 7f1f75ab6700 1 --2- 172.21.15.189:0/620828451 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1f700aeb80 0x7f1f700a9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:34.130 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.124+0000 7f1f75ab6700 1 -- 172.21.15.189:0/620828451 >> v1:172.21.15.189:6789/0 conn(0x7f1f70064830 legacy=0x7f1f700ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:34.131 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.124+0000 7f1f75ab6700 1 -- 172.21.15.189:0/620828451 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f700af330 con 0x7f1f70064e80 2020-11-20T12:16:34.131 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.124+0000 7f1f75ab6700 1 --2- 172.21.15.189:0/620828451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f70064e80 0x7f1f700aadd0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f1f6000b9a0 tx=0x7f1f60006df0).ready entity=mon.1 client_cookie=db5214db6669cb68 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:34.131 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.124+0000 7f1f74ab4700 1 -- 172.21.15.189:0/620828451 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f60009840 con 0x7f1f70064e80 2020-11-20T12:16:34.131 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.124+0000 7f1f74ab4700 1 -- 172.21.15.189:0/620828451 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f600099a0 con 0x7f1f70064e80 2020-11-20T12:16:34.131 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.124+0000 7f1f74ab4700 1 -- 172.21.15.189:0/620828451 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f60009d20 con 0x7f1f70064e80 2020-11-20T12:16:34.132 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.124+0000 7f1f76ab8700 1 -- 172.21.15.189:0/620828451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f70064e80 msgr2=0x7f1f700aadd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:34.132 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.124+0000 7f1f76ab8700 1 --2- 172.21.15.189:0/620828451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f70064e80 0x7f1f700aadd0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7f1f6000b9a0 tx=0x7f1f60006df0).stop 2020-11-20T12:16:34.132 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.124+0000 7f1f76ab8700 1 -- 172.21.15.189:0/620828451 shutdown_connections 2020-11-20T12:16:34.132 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.124+0000 7f1f76ab8700 1 --2- 172.21.15.189:0/620828451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f70064e80 0x7f1f700aadd0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:34.132 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.124+0000 7f1f76ab8700 1 --2- 172.21.15.189:0/620828451 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1f700aeb80 0x7f1f700a9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:34.133 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.124+0000 7f1f76ab8700 1 -- 172.21.15.189:0/620828451 >> 172.21.15.189:0/620828451 conn(0x7f1f70062080 msgr2=0x7f1f70062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:34.133 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.124+0000 7f1f76ab8700 1 -- 172.21.15.189:0/620828451 shutdown_connections 2020-11-20T12:16:34.133 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.124+0000 7f1f76ab8700 1 -- 172.21.15.189:0/620828451 wait complete. 2020-11-20T12:16:34.135 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.128+0000 7f1f76ab8700 1 Processor -- start 2020-11-20T12:16:34.136 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.128+0000 7f1f76ab8700 1 -- start start 2020-11-20T12:16:34.136 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.128+0000 7f1f76ab8700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1f70064830 0x7f1f70143080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:34.136 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.128+0000 7f1f76ab8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f70064e80 0x7f1f700ad190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:34.136 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.128+0000 7f1f76ab8700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f700aeb80 0x7f1f70151910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:34.136 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.128+0000 7f1f76ab8700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7014dd00 con 0x7f1f700aeb80 2020-11-20T12:16:34.137 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.128+0000 7f1f76ab8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f701415e0 con 0x7f1f70064e80 2020-11-20T12:16:34.137 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.128+0000 7f1f76ab8700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f70151470 con 0x7f1f70064830 2020-11-20T12:16:34.137 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.128+0000 7f1f752b5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f70064e80 0x7f1f700ad190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:34.137 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.128+0000 7f1f75ab6700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1f70064830 0x7f1f70143080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:34.137 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.128+0000 7f1f752b5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f70064e80 0x7f1f700ad190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51182/0 (socket says 172.21.15.189:51182) 2020-11-20T12:16:34.138 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.128+0000 7f1f752b5700 1 -- 172.21.15.189:0/407259929 learned_addr learned my addr 172.21.15.189:0/407259929 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:16:34.138 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.128+0000 7f1f762b7700 1 --2- 172.21.15.189:0/407259929 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f700aeb80 0x7f1f70151910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:34.138 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.128+0000 7f1f752b5700 1 -- 172.21.15.189:0/407259929 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1f70064830 msgr2=0x7f1f70143080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:34.138 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.128+0000 7f1f752b5700 1 --2- 172.21.15.189:0/407259929 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1f70064830 0x7f1f70143080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:34.138 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.128+0000 7f1f752b5700 1 -- 172.21.15.189:0/407259929 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f700aeb80 msgr2=0x7f1f70151910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:34.139 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.128+0000 7f1f752b5700 1 --2- 172.21.15.189:0/407259929 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f700aeb80 0x7f1f70151910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:34.139 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.128+0000 7f1f752b5700 1 -- 172.21.15.189:0/407259929 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f70151e00 con 0x7f1f70064e80 2020-11-20T12:16:34.139 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.128+0000 7f1f762b7700 1 --2- 172.21.15.189:0/407259929 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f700aeb80 0x7f1f70151910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:16:34.139 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.128+0000 7f1f752b5700 1 --2- 172.21.15.189:0/407259929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f70064e80 0x7f1f700ad190 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f1f6c00ef50 tx=0x7f1f6c0091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:34.139 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.128+0000 7f1f5dffb700 1 -- 172.21.15.189:0/407259929 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c00ccc0 con 0x7f1f70064e80 2020-11-20T12:16:34.140 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.128+0000 7f1f5dffb700 1 -- 172.21.15.189:0/407259929 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f6c00ce20 con 0x7f1f70064e80 2020-11-20T12:16:34.140 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.128+0000 7f1f5dffb700 1 -- 172.21.15.189:0/407259929 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c012620 con 0x7f1f70064e80 2020-11-20T12:16:34.140 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.132+0000 7f1f76ab8700 1 -- 172.21.15.189:0/407259929 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f701520b0 con 0x7f1f70064e80 2020-11-20T12:16:34.140 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.132+0000 7f1f76ab8700 1 -- 172.21.15.189:0/407259929 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f7014e300 con 0x7f1f70064e80 2020-11-20T12:16:34.141 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.132+0000 7f1f76ab8700 1 -- 172.21.15.189:0/407259929 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f701512b0 con 0x7f1f70064e80 2020-11-20T12:16:34.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.132+0000 7f1f5dffb700 1 -- 172.21.15.189:0/407259929 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f1f6c022070 con 0x7f1f70064e80 2020-11-20T12:16:34.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.132+0000 7f1f5dffb700 1 --2- 172.21.15.189:0/407259929 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1f480557c0 0x7f1f48057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:34.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.132+0000 7f1f5dffb700 1 -- 172.21.15.189:0/407259929 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6847+0+0 (secure 0 0 0) 0x7f1f6c06f110 con 0x7f1f70064e80 2020-11-20T12:16:34.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.132+0000 7f1f75ab6700 1 --2- 172.21.15.189:0/407259929 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1f480557c0 0x7f1f48057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:34.143 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.132+0000 7f1f75ab6700 1 --2- 172.21.15.189:0/407259929 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1f480557c0 0x7f1f48057c80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f1f60006e30 tx=0x7f1f60015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:34.143 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.136+0000 7f1f5dffb700 1 -- 172.21.15.189:0/407259929 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1f6c044cb0 con 0x7f1f70064e80 2020-11-20T12:16:34.344 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.340+0000 7f1f76ab8700 1 -- 172.21.15.189:0/407259929 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f700446d0 con 0x7f1f70064e80 2020-11-20T12:16:34.345 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.340+0000 7f1f5dffb700 1 -- 172.21.15.189:0/407259929 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+1785 (secure 0 0 0) 0x7f1f6c016090 con 0x7f1f70064e80 2020-11-20T12:16:34.345 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:16:34.345 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:16:33.726023+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5075":{"gid":5075,"name":"b","rank":0,"incarnation":47,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.47:6833/563711894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":563711894},{"type":"v1","addr":"172.21.15.47:6833","nonce":563711894}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:16:34.348 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.340+0000 7f1f76ab8700 1 -- 172.21.15.189:0/407259929 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1f480557c0 msgr2=0x7f1f48057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:34.348 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.340+0000 7f1f76ab8700 1 --2- 172.21.15.189:0/407259929 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1f480557c0 0x7f1f48057c80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f1f60006e30 tx=0x7f1f60015000).stop 2020-11-20T12:16:34.348 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.340+0000 7f1f76ab8700 1 -- 172.21.15.189:0/407259929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f70064e80 msgr2=0x7f1f700ad190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:34.348 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.340+0000 7f1f76ab8700 1 --2- 172.21.15.189:0/407259929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f70064e80 0x7f1f700ad190 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f1f6c00ef50 tx=0x7f1f6c0091d0).stop 2020-11-20T12:16:34.349 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.340+0000 7f1f76ab8700 1 -- 172.21.15.189:0/407259929 shutdown_connections 2020-11-20T12:16:34.349 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.340+0000 7f1f76ab8700 1 --2- 172.21.15.189:0/407259929 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f1f480557c0 0x7f1f48057c80 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:34.349 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.340+0000 7f1f76ab8700 1 --2- 172.21.15.189:0/407259929 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f1f700aeb80 0x7f1f70151910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:34.350 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.340+0000 7f1f76ab8700 1 --2- 172.21.15.189:0/407259929 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f1f70064830 0x7f1f70143080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:34.350 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.340+0000 7f1f76ab8700 1 --2- 172.21.15.189:0/407259929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f70064e80 0x7f1f700ad190 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:34.350 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.340+0000 7f1f76ab8700 1 -- 172.21.15.189:0/407259929 >> 172.21.15.189:0/407259929 conn(0x7f1f70062080 msgr2=0x7f1f7013ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:34.351 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.340+0000 7f1f76ab8700 1 -- 172.21.15.189:0/407259929 shutdown_connections 2020-11-20T12:16:34.351 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.340+0000 7f1f76ab8700 1 -- 172.21.15.189:0/407259929 wait complete. 2020-11-20T12:16:34.352 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 48 2020-11-20T12:16:34.360 INFO:tasks.mds_thrash.fs.[None]:mds.a down, removed from mdsmap 2020-11-20T12:16:34.361 INFO:tasks.mds_thrash.fs.[None]:waiting for mds cluster to stabilize... 2020-11-20T12:16:34.361 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:16:34.488 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.480+0000 7f7418c56700 1 Processor -- start 2020-11-20T12:16:34.488 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.480+0000 7f7418c56700 1 -- start start 2020-11-20T12:16:34.488 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7418c56700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f74140a8840 0x7f741413dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:34.489 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7418c56700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f741413e100 0x7f74141371b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:34.489 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7418c56700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f74141376a0 0x7f7414137b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:34.489 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7418c56700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7414138040 con 0x7f74140a8840 2020-11-20T12:16:34.489 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7418c56700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7414066800 con 0x7f74141376a0 2020-11-20T12:16:34.489 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7418c56700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7414066aa0 con 0x7f741413e100 2020-11-20T12:16:34.490 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7413fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f74141376a0 0x7f7414137b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:34.490 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7413fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f74141376a0 0x7f7414137b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51190/0 (socket says 172.21.15.189:51190) 2020-11-20T12:16:34.490 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7413fff700 1 -- 172.21.15.189:0/2285087924 learned_addr learned my addr 172.21.15.189:0/2285087924 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:16:34.490 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f74137fe700 1 --2- 172.21.15.189:0/2285087924 >> v2:172.21.15.47:3300/0 conn(0x7f74140a8840 0x7f741413dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:34.490 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7412ffd700 1 --2- 172.21.15.189:0/2285087924 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f741413e100 0x7f74141371b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:34.491 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7413fff700 1 -- 172.21.15.189:0/2285087924 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f741413e100 msgr2=0x7f74141371b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:34.491 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7413fff700 1 --2- 172.21.15.189:0/2285087924 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f741413e100 0x7f74141371b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:34.491 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7413fff700 1 -- 172.21.15.189:0/2285087924 >> v2:172.21.15.47:3300/0 conn(0x7f74140a8840 msgr2=0x7f741413dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:34.491 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7413fff700 1 --2- 172.21.15.189:0/2285087924 >> v2:172.21.15.47:3300/0 conn(0x7f74140a8840 0x7f741413dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:34.492 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7413fff700 1 -- 172.21.15.189:0/2285087924 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74140672f0 con 0x7f74141376a0 2020-11-20T12:16:34.492 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7413fff700 1 --2- 172.21.15.189:0/2285087924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74141376a0 0x7f7414137b50 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f7414065ae0 tx=0x7f740c009290).ready entity=mon.1 client_cookie=80fab8a5952500 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:34.495 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f74127fc700 1 -- 172.21.15.189:0/2285087924 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740c00bbc0 con 0x7f74141376a0 2020-11-20T12:16:34.496 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f74127fc700 1 -- 172.21.15.189:0/2285087924 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f740c00bd20 con 0x7f74141376a0 2020-11-20T12:16:34.496 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f74127fc700 1 -- 172.21.15.189:0/2285087924 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740c011610 con 0x7f74141376a0 2020-11-20T12:16:34.496 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7418c56700 1 -- 172.21.15.189:0/2285087924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74141376a0 msgr2=0x7f7414137b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:34.496 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7418c56700 1 --2- 172.21.15.189:0/2285087924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74141376a0 0x7f7414137b50 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f7414065ae0 tx=0x7f740c009290).stop 2020-11-20T12:16:34.497 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7418c56700 1 -- 172.21.15.189:0/2285087924 shutdown_connections 2020-11-20T12:16:34.497 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7418c56700 1 --2- 172.21.15.189:0/2285087924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74141376a0 0x7f7414137b50 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:34.497 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7418c56700 1 --2- 172.21.15.189:0/2285087924 >> v2:172.21.15.47:3300/0 conn(0x7f74140a8840 0x7f741413dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:34.497 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7418c56700 1 --2- 172.21.15.189:0/2285087924 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f741413e100 0x7f74141371b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:34.497 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7418c56700 1 -- 172.21.15.189:0/2285087924 >> 172.21.15.189:0/2285087924 conn(0x7f7414060170 msgr2=0x7f7414060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:34.498 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7418c56700 1 -- 172.21.15.189:0/2285087924 shutdown_connections 2020-11-20T12:16:34.498 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7418c56700 1 -- 172.21.15.189:0/2285087924 wait complete. 2020-11-20T12:16:34.498 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7418c56700 1 Processor -- start 2020-11-20T12:16:34.498 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.484+0000 7f7418c56700 1 -- start start 2020-11-20T12:16:34.498 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.488+0000 7f7418c56700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74140a8840 0x7f741413cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:34.499 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.488+0000 7f7418c56700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74141376a0 0x7f741413a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:34.499 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.488+0000 7f7418c56700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f741413e100 0x7f741413ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:34.499 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.488+0000 7f7418c56700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741413b5c0 con 0x7f74140a8840 2020-11-20T12:16:34.499 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.488+0000 7f7418c56700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7414139c30 con 0x7f74141376a0 2020-11-20T12:16:34.499 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.488+0000 7f7418c56700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7414139ed0 con 0x7f741413e100 2020-11-20T12:16:34.500 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.488+0000 7f7412ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74141376a0 0x7f741413a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:34.500 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.488+0000 7f7412ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74141376a0 0x7f741413a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51194/0 (socket says 172.21.15.189:51194) 2020-11-20T12:16:34.500 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.488+0000 7f7412ffd700 1 -- 172.21.15.189:0/2175738140 learned_addr learned my addr 172.21.15.189:0/2175738140 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:16:34.501 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.488+0000 7f74137fe700 1 --2- 172.21.15.189:0/2175738140 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74140a8840 0x7f741413cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:34.501 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.488+0000 7f7413fff700 1 --2- 172.21.15.189:0/2175738140 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f741413e100 0x7f741413ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:34.501 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.488+0000 7f7412ffd700 1 -- 172.21.15.189:0/2175738140 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f741413e100 msgr2=0x7f741413ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:34.501 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.488+0000 7f7412ffd700 1 --2- 172.21.15.189:0/2175738140 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f741413e100 0x7f741413ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:34.502 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.488+0000 7f7412ffd700 1 -- 172.21.15.189:0/2175738140 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74140a8840 msgr2=0x7f741413cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:34.502 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.488+0000 7f7412ffd700 1 --2- 172.21.15.189:0/2175738140 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74140a8840 0x7f741413cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:34.502 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.488+0000 7f7412ffd700 1 -- 172.21.15.189:0/2175738140 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f741413a860 con 0x7f74141376a0 2020-11-20T12:16:34.502 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.488+0000 7f7412ffd700 1 --2- 172.21.15.189:0/2175738140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74141376a0 0x7f741413a370 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f740800d8e0 tx=0x7f7408008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:34.503 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.488+0000 7f73fb7fe700 1 -- 172.21.15.189:0/2175738140 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74080125f0 con 0x7f74141376a0 2020-11-20T12:16:34.503 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.488+0000 7f73fb7fe700 1 -- 172.21.15.189:0/2175738140 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7408012750 con 0x7f74141376a0 2020-11-20T12:16:34.503 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.488+0000 7f7418c56700 1 -- 172.21.15.189:0/2175738140 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f741413b210 con 0x7f74141376a0 2020-11-20T12:16:34.503 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.488+0000 7f73fb7fe700 1 -- 172.21.15.189:0/2175738140 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7408012af0 con 0x7f74141376a0 2020-11-20T12:16:34.503 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.488+0000 7f7418c56700 1 -- 172.21.15.189:0/2175738140 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7414149c40 con 0x7f74141376a0 2020-11-20T12:16:34.504 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.488+0000 7f7418c56700 1 -- 172.21.15.189:0/2175738140 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7414137440 con 0x7f74141376a0 2020-11-20T12:16:34.504 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.492+0000 7f73fb7fe700 1 -- 172.21.15.189:0/2175738140 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f7408013480 con 0x7f74141376a0 2020-11-20T12:16:34.504 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.492+0000 7f73fb7fe700 1 --2- 172.21.15.189:0/2175738140 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f73f00557c0 0x7f73f0057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:34.505 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.492+0000 7f73fb7fe700 1 -- 172.21.15.189:0/2175738140 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6847+0+0 (secure 0 0 0) 0x7f740806e830 con 0x7f74141376a0 2020-11-20T12:16:34.505 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.492+0000 7f74137fe700 1 --2- 172.21.15.189:0/2175738140 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f73f00557c0 0x7f73f0057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:34.505 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.492+0000 7f74137fe700 1 --2- 172.21.15.189:0/2175738140 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f73f00557c0 0x7f73f0057c80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f740400ba00 tx=0x7f7404008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:34.506 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.500+0000 7f73fb7fe700 1 -- 172.21.15.189:0/2175738140 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7408044370 con 0x7f74141376a0 2020-11-20T12:16:34.707 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.704+0000 7f7418c56700 1 -- 172.21.15.189:0/2175738140 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f741413a090 con 0x7f74141376a0 2020-11-20T12:16:34.708 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.704+0000 7f73fb7fe700 1 -- 172.21.15.189:0/2175738140 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+1785 (secure 0 0 0) 0x7f7408012e60 con 0x7f74141376a0 2020-11-20T12:16:34.709 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:16:34.709 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:16:33.726023+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5075":{"gid":5075,"name":"b","rank":0,"incarnation":47,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.47:6833/563711894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":563711894},{"type":"v1","addr":"172.21.15.47:6833","nonce":563711894}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:16:34.714 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.708+0000 7f7418c56700 1 -- 172.21.15.189:0/2175738140 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f73f00557c0 msgr2=0x7f73f0057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:34.715 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.708+0000 7f7418c56700 1 --2- 172.21.15.189:0/2175738140 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f73f00557c0 0x7f73f0057c80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f740400ba00 tx=0x7f7404008040).stop 2020-11-20T12:16:34.715 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.708+0000 7f7418c56700 1 -- 172.21.15.189:0/2175738140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74141376a0 msgr2=0x7f741413a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:34.715 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.708+0000 7f7418c56700 1 --2- 172.21.15.189:0/2175738140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74141376a0 0x7f741413a370 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f740800d8e0 tx=0x7f7408008040).stop 2020-11-20T12:16:34.715 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.708+0000 7f7418c56700 1 -- 172.21.15.189:0/2175738140 shutdown_connections 2020-11-20T12:16:34.715 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.708+0000 7f7418c56700 1 --2- 172.21.15.189:0/2175738140 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f73f00557c0 0x7f73f0057c80 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:34.716 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.708+0000 7f7418c56700 1 --2- 172.21.15.189:0/2175738140 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f74140a8840 0x7f741413cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:34.716 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.708+0000 7f7418c56700 1 --2- 172.21.15.189:0/2175738140 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f741413e100 0x7f741413ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:34.716 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.708+0000 7f7418c56700 1 --2- 172.21.15.189:0/2175738140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74141376a0 0x7f741413a370 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:34.716 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.708+0000 7f7418c56700 1 -- 172.21.15.189:0/2175738140 >> 172.21.15.189:0/2175738140 conn(0x7f7414060170 msgr2=0x7f7414136930 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:34.716 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.708+0000 7f7418c56700 1 -- 172.21.15.189:0/2175738140 shutdown_connections 2020-11-20T12:16:34.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:34.708+0000 7f7418c56700 1 -- 172.21.15.189:0/2175738140 wait complete. 2020-11-20T12:16:34.717 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 48 2020-11-20T12:16:34.728 INFO:tasks.mds_thrash.fs.[None]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 48, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2020-11-20T12:04:54.012110+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 48, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5075": { "addr": "172.21.15.47:6833/563711894", "addrs": { "addrvec": [ { "addr": "172.21.15.47:6832", "nonce": 563711894, "type": "v2" }, { "addr": "172.21.15.47:6833", "nonce": 563711894, "type": "v1" } ] }, "export_targets": [], "features": 4540138297136906239, "flags": 0, "gid": 5075, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2020-11-20T12:16:33.726023+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5075 } } } ], "standbys": [] } 2020-11-20T12:16:36.730 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:16:36.864 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259759e700 1 Processor -- start 2020-11-20T12:16:36.865 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259759e700 1 -- start start 2020-11-20T12:16:36.870 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259759e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2598067b50 0x7f2598136fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:36.870 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259759e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2598068020 con 0x7f25981374a0 2020-11-20T12:16:36.870 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259759e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25980682c0 con 0x7f25980612b0 2020-11-20T12:16:36.871 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259759e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2598061690 con 0x7f2598067b50 2020-11-20T12:16:36.871 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259659c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f25980612b0 0x7f25980a0960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52824/0 (socket says 172.21.15.189:52824) 2020-11-20T12:16:36.871 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259659c700 1 -- 172.21.15.189:0/3937453619 learned_addr learned my addr 172.21.15.189:0/3937453619 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:16:36.871 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f2595d9b700 1 --2- 172.21.15.189:0/3937453619 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2598067b50 0x7f2598136fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:36.872 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259559a700 1 -- 172.21.15.189:0/3937453619 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750855048 0 0) 0x7f25980682c0 con 0x7f25980612b0 2020-11-20T12:16:36.872 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259559a700 1 -- 172.21.15.189:0/3937453619 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f257c003240 con 0x7f25980612b0 2020-11-20T12:16:36.872 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259559a700 1 -- 172.21.15.189:0/3937453619 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 437496797 0 0) 0x7f257c003240 con 0x7f25980612b0 2020-11-20T12:16:36.872 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259559a700 1 -- 172.21.15.189:0/3937453619 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2598067b50 msgr2=0x7f2598136fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:36.873 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259559a700 1 --2- 172.21.15.189:0/3937453619 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2598067b50 0x7f2598136fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:36.873 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259559a700 1 -- 172.21.15.189:0/3937453619 >> v1:172.21.15.47:6789/0 conn(0x7f25981374a0 legacy=0x7f2598133a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:36.873 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259559a700 1 -- 172.21.15.189:0/3937453619 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25981378c0 con 0x7f25980612b0 2020-11-20T12:16:36.874 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259559a700 1 -- 172.21.15.189:0/3937453619 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f25880014e0 con 0x7f25980612b0 2020-11-20T12:16:36.874 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259559a700 1 -- 172.21.15.189:0/3937453619 >> v1:172.21.15.189:6789/0 conn(0x7f25980612b0 legacy=0x7f25980a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:36.874 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259559a700 1 --2- 172.21.15.189:0/3937453619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c003730 0x7f257c003b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:36.875 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259559a700 1 --2- 172.21.15.189:0/3937453619 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2598067b50 0x7f257c0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:36.875 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259559a700 1 --2- 172.21.15.189:0/3937453619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f257c0049d0 0x7f257c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:36.875 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259559a700 1 -- 172.21.15.189:0/3937453619 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2598061690 con 0x7f257c0049d0 2020-11-20T12:16:36.876 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259559a700 1 -- 172.21.15.189:0/3937453619 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257c004140 con 0x7f257c003730 2020-11-20T12:16:36.876 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259559a700 1 -- 172.21.15.189:0/3937453619 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f257c004280 con 0x7f2598067b50 2020-11-20T12:16:36.876 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259659c700 1 --2- 172.21.15.189:0/3937453619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c003730 0x7f257c003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:36.878 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f2596d9d700 1 --2- 172.21.15.189:0/3937453619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f257c0049d0 0x7f257c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:36.879 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f2595d9b700 1 --2- 172.21.15.189:0/3937453619 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2598067b50 0x7f257c0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:36.879 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259659c700 1 -- 172.21.15.189:0/3937453619 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2598067b50 msgr2=0x7f257c0044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:36.879 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259659c700 1 --2- 172.21.15.189:0/3937453619 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2598067b50 0x7f257c0044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:36.879 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259659c700 1 -- 172.21.15.189:0/3937453619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f257c0049d0 msgr2=0x7f257c004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:36.879 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259659c700 1 --2- 172.21.15.189:0/3937453619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f257c0049d0 0x7f257c004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:36.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.860+0000 7f259659c700 1 -- 172.21.15.189:0/3937453619 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f257c003240 con 0x7f257c003730 2020-11-20T12:16:36.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.864+0000 7f259659c700 1 --2- 172.21.15.189:0/3937453619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c003730 0x7f257c003b10 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f258800c440 tx=0x7f2588007750).ready entity=mon.1 client_cookie=9d596a07bd1d2b42 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:36.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.864+0000 7f259559a700 1 -- 172.21.15.189:0/3937453619 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25880070d0 con 0x7f257c003730 2020-11-20T12:16:36.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.864+0000 7f259559a700 1 -- 172.21.15.189:0/3937453619 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2588007230 con 0x7f257c003730 2020-11-20T12:16:36.881 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.864+0000 7f259559a700 1 -- 172.21.15.189:0/3937453619 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25880055f0 con 0x7f257c003730 2020-11-20T12:16:36.881 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.864+0000 7f259759e700 1 -- 172.21.15.189:0/3937453619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c003730 msgr2=0x7f257c003b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:36.882 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.864+0000 7f259759e700 1 --2- 172.21.15.189:0/3937453619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c003730 0x7f257c003b10 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f258800c440 tx=0x7f2588007750).stop 2020-11-20T12:16:36.882 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.864+0000 7f2596d9d700 1 -- 172.21.15.189:0/3937453619 reap_dead start 2020-11-20T12:16:36.882 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.864+0000 7f259759e700 1 -- 172.21.15.189:0/3937453619 shutdown_connections 2020-11-20T12:16:36.882 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.864+0000 7f259759e700 1 -- 172.21.15.189:0/3937453619 >> 172.21.15.189:0/3937453619 conn(0x7f2598063210 msgr2=0x7f2598060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:36.883 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.864+0000 7f259759e700 1 -- 172.21.15.189:0/3937453619 shutdown_connections 2020-11-20T12:16:36.883 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.864+0000 7f259759e700 1 -- 172.21.15.189:0/3937453619 wait complete. 2020-11-20T12:16:36.883 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.864+0000 7f259759e700 1 Processor -- start 2020-11-20T12:16:36.883 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.864+0000 7f259759e700 1 -- start start 2020-11-20T12:16:36.886 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.864+0000 7f259759e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c003730 0x7f2598137ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:36.886 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.864+0000 7f259759e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25980a1210 0x7f2598138fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:36.886 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.864+0000 7f259759e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2598139490 0x7f2598148f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:36.887 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.864+0000 7f259759e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25980a1690 con 0x7f25980a1210 2020-11-20T12:16:36.887 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.864+0000 7f259759e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25980a4120 con 0x7f257c003730 2020-11-20T12:16:36.887 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.864+0000 7f259759e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25981384d0 con 0x7f2598139490 2020-11-20T12:16:36.887 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.864+0000 7f259659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c003730 0x7f2598137ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:36.888 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.864+0000 7f259659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c003730 0x7f2598137ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51214/0 (socket says 172.21.15.189:51214) 2020-11-20T12:16:36.888 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.864+0000 7f259659c700 1 -- 172.21.15.189:0/933889500 learned_addr learned my addr 172.21.15.189:0/933889500 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:16:36.888 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.868+0000 7f2595d9b700 1 --2- 172.21.15.189:0/933889500 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25980a1210 0x7f2598138fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:36.888 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.868+0000 7f2596d9d700 1 --2- 172.21.15.189:0/933889500 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2598139490 0x7f2598148f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:36.888 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.868+0000 7f259659c700 1 -- 172.21.15.189:0/933889500 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2598139490 msgr2=0x7f2598148f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:36.889 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.868+0000 7f259659c700 1 --2- 172.21.15.189:0/933889500 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2598139490 0x7f2598148f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:36.889 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.868+0000 7f259659c700 1 -- 172.21.15.189:0/933889500 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25980a1210 msgr2=0x7f2598138fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:36.889 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.868+0000 7f259659c700 1 --2- 172.21.15.189:0/933889500 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25980a1210 0x7f2598138fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:36.889 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.868+0000 7f259659c700 1 -- 172.21.15.189:0/933889500 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2598138770 con 0x7f257c003730 2020-11-20T12:16:36.889 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.868+0000 7f2595d9b700 1 --2- 172.21.15.189:0/933889500 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25980a1210 0x7f2598138fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:16:36.890 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.868+0000 7f259659c700 1 --2- 172.21.15.189:0/933889500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c003730 0x7f2598137ea0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7f2588005e40 tx=0x7f2588015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:36.890 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.868+0000 7f25867fc700 1 -- 172.21.15.189:0/933889500 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25880073a0 con 0x7f257c003730 2020-11-20T12:16:36.890 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.868+0000 7f25867fc700 1 -- 172.21.15.189:0/933889500 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2588000e90 con 0x7f257c003730 2020-11-20T12:16:36.890 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.868+0000 7f25867fc700 1 -- 172.21.15.189:0/933889500 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2588021690 con 0x7f257c003730 2020-11-20T12:16:36.890 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.868+0000 7f259759e700 1 -- 172.21.15.189:0/933889500 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25981398f0 con 0x7f257c003730 2020-11-20T12:16:36.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.868+0000 7f259759e700 1 -- 172.21.15.189:0/933889500 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25980a6cc0 con 0x7f257c003730 2020-11-20T12:16:36.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.868+0000 7f259759e700 1 -- 172.21.15.189:0/933889500 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25981312a0 con 0x7f257c003730 2020-11-20T12:16:36.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.868+0000 7f25867fc700 1 -- 172.21.15.189:0/933889500 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f2588010070 con 0x7f257c003730 2020-11-20T12:16:36.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.868+0000 7f25867fc700 1 --2- 172.21.15.189:0/933889500 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f256c0557c0 0x7f256c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:36.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.868+0000 7f25867fc700 1 -- 172.21.15.189:0/933889500 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6847+0+0 (secure 0 0 0) 0x7f258801c040 con 0x7f257c003730 2020-11-20T12:16:36.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.868+0000 7f2595d9b700 1 --2- 172.21.15.189:0/933889500 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f256c0557c0 0x7f256c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:36.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.868+0000 7f2595d9b700 1 --2- 172.21.15.189:0/933889500 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f256c0557c0 0x7f256c057c80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f258c006a90 tx=0x7f258c008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:36.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:36.876+0000 7f25867fc700 1 -- 172.21.15.189:0/933889500 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2588045970 con 0x7f257c003730 2020-11-20T12:16:37.122 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:37.116+0000 7f259759e700 1 -- 172.21.15.189:0/933889500 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25980446d0 con 0x7f257c003730 2020-11-20T12:16:37.123 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:37.116+0000 7f25867fc700 1 -- 172.21.15.189:0/933889500 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+1782 (secure 0 0 0) 0x7f2588026030 con 0x7f257c003730 2020-11-20T12:16:37.123 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:16:37.123 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:16:35.736803+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5075":{"gid":5075,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.47:6833/563711894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":563711894},{"type":"v1","addr":"172.21.15.47:6833","nonce":563711894}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:16:37.125 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:37.120+0000 7f259759e700 1 -- 172.21.15.189:0/933889500 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f256c0557c0 msgr2=0x7f256c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:37.125 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:37.120+0000 7f259759e700 1 --2- 172.21.15.189:0/933889500 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f256c0557c0 0x7f256c057c80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f258c006a90 tx=0x7f258c008040).stop 2020-11-20T12:16:37.126 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:37.120+0000 7f259759e700 1 -- 172.21.15.189:0/933889500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c003730 msgr2=0x7f2598137ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:37.126 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:37.120+0000 7f259759e700 1 --2- 172.21.15.189:0/933889500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c003730 0x7f2598137ea0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7f2588005e40 tx=0x7f2588015000).stop 2020-11-20T12:16:37.126 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:37.120+0000 7f259759e700 1 -- 172.21.15.189:0/933889500 shutdown_connections 2020-11-20T12:16:37.126 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:37.120+0000 7f259759e700 1 --2- 172.21.15.189:0/933889500 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2598139490 0x7f2598148f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:37.126 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:37.120+0000 7f259759e700 1 --2- 172.21.15.189:0/933889500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f257c003730 0x7f2598137ea0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:37.127 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:37.120+0000 7f259759e700 1 --2- 172.21.15.189:0/933889500 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f256c0557c0 0x7f256c057c80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:37.127 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:37.120+0000 7f259759e700 1 --2- 172.21.15.189:0/933889500 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f25980a1210 0x7f2598138fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:37.127 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:37.120+0000 7f259759e700 1 -- 172.21.15.189:0/933889500 >> 172.21.15.189:0/933889500 conn(0x7f2598063210 msgr2=0x7f2598134a10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:37.127 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:37.120+0000 7f259759e700 1 -- 172.21.15.189:0/933889500 shutdown_connections 2020-11-20T12:16:37.128 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:37.120+0000 7f259759e700 1 -- 172.21.15.189:0/933889500 wait complete. 2020-11-20T12:16:37.128 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 50 2020-11-20T12:16:37.141 INFO:tasks.mds_thrash.fs.[None]:mds.b has gained rank=0, replacing gid=5389 2020-11-20T12:16:37.142 INFO:tasks.mds_thrash.fs.[None]:waiting for 3 secs before reviving mds.a 2020-11-20T12:16:38.765 INFO:tasks.ceph.mds.b.smithi047.stderr:2020-11-20T12:16:38.758+0000 7f4610c91700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:16:40.145 INFO:tasks.mds_thrash.fs.[None]:reviving mds.a 2020-11-20T12:16:40.145 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:16:40.146 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:16:40.164 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:16:40.165 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:16:40.254 INFO:tasks.ceph.mds.a.smithi189.stdout:starting mds.a at 2020-11-20T12:16:40.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.304+0000 7faf0d07d700 1 Processor -- start 2020-11-20T12:16:40.312 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.304+0000 7faf0d07d700 1 -- start start 2020-11-20T12:16:40.324 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.308+0000 7faf0d07d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faf080623c0 0x7faf08138290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:40.325 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.308+0000 7faf0d07d700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7faf08061040 0x7faf0813bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:40.325 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.308+0000 7faf0d07d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf08065ca0 con 0x7faf08060bc0 2020-11-20T12:16:40.325 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.308+0000 7faf0d07d700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf080687e0 con 0x7faf08061040 2020-11-20T12:16:40.325 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.308+0000 7faf0d07d700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf08138ba0 con 0x7faf080623c0 2020-11-20T12:16:40.326 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.308+0000 7faeff7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7faf08060bc0 0x7faf0813b860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52856/0 (socket says 172.21.15.189:52856) 2020-11-20T12:16:40.326 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.308+0000 7faeff7fe700 1 -- 172.21.15.189:0/3163870817 learned_addr learned my addr 172.21.15.189:0/3163870817 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:16:40.326 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.308+0000 7faf07fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faf080623c0 0x7faf08138290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:40.326 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.308+0000 7faf0c87c700 1 --2- 172.21.15.189:0/3163870817 >> v2:172.21.15.47:3300/0 conn(0x7faf08061040 0x7faf0813bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:40.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.308+0000 7faf07fff700 1 -- 172.21.15.189:0/3163870817 >> v2:172.21.15.47:3300/0 conn(0x7faf08061040 msgr2=0x7faf0813bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:16:40.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.308+0000 7faf07fff700 1 --2- 172.21.15.189:0/3163870817 >> v2:172.21.15.47:3300/0 conn(0x7faf08061040 0x7faf0813bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:40.327 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.308+0000 7faf077fe700 1 -- 172.21.15.189:0/3163870817 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852138240 0 0) 0x7faf08065ca0 con 0x7faf08060bc0 2020-11-20T12:16:40.328 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.308+0000 7faf07fff700 1 -- 172.21.15.189:0/3163870817 >> v1:172.21.15.189:6789/0 conn(0x7faf08060bc0 legacy=0x7faf0813b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:40.328 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.308+0000 7faf07fff700 1 -- 172.21.15.189:0/3163870817 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf08138e40 con 0x7faf080623c0 2020-11-20T12:16:40.328 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.308+0000 7faf07fff700 1 --2- 172.21.15.189:0/3163870817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf080623c0 0x7faf08138290 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7faf080ae8f0 tx=0x7faef8006df0).ready entity=mon.1 client_cookie=a8f0f97c25554c53 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:40.328 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.308+0000 7faf077fe700 1 -- 172.21.15.189:0/3163870817 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef8009840 con 0x7faf080623c0 2020-11-20T12:16:40.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.308+0000 7faf077fe700 1 -- 172.21.15.189:0/3163870817 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faef80099a0 con 0x7faf080623c0 2020-11-20T12:16:40.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.308+0000 7faf077fe700 1 -- 172.21.15.189:0/3163870817 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef8009d20 con 0x7faf080623c0 2020-11-20T12:16:40.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.308+0000 7faf0d07d700 1 -- 172.21.15.189:0/3163870817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf080623c0 msgr2=0x7faf08138290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:40.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.308+0000 7faf0d07d700 1 --2- 172.21.15.189:0/3163870817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf080623c0 0x7faf08138290 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7faf080ae8f0 tx=0x7faef8006df0).stop 2020-11-20T12:16:40.329 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.308+0000 7faf0d07d700 1 -- 172.21.15.189:0/3163870817 shutdown_connections 2020-11-20T12:16:40.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.308+0000 7faf0d07d700 1 --2- 172.21.15.189:0/3163870817 >> v2:172.21.15.47:3300/0 conn(0x7faf08061040 0x7faf0813bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:40.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.308+0000 7faf0d07d700 1 --2- 172.21.15.189:0/3163870817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf080623c0 0x7faf08138290 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:40.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.308+0000 7faf0d07d700 1 -- 172.21.15.189:0/3163870817 >> 172.21.15.189:0/3163870817 conn(0x7faf08065fe0 msgr2=0x7faf08061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:40.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.308+0000 7faf0d07d700 1 -- 172.21.15.189:0/3163870817 shutdown_connections 2020-11-20T12:16:40.330 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.308+0000 7faf0d07d700 1 -- 172.21.15.189:0/3163870817 wait complete. 2020-11-20T12:16:40.331 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.312+0000 7faf0d07d700 1 Processor -- start 2020-11-20T12:16:40.331 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.312+0000 7faf0d07d700 1 -- start start 2020-11-20T12:16:40.331 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.312+0000 7faf0d07d700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7faf08060bc0 0x7faf080a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:40.331 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.312+0000 7faf0d07d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf08061040 0x7faf08149760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:40.331 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.312+0000 7faf0d07d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf080623c0 0x7faf080a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:40.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.312+0000 7faf0d07d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0814cd30 con 0x7faf08061040 2020-11-20T12:16:40.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.312+0000 7faf0d07d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0814cf60 con 0x7faf080623c0 2020-11-20T12:16:40.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.312+0000 7faf0d07d700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf0814d200 con 0x7faf08060bc0 2020-11-20T12:16:40.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.312+0000 7faf0c87c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf080623c0 0x7faf080a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:40.332 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.312+0000 7faf0c87c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf080623c0 0x7faf080a3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51244/0 (socket says 172.21.15.189:51244) 2020-11-20T12:16:40.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.312+0000 7faf0c87c700 1 -- 172.21.15.189:0/3514439293 learned_addr learned my addr 172.21.15.189:0/3514439293 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:16:40.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.312+0000 7faeff7fe700 1 --2- 172.21.15.189:0/3514439293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf08061040 0x7faf08149760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:40.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.312+0000 7faf07fff700 1 --2- 172.21.15.189:0/3514439293 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7faf08060bc0 0x7faf080a0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:40.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.312+0000 7faeff7fe700 1 -- 172.21.15.189:0/3514439293 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7faf08060bc0 msgr2=0x7faf080a0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:40.333 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.312+0000 7faeff7fe700 1 --2- 172.21.15.189:0/3514439293 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7faf08060bc0 0x7faf080a0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:40.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.312+0000 7faeff7fe700 1 -- 172.21.15.189:0/3514439293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf080623c0 msgr2=0x7faf080a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:40.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.312+0000 7faeff7fe700 1 --2- 172.21.15.189:0/3514439293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf080623c0 0x7faf080a3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:40.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.312+0000 7faeff7fe700 1 -- 172.21.15.189:0/3514439293 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf0814d4a0 con 0x7faf08061040 2020-11-20T12:16:40.334 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.312+0000 7faeff7fe700 1 --2- 172.21.15.189:0/3514439293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf08061040 0x7faf08149760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7faef000c760 tx=0x7faef0007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:40.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.312+0000 7faf04ff9700 1 -- 172.21.15.189:0/3514439293 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef00072e0 con 0x7faf08061040 2020-11-20T12:16:40.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.312+0000 7faf04ff9700 1 -- 172.21.15.189:0/3514439293 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faef0007440 con 0x7faf08061040 2020-11-20T12:16:40.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.312+0000 7faf0d07d700 1 -- 172.21.15.189:0/3514439293 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf08149c50 con 0x7faf08061040 2020-11-20T12:16:40.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.312+0000 7faf04ff9700 1 -- 172.21.15.189:0/3514439293 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef0005820 con 0x7faf08061040 2020-11-20T12:16:40.335 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.312+0000 7faf0d07d700 1 -- 172.21.15.189:0/3514439293 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf080a11e0 con 0x7faf08061040 2020-11-20T12:16:40.336 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.316+0000 7faf0d07d700 1 -- 172.21.15.189:0/3514439293 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf080663c0 con 0x7faf08061040 2020-11-20T12:16:40.336 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.316+0000 7faf04ff9700 1 -- 172.21.15.189:0/3514439293 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7faef0010070 con 0x7faf08061040 2020-11-20T12:16:40.336 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.316+0000 7faf04ff9700 1 --2- 172.21.15.189:0/3514439293 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7faee40557c0 0x7faee4057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:40.336 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.316+0000 7faf04ff9700 1 -- 172.21.15.189:0/3514439293 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6847+0+0 (secure 0 0 0) 0x7faef006df80 con 0x7faf08061040 2020-11-20T12:16:40.336 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.316+0000 7faf07fff700 1 --2- 172.21.15.189:0/3514439293 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7faee40557c0 0x7faee4057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:40.337 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.316+0000 7faf07fff700 1 --2- 172.21.15.189:0/3514439293 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7faee40557c0 0x7faee4057c80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7faef8006e30 tx=0x7faef8015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:40.338 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.324+0000 7faf04ff9700 1 -- 172.21.15.189:0/3514439293 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7faef0043b20 con 0x7faf08061040 2020-11-20T12:16:40.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.568+0000 7faf0d07d700 1 -- 172.21.15.189:0/3514439293 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf0814d630 con 0x7faf08061040 2020-11-20T12:16:40.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.568+0000 7faf04ff9700 1 -- 172.21.15.189:0/3514439293 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+1782 (secure 0 0 0) 0x7faef0047140 con 0x7faf08061040 2020-11-20T12:16:40.574 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:16:40.574 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:16:35.736803+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5075":{"gid":5075,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.47:6833/563711894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":563711894},{"type":"v1","addr":"172.21.15.47:6833","nonce":563711894}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:16:40.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.572+0000 7faf0d07d700 1 -- 172.21.15.189:0/3514439293 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7faee40557c0 msgr2=0x7faee4057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:40.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.572+0000 7faf0d07d700 1 --2- 172.21.15.189:0/3514439293 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7faee40557c0 0x7faee4057c80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7faef8006e30 tx=0x7faef8015000).stop 2020-11-20T12:16:40.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.572+0000 7faf0d07d700 1 -- 172.21.15.189:0/3514439293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf08061040 msgr2=0x7faf08149760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:40.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.572+0000 7faf0d07d700 1 --2- 172.21.15.189:0/3514439293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf08061040 0x7faf08149760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7faef000c760 tx=0x7faef0007c20).stop 2020-11-20T12:16:40.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.572+0000 7faf0d07d700 1 -- 172.21.15.189:0/3514439293 shutdown_connections 2020-11-20T12:16:40.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.572+0000 7faf0d07d700 1 --2- 172.21.15.189:0/3514439293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf080623c0 0x7faf080a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:40.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.572+0000 7faf0d07d700 1 --2- 172.21.15.189:0/3514439293 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7faf08060bc0 0x7faf080a0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:40.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.572+0000 7faf0d07d700 1 --2- 172.21.15.189:0/3514439293 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7faee40557c0 0x7faee4057c80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:40.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.572+0000 7faf0d07d700 1 --2- 172.21.15.189:0/3514439293 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7faf08061040 0x7faf08149760 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:40.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.572+0000 7faf0d07d700 1 -- 172.21.15.189:0/3514439293 >> 172.21.15.189:0/3514439293 conn(0x7faf08065fe0 msgr2=0x7faf080615e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:40.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.572+0000 7faf0d07d700 1 -- 172.21.15.189:0/3514439293 shutdown_connections 2020-11-20T12:16:40.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:40.572+0000 7faf0d07d700 1 -- 172.21.15.189:0/3514439293 wait complete. 2020-11-20T12:16:40.579 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 50 2020-11-20T12:16:40.588 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2020-11-20T12:16:42.590 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:16:42.722 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.716+0000 7f057759e700 1 Processor -- start 2020-11-20T12:16:42.722 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.716+0000 7f057759e700 1 -- start start 2020-11-20T12:16:42.723 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.720+0000 7f057759e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0578067b50 0x7f0578136fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:42.725 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.720+0000 7f057659c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f05780612b0 0x7f05780a0960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52872/0 (socket says 172.21.15.189:52872) 2020-11-20T12:16:42.725 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.720+0000 7f057659c700 1 -- 172.21.15.189:0/3219086165 learned_addr learned my addr 172.21.15.189:0/3219086165 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:16:42.725 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.720+0000 7f057759e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0578068020 con 0x7f05781374a0 2020-11-20T12:16:42.725 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.720+0000 7f056ffff700 1 --2- 172.21.15.189:0/3219086165 >> v2:172.21.15.189:3300/0 conn(0x7f0578067b50 0x7f0578136fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:42.726 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.720+0000 7f057759e700 1 -- 172.21.15.189:0/3219086165 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05780682c0 con 0x7f05780612b0 2020-11-20T12:16:42.726 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.720+0000 7f057759e700 1 -- 172.21.15.189:0/3219086165 --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0578137900 con 0x7f0578067b50 2020-11-20T12:16:42.726 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.720+0000 7f056ffff700 1 -- 172.21.15.189:0/3219086165 >> v1:172.21.15.189:6789/0 conn(0x7f05780612b0 legacy=0x7f05780a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:42.726 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.720+0000 7f056ffff700 1 -- 172.21.15.189:0/3219086165 >> v1:172.21.15.47:6789/0 conn(0x7f05781374a0 legacy=0x7f0578133a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:42.727 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.720+0000 7f056ffff700 1 -- 172.21.15.189:0/3219086165 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0578061690 con 0x7f0578067b50 2020-11-20T12:16:42.727 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.720+0000 7f056ffff700 1 --2- 172.21.15.189:0/3219086165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0578067b50 0x7f0578136fb0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f056000cb30 tx=0x7f0560006e80).ready entity=mon.1 client_cookie=df8c3c56e16d6bf1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:42.728 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.720+0000 7f0575d9b700 1 -- 172.21.15.189:0/3219086165 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056000aa50 con 0x7f0578067b50 2020-11-20T12:16:42.728 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.720+0000 7f0575d9b700 1 -- 172.21.15.189:0/3219086165 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056000abb0 con 0x7f0578067b50 2020-11-20T12:16:42.729 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.720+0000 7f0575d9b700 1 -- 172.21.15.189:0/3219086165 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056000ad20 con 0x7f0578067b50 2020-11-20T12:16:42.729 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.724+0000 7f057759e700 1 -- 172.21.15.189:0/3219086165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0578067b50 msgr2=0x7f0578136fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:42.729 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.724+0000 7f057759e700 1 --2- 172.21.15.189:0/3219086165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0578067b50 0x7f0578136fb0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f056000cb30 tx=0x7f0560006e80).stop 2020-11-20T12:16:42.729 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.724+0000 7f057759e700 1 -- 172.21.15.189:0/3219086165 shutdown_connections 2020-11-20T12:16:42.730 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.724+0000 7f057759e700 1 --2- 172.21.15.189:0/3219086165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0578067b50 0x7f0578136fb0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:42.730 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.724+0000 7f057759e700 1 -- 172.21.15.189:0/3219086165 >> 172.21.15.189:0/3219086165 conn(0x7f0578063210 msgr2=0x7f0578060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:42.730 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.724+0000 7f057759e700 1 -- 172.21.15.189:0/3219086165 shutdown_connections 2020-11-20T12:16:42.730 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.724+0000 7f057759e700 1 -- 172.21.15.189:0/3219086165 wait complete. 2020-11-20T12:16:42.732 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.724+0000 7f057759e700 1 Processor -- start 2020-11-20T12:16:42.733 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.724+0000 7f057759e700 1 -- start start 2020-11-20T12:16:42.733 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.724+0000 7f057759e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05780612b0 0x7f05780a3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:42.733 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.724+0000 7f057759e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0578067b50 0x7f05780a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:42.733 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.724+0000 7f057759e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f05781374a0 0x7f0578138aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:42.733 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.724+0000 7f057759e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05781383d0 con 0x7f05781374a0 2020-11-20T12:16:42.734 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.724+0000 7f057759e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0578146580 con 0x7f05780612b0 2020-11-20T12:16:42.734 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.724+0000 7f057759e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0578138600 con 0x7f0578067b50 2020-11-20T12:16:42.734 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.724+0000 7f057659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05780612b0 0x7f05780a3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:42.734 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.724+0000 7f057659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05780612b0 0x7f05780a3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51256/0 (socket says 172.21.15.189:51256) 2020-11-20T12:16:42.734 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.724+0000 7f057659c700 1 -- 172.21.15.189:0/1217065647 learned_addr learned my addr 172.21.15.189:0/1217065647 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:16:42.735 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.728+0000 7f0576d9d700 1 --2- 172.21.15.189:0/1217065647 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f05781374a0 0x7f0578138aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:42.735 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.728+0000 7f056ffff700 1 --2- 172.21.15.189:0/1217065647 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0578067b50 0x7f05780a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:42.735 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.728+0000 7f057659c700 1 -- 172.21.15.189:0/1217065647 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0578067b50 msgr2=0x7f05780a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:42.735 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.728+0000 7f057659c700 1 --2- 172.21.15.189:0/1217065647 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0578067b50 0x7f05780a3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:42.736 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.728+0000 7f057659c700 1 -- 172.21.15.189:0/1217065647 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f05781374a0 msgr2=0x7f0578138aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:42.736 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.728+0000 7f057659c700 1 --2- 172.21.15.189:0/1217065647 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f05781374a0 0x7f0578138aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:42.736 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.728+0000 7f057659c700 1 -- 172.21.15.189:0/1217065647 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057813a2a0 con 0x7f05780612b0 2020-11-20T12:16:42.736 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.728+0000 7f0576d9d700 1 --2- 172.21.15.189:0/1217065647 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f05781374a0 0x7f0578138aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:16:42.736 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.728+0000 7f056ffff700 1 --2- 172.21.15.189:0/1217065647 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0578067b50 0x7f05780a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:16:42.737 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.728+0000 7f057659c700 1 --2- 172.21.15.189:0/1217065647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05780612b0 0x7f05780a3980 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f056800c710 tx=0x7f0568007c40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:42.737 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.728+0000 7f056e7fc700 1 -- 172.21.15.189:0/1217065647 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0568007320 con 0x7f05780612b0 2020-11-20T12:16:42.737 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.728+0000 7f056e7fc700 1 -- 172.21.15.189:0/1217065647 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0568007480 con 0x7f05780612b0 2020-11-20T12:16:42.737 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.728+0000 7f056e7fc700 1 -- 172.21.15.189:0/1217065647 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0568005800 con 0x7f05780612b0 2020-11-20T12:16:42.737 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.728+0000 7f057759e700 1 -- 172.21.15.189:0/1217065647 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05780a73c0 con 0x7f05780612b0 2020-11-20T12:16:42.738 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.728+0000 7f057759e700 1 -- 172.21.15.189:0/1217065647 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0578139130 con 0x7f05780612b0 2020-11-20T12:16:42.738 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.728+0000 7f057759e700 1 -- 172.21.15.189:0/1217065647 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05780446d0 con 0x7f05780612b0 2020-11-20T12:16:42.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.728+0000 7f056e7fc700 1 -- 172.21.15.189:0/1217065647 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f0568010030 con 0x7f05780612b0 2020-11-20T12:16:42.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.728+0000 7f056e7fc700 1 --2- 172.21.15.189:0/1217065647 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f05540557c0 0x7f0554057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:16:42.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.728+0000 7f056e7fc700 1 -- 172.21.15.189:0/1217065647 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6847+0+0 (secure 0 0 0) 0x7f056806e630 con 0x7f05780612b0 2020-11-20T12:16:42.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.728+0000 7f056ffff700 1 --2- 172.21.15.189:0/1217065647 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f05540557c0 0x7f0554057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:16:42.740 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.728+0000 7f056ffff700 1 --2- 172.21.15.189:0/1217065647 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f05540557c0 0x7f0554057c80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f0560006bb0 tx=0x7f0560006e80).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:16:42.740 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.736+0000 7f056e7fc700 1 -- 172.21.15.189:0/1217065647 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f05680441d0 con 0x7f05780612b0 2020-11-20T12:16:42.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.952+0000 7f057759e700 1 -- 172.21.15.189:0/1217065647 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f057805cdd0 con 0x7f05780612b0 2020-11-20T12:16:42.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.952+0000 7f056e7fc700 1 -- 172.21.15.189:0/1217065647 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f05680477f0 con 0x7f05780612b0 2020-11-20T12:16:42.958 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:16:42.958 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5572,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/344330560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":344330560},{"type":"v1","addr":"172.21.15.189:6835","nonce":344330560}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":51}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:16:35.736803+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5075":{"gid":5075,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.47:6833/563711894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":563711894},{"type":"v1","addr":"172.21.15.47:6833","nonce":563711894}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:16:42.960 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.956+0000 7f057759e700 1 -- 172.21.15.189:0/1217065647 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f05540557c0 msgr2=0x7f0554057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:42.960 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.956+0000 7f057759e700 1 --2- 172.21.15.189:0/1217065647 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f05540557c0 0x7f0554057c80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f0560006bb0 tx=0x7f0560006e80).stop 2020-11-20T12:16:42.960 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.956+0000 7f057759e700 1 -- 172.21.15.189:0/1217065647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05780612b0 msgr2=0x7f05780a3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:16:42.961 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.956+0000 7f057759e700 1 --2- 172.21.15.189:0/1217065647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05780612b0 0x7f05780a3980 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f056800c710 tx=0x7f0568007c40).stop 2020-11-20T12:16:42.961 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.956+0000 7f057759e700 1 -- 172.21.15.189:0/1217065647 shutdown_connections 2020-11-20T12:16:42.961 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.956+0000 7f057759e700 1 --2- 172.21.15.189:0/1217065647 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f05540557c0 0x7f0554057c80 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:42.961 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.956+0000 7f057759e700 1 --2- 172.21.15.189:0/1217065647 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f05781374a0 0x7f0578138aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:42.962 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.956+0000 7f057759e700 1 --2- 172.21.15.189:0/1217065647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05780612b0 0x7f05780a3980 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:42.962 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.956+0000 7f057759e700 1 --2- 172.21.15.189:0/1217065647 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0578067b50 0x7f05780a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:16:42.962 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.956+0000 7f057759e700 1 -- 172.21.15.189:0/1217065647 >> 172.21.15.189:0/1217065647 conn(0x7f0578063210 msgr2=0x7f0578134a10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:16:42.962 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.956+0000 7f057759e700 1 -- 172.21.15.189:0/1217065647 shutdown_connections 2020-11-20T12:16:42.962 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:16:42.956+0000 7f057759e700 1 -- 172.21.15.189:0/1217065647 wait complete. 2020-11-20T12:16:42.963 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 51 2020-11-20T12:16:42.977 INFO:tasks.mds_thrash.fs.[None]:mds.a reported in up:standby state 2020-11-20T12:16:42.978 INFO:tasks.mds_thrash.fs.[None]:waiting for 55 secs before thrashing 2020-11-20T12:16:55.174 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:16:55.187 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:17:25.328 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:17:25.334 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:17:37.978 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:38.108 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.103+0000 7fa78f59e700 1 Processor -- start 2020-11-20T12:17:38.109 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.103+0000 7fa78f59e700 1 -- start start 2020-11-20T12:17:38.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.103+0000 7fa78f59e700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa7900a0a90 0x7fa7900a8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:38.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.103+0000 7fa78f59e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa7900a9cc0 0x7fa7900aa170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:38.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.103+0000 7fa78f59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa790066ee0 con 0x7fa79013dc20 2020-11-20T12:17:38.111 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.103+0000 7fa78f59e700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7900a96e0 con 0x7fa7900a0a90 2020-11-20T12:17:38.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.103+0000 7fa78f59e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa790061890 con 0x7fa7900a9cc0 2020-11-20T12:17:38.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.103+0000 7fa78dd9b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa79013dc20 0x7fa790061180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52888/0 (socket says 172.21.15.189:52888) 2020-11-20T12:17:38.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.103+0000 7fa78dd9b700 1 -- 172.21.15.189:0/2196967125 learned_addr learned my addr 172.21.15.189:0/2196967125 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:17:38.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.103+0000 7fa78e59c700 1 --2- 172.21.15.189:0/2196967125 >> v2:172.21.15.47:3300/0 conn(0x7fa7900a0a90 0x7fa7900a8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:38.112 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.103+0000 7fa78ed9d700 1 --2- 172.21.15.189:0/2196967125 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa7900a9cc0 0x7fa7900aa170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:38.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.103+0000 7fa78d59a700 1 -- 172.21.15.189:0/2196967125 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189958648 0 0) 0x7fa790066ee0 con 0x7fa79013dc20 2020-11-20T12:17:38.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.103+0000 7fa78d59a700 1 -- 172.21.15.189:0/2196967125 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa774003240 con 0x7fa79013dc20 2020-11-20T12:17:38.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.103+0000 7fa78ed9d700 1 -- 172.21.15.189:0/2196967125 >> v2:172.21.15.47:3300/0 conn(0x7fa7900a0a90 msgr2=0x7fa7900a8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:38.113 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.103+0000 7fa78ed9d700 1 --2- 172.21.15.189:0/2196967125 >> v2:172.21.15.47:3300/0 conn(0x7fa7900a0a90 0x7fa7900a8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:38.114 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.103+0000 7fa78ed9d700 1 -- 172.21.15.189:0/2196967125 >> v1:172.21.15.189:6789/0 conn(0x7fa79013dc20 legacy=0x7fa790061180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:38.114 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.103+0000 7fa78ed9d700 1 -- 172.21.15.189:0/2196967125 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa790067180 con 0x7fa7900a9cc0 2020-11-20T12:17:38.114 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.103+0000 7fa78ed9d700 1 --2- 172.21.15.189:0/2196967125 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa7900a9cc0 0x7fa7900aa170 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fa788009c90 tx=0x7fa7880091d0).ready entity=mon.2 client_cookie=5925c12c16c4bcc2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:38.115 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.103+0000 7fa78d59a700 1 -- 172.21.15.189:0/2196967125 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78800b9c0 con 0x7fa7900a9cc0 2020-11-20T12:17:38.115 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.103+0000 7fa78d59a700 1 -- 172.21.15.189:0/2196967125 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa78800bb20 con 0x7fa7900a9cc0 2020-11-20T12:17:38.115 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.103+0000 7fa78d59a700 1 -- 172.21.15.189:0/2196967125 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78800bea0 con 0x7fa7900a9cc0 2020-11-20T12:17:38.118 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78f59e700 1 -- 172.21.15.189:0/2196967125 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa7900a9cc0 msgr2=0x7fa7900aa170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:38.119 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78f59e700 1 --2- 172.21.15.189:0/2196967125 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa7900a9cc0 0x7fa7900aa170 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fa788009c90 tx=0x7fa7880091d0).stop 2020-11-20T12:17:38.119 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78f59e700 1 -- 172.21.15.189:0/2196967125 shutdown_connections 2020-11-20T12:17:38.119 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78f59e700 1 --2- 172.21.15.189:0/2196967125 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa7900a9cc0 0x7fa7900aa170 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:38.119 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78f59e700 1 --2- 172.21.15.189:0/2196967125 >> v2:172.21.15.47:3300/0 conn(0x7fa7900a0a90 0x7fa7900a8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:38.120 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78f59e700 1 -- 172.21.15.189:0/2196967125 >> 172.21.15.189:0/2196967125 conn(0x7fa790063210 msgr2=0x7fa790060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:38.120 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78f59e700 1 -- 172.21.15.189:0/2196967125 shutdown_connections 2020-11-20T12:17:38.120 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78f59e700 1 -- 172.21.15.189:0/2196967125 wait complete. 2020-11-20T12:17:38.120 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78f59e700 1 Processor -- start 2020-11-20T12:17:38.121 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78f59e700 1 -- start start 2020-11-20T12:17:38.121 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78f59e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa7900a0a90 0x7fa7900a2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:38.121 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78f59e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7900a9cc0 0x7fa790149690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:38.121 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78f59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa79013dc20 0x7fa790132610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:38.122 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78f59e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7900abb30 con 0x7fa7900a9cc0 2020-11-20T12:17:38.122 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78f59e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7900ad090 con 0x7fa79013dc20 2020-11-20T12:17:38.122 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78f59e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7900ad330 con 0x7fa7900a0a90 2020-11-20T12:17:38.122 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78dd9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7900a9cc0 0x7fa790149690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:38.122 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78e59c700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa7900a0a90 0x7fa7900a2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:38.123 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78e59c700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa7900a0a90 0x7fa7900a2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.189:44304/0 (socket says 172.21.15.189:44304) 2020-11-20T12:17:38.123 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78dd9b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7900a9cc0 0x7fa790149690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.189:57496/0 (socket says 172.21.15.189:57496) 2020-11-20T12:17:38.123 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78e59c700 1 -- 172.21.15.189:0/2251171858 learned_addr learned my addr 172.21.15.189:0/2251171858 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:17:38.123 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78ed9d700 1 --2- 172.21.15.189:0/2251171858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa79013dc20 0x7fa790132610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:38.124 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78dd9b700 1 -- 172.21.15.189:0/2251171858 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa7900a0a90 msgr2=0x7fa7900a2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:38.124 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78dd9b700 1 --2- 172.21.15.189:0/2251171858 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa7900a0a90 0x7fa7900a2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:38.124 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78dd9b700 1 -- 172.21.15.189:0/2251171858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa79013dc20 msgr2=0x7fa790132610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:38.125 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78dd9b700 1 --2- 172.21.15.189:0/2251171858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa79013dc20 0x7fa790132610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:38.125 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78dd9b700 1 -- 172.21.15.189:0/2251171858 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7900ad7d0 con 0x7fa7900a9cc0 2020-11-20T12:17:38.125 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78dd9b700 1 --2- 172.21.15.189:0/2251171858 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7900a9cc0 0x7fa790149690 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fa79013e090 tx=0x7fa7840079e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:38.126 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa77e7fc700 1 -- 172.21.15.189:0/2251171858 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7840072e0 con 0x7fa7900a9cc0 2020-11-20T12:17:38.126 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa77e7fc700 1 -- 172.21.15.189:0/2251171858 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa784007460 con 0x7fa7900a9cc0 2020-11-20T12:17:38.126 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa78f59e700 1 -- 172.21.15.189:0/2251171858 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa790139b40 con 0x7fa7900a9cc0 2020-11-20T12:17:38.126 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.107+0000 7fa77e7fc700 1 -- 172.21.15.189:0/2251171858 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa784005840 con 0x7fa7900a9cc0 2020-11-20T12:17:38.127 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.111+0000 7fa78f59e700 1 -- 172.21.15.189:0/2251171858 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7900ab8e0 con 0x7fa7900a9cc0 2020-11-20T12:17:38.127 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.111+0000 7fa78f59e700 1 -- 172.21.15.189:0/2251171858 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7900ad4f0 con 0x7fa7900a9cc0 2020-11-20T12:17:38.127 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.111+0000 7fa77e7fc700 1 -- 172.21.15.189:0/2251171858 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fa784010070 con 0x7fa7900a9cc0 2020-11-20T12:17:38.127 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.111+0000 7fa77e7fc700 1 --2- 172.21.15.189:0/2251171858 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa7640557c0 0x7fa764057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:38.127 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.111+0000 7fa77e7fc700 1 -- 172.21.15.189:0/2251171858 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6847+0+0 (secure 0 0 0) 0x7fa78406dd00 con 0x7fa7900a9cc0 2020-11-20T12:17:38.128 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.111+0000 7fa78e59c700 1 --2- 172.21.15.189:0/2251171858 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa7640557c0 0x7fa764057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:38.128 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.111+0000 7fa78e59c700 1 --2- 172.21.15.189:0/2251171858 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa7640557c0 0x7fa764057c80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fa780006a90 tx=0x7fa780006df0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:38.129 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.115+0000 7fa77e7fc700 1 -- 172.21.15.189:0/2251171858 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa784071030 con 0x7fa7900a9cc0 2020-11-20T12:17:38.381 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.375+0000 7fa78f59e700 1 -- 172.21.15.189:0/2251171858 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7900a0e70 con 0x7fa7900a9cc0 2020-11-20T12:17:38.382 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.375+0000 7fa77e7fc700 1 -- 172.21.15.189:0/2251171858 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2134 (secure 0 0 0) 0x7fa7840438a0 con 0x7fa7900a9cc0 2020-11-20T12:17:38.382 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:17:38.382 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5572,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/344330560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":344330560},{"type":"v1","addr":"172.21.15.189:6835","nonce":344330560}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":51}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:16:35.736803+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5075":{"gid":5075,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.47:6833/563711894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":563711894},{"type":"v1","addr":"172.21.15.47:6833","nonce":563711894}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:17:38.384 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.379+0000 7fa78f59e700 1 -- 172.21.15.189:0/2251171858 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa7640557c0 msgr2=0x7fa764057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:38.385 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.379+0000 7fa78f59e700 1 --2- 172.21.15.189:0/2251171858 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa7640557c0 0x7fa764057c80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fa780006a90 tx=0x7fa780006df0).stop 2020-11-20T12:17:38.385 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.379+0000 7fa78f59e700 1 -- 172.21.15.189:0/2251171858 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7900a9cc0 msgr2=0x7fa790149690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:38.385 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.379+0000 7fa78f59e700 1 --2- 172.21.15.189:0/2251171858 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7900a9cc0 0x7fa790149690 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fa79013e090 tx=0x7fa7840079e0).stop 2020-11-20T12:17:38.385 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.379+0000 7fa78f59e700 1 -- 172.21.15.189:0/2251171858 shutdown_connections 2020-11-20T12:17:38.386 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.379+0000 7fa78f59e700 1 --2- 172.21.15.189:0/2251171858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa79013dc20 0x7fa790132610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:38.386 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.379+0000 7fa78f59e700 1 --2- 172.21.15.189:0/2251171858 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa7900a0a90 0x7fa7900a2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:38.386 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.379+0000 7fa78f59e700 1 --2- 172.21.15.189:0/2251171858 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa7640557c0 0x7fa764057c80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:38.386 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.379+0000 7fa78f59e700 1 --2- 172.21.15.189:0/2251171858 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa7900a9cc0 0x7fa790149690 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:38.387 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.379+0000 7fa78f59e700 1 -- 172.21.15.189:0/2251171858 >> 172.21.15.189:0/2251171858 conn(0x7fa790063210 msgr2=0x7fa7900aa0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:38.387 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.379+0000 7fa78f59e700 1 -- 172.21.15.189:0/2251171858 shutdown_connections 2020-11-20T12:17:38.387 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.379+0000 7fa78f59e700 1 -- 172.21.15.189:0/2251171858 wait complete. 2020-11-20T12:17:38.387 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 51 2020-11-20T12:17:38.396 INFO:tasks.mds_thrash.fs.[None]:kill mds.b (rank=0) 2020-11-20T12:17:38.397 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:17:38.397 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:17:38.410 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:17:38.411 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:38.536 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.531+0000 7f8895c3d700 1 Processor -- start 2020-11-20T12:17:38.536 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.531+0000 7f8895c3d700 1 -- start start 2020-11-20T12:17:38.537 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.531+0000 7f8895c3d700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8890067b50 0x7f8890136fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:38.538 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.531+0000 7f8895c3d700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8890068020 con 0x7f88901374a0 2020-11-20T12:17:38.538 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.531+0000 7f8895c3d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88900682c0 con 0x7f88900612b0 2020-11-20T12:17:38.538 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.531+0000 7f8895c3d700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8890061690 con 0x7f8890067b50 2020-11-20T12:17:38.538 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.531+0000 7f8894c3b700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f88900612b0 0x7f88900a0960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52902/0 (socket says 172.21.15.189:52902) 2020-11-20T12:17:38.539 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.531+0000 7f8894c3b700 1 -- 172.21.15.189:0/3039964122 learned_addr learned my addr 172.21.15.189:0/3039964122 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:17:38.539 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.531+0000 7f888ffff700 1 --2- 172.21.15.189:0/3039964122 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8890067b50 0x7f8890136fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:38.540 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.531+0000 7f888f7fe700 1 -- 172.21.15.189:0/3039964122 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6699590 0 0) 0x7f88900682c0 con 0x7f88900612b0 2020-11-20T12:17:38.541 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.531+0000 7f888f7fe700 1 -- 172.21.15.189:0/3039964122 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8874004350 con 0x7f88900612b0 2020-11-20T12:17:38.541 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.531+0000 7f888f7fe700 1 -- 172.21.15.189:0/3039964122 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4182092511 0 0) 0x7f8890068020 con 0x7f88901374a0 2020-11-20T12:17:38.541 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.531+0000 7f888f7fe700 1 -- 172.21.15.189:0/3039964122 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88900682c0 con 0x7f88901374a0 2020-11-20T12:17:38.541 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.531+0000 7f888ffff700 1 -- 172.21.15.189:0/3039964122 >> v1:172.21.15.189:6789/0 conn(0x7f88900612b0 legacy=0x7f88900a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:38.542 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.531+0000 7f888ffff700 1 -- 172.21.15.189:0/3039964122 >> v1:172.21.15.47:6789/0 conn(0x7f88901374a0 legacy=0x7f8890133a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:38.542 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.531+0000 7f888ffff700 1 -- 172.21.15.189:0/3039964122 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88901378c0 con 0x7f8890067b50 2020-11-20T12:17:38.542 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.535+0000 7f888ffff700 1 --2- 172.21.15.189:0/3039964122 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8890067b50 0x7f8890136fb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f887c00b950 tx=0x7f887c006df0).ready entity=mon.2 client_cookie=331594724f34384c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:38.542 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.535+0000 7f888f7fe700 1 -- 172.21.15.189:0/3039964122 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c0095c0 con 0x7f8890067b50 2020-11-20T12:17:38.543 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.535+0000 7f888f7fe700 1 -- 172.21.15.189:0/3039964122 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887c009720 con 0x7f8890067b50 2020-11-20T12:17:38.543 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.535+0000 7f888f7fe700 1 -- 172.21.15.189:0/3039964122 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c009aa0 con 0x7f8890067b50 2020-11-20T12:17:38.543 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.535+0000 7f8895c3d700 1 -- 172.21.15.189:0/3039964122 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8890067b50 msgr2=0x7f8890136fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:38.543 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.535+0000 7f8895c3d700 1 --2- 172.21.15.189:0/3039964122 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8890067b50 0x7f8890136fb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f887c00b950 tx=0x7f887c006df0).stop 2020-11-20T12:17:38.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.535+0000 7f8895c3d700 1 -- 172.21.15.189:0/3039964122 shutdown_connections 2020-11-20T12:17:38.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.535+0000 7f8895c3d700 1 --2- 172.21.15.189:0/3039964122 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8890067b50 0x7f8890136fb0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:38.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.535+0000 7f8895c3d700 1 -- 172.21.15.189:0/3039964122 >> 172.21.15.189:0/3039964122 conn(0x7f8890063210 msgr2=0x7f8890060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:38.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.535+0000 7f8895c3d700 1 -- 172.21.15.189:0/3039964122 shutdown_connections 2020-11-20T12:17:38.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.535+0000 7f8895c3d700 1 -- 172.21.15.189:0/3039964122 wait complete. 2020-11-20T12:17:38.546 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.535+0000 7f8895c3d700 1 Processor -- start 2020-11-20T12:17:38.547 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.535+0000 7f8895c3d700 1 -- start start 2020-11-20T12:17:38.547 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.539+0000 7f8895c3d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88900612b0 0x7f88900a6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:38.547 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.539+0000 7f8895c3d700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8890067b50 0x7f8890145a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:38.547 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.539+0000 7f8895c3d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88901374a0 0x7f8890145f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:38.548 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.539+0000 7f8895c3d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8890149540 con 0x7f88901374a0 2020-11-20T12:17:38.548 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.539+0000 7f8895c3d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88901497e0 con 0x7f88900612b0 2020-11-20T12:17:38.548 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.539+0000 7f8895c3d700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8890149a80 con 0x7f8890067b50 2020-11-20T12:17:38.548 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.539+0000 7f8894c3b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88900612b0 0x7f88900a6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:38.549 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.539+0000 7f8894c3b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88900612b0 0x7f88900a6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51286/0 (socket says 172.21.15.189:51286) 2020-11-20T12:17:38.549 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.539+0000 7f8894c3b700 1 -- 172.21.15.189:0/2763117372 learned_addr learned my addr 172.21.15.189:0/2763117372 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:17:38.549 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.539+0000 7f889543c700 1 --2- 172.21.15.189:0/2763117372 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88901374a0 0x7f8890145f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:38.549 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.539+0000 7f888ffff700 1 --2- 172.21.15.189:0/2763117372 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8890067b50 0x7f8890145a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:38.549 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.539+0000 7f8894c3b700 1 -- 172.21.15.189:0/2763117372 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8890067b50 msgr2=0x7f8890145a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:38.550 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.539+0000 7f8894c3b700 1 --2- 172.21.15.189:0/2763117372 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8890067b50 0x7f8890145a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:38.550 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.539+0000 7f8894c3b700 1 -- 172.21.15.189:0/2763117372 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88901374a0 msgr2=0x7f8890145f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:38.550 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.539+0000 7f8894c3b700 1 --2- 172.21.15.189:0/2763117372 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88901374a0 0x7f8890145f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:38.550 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.539+0000 7f8894c3b700 1 -- 172.21.15.189:0/2763117372 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88900a4c90 con 0x7f88900612b0 2020-11-20T12:17:38.551 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.539+0000 7f8894c3b700 1 --2- 172.21.15.189:0/2763117372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88900612b0 0x7f88900a6e20 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f888400ee40 tx=0x7f8884001ba0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:38.551 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.539+0000 7f888cff9700 1 -- 172.21.15.189:0/2763117372 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8884013070 con 0x7f88900612b0 2020-11-20T12:17:38.551 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.539+0000 7f888cff9700 1 -- 172.21.15.189:0/2763117372 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f888400e2e0 con 0x7f88900612b0 2020-11-20T12:17:38.551 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.539+0000 7f888cff9700 1 -- 172.21.15.189:0/2763117372 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8884007170 con 0x7f88900612b0 2020-11-20T12:17:38.551 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.539+0000 7f8895c3d700 1 -- 172.21.15.189:0/2763117372 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88900ac5a0 con 0x7f88900612b0 2020-11-20T12:17:38.552 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.539+0000 7f8895c3d700 1 -- 172.21.15.189:0/2763117372 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8890148fc0 con 0x7f88900612b0 2020-11-20T12:17:38.552 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.539+0000 7f8895c3d700 1 -- 172.21.15.189:0/2763117372 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88901317c0 con 0x7f88900612b0 2020-11-20T12:17:38.552 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.539+0000 7f888cff9700 1 -- 172.21.15.189:0/2763117372 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f8884020070 con 0x7f88900612b0 2020-11-20T12:17:38.553 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.543+0000 7f888cff9700 1 --2- 172.21.15.189:0/2763117372 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f8868055810 0x7f8868057cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:38.553 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.543+0000 7f888cff9700 1 -- 172.21.15.189:0/2763117372 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6847+0+0 (secure 0 0 0) 0x7f8884072390 con 0x7f88900612b0 2020-11-20T12:17:38.554 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.543+0000 7f888ffff700 1 --2- 172.21.15.189:0/2763117372 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f8868055810 0x7f8868057cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:38.554 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.543+0000 7f888ffff700 1 --2- 172.21.15.189:0/2763117372 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f8868055810 0x7f8868057cd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f8890067f50 tx=0x7f887c015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:38.554 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.547+0000 7f888cff9700 1 -- 172.21.15.189:0/2763117372 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f88901317c0 con 0x7f88900612b0 2020-11-20T12:17:38.795 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.791+0000 7f8895c3d700 1 -- 172.21.15.189:0/2763117372 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88901317c0 con 0x7f88900612b0 2020-11-20T12:17:38.796 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.791+0000 7f888cff9700 1 -- 172.21.15.189:0/2763117372 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f88901317c0 con 0x7f88900612b0 2020-11-20T12:17:38.796 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:17:38.797 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5572,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/344330560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":344330560},{"type":"v1","addr":"172.21.15.189:6835","nonce":344330560}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":51}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:16:35.736803+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5075":{"gid":5075,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.47:6833/563711894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":563711894},{"type":"v1","addr":"172.21.15.47:6833","nonce":563711894}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:17:38.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.791+0000 7f8895c3d700 1 -- 172.21.15.189:0/2763117372 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f8868055810 msgr2=0x7f8868057cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:38.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.791+0000 7f8895c3d700 1 --2- 172.21.15.189:0/2763117372 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f8868055810 0x7f8868057cd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f8890067f50 tx=0x7f887c015000).stop 2020-11-20T12:17:38.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.791+0000 7f8895c3d700 1 -- 172.21.15.189:0/2763117372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88900612b0 msgr2=0x7f88900a6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:38.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.791+0000 7f8895c3d700 1 --2- 172.21.15.189:0/2763117372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88900612b0 0x7f88900a6e20 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f888400ee40 tx=0x7f8884001ba0).stop 2020-11-20T12:17:38.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.791+0000 7f8895c3d700 1 -- 172.21.15.189:0/2763117372 shutdown_connections 2020-11-20T12:17:38.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.791+0000 7f8895c3d700 1 --2- 172.21.15.189:0/2763117372 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f8868055810 0x7f8868057cd0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:38.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.791+0000 7f8895c3d700 1 --2- 172.21.15.189:0/2763117372 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88901374a0 0x7f8890145f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:38.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.791+0000 7f8895c3d700 1 --2- 172.21.15.189:0/2763117372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88900612b0 0x7f88900a6e20 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:38.801 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.791+0000 7f8895c3d700 1 --2- 172.21.15.189:0/2763117372 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8890067b50 0x7f8890145a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:38.801 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.791+0000 7f8895c3d700 1 -- 172.21.15.189:0/2763117372 >> 172.21.15.189:0/2763117372 conn(0x7f8890063210 msgr2=0x7f8890132f50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:38.801 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.791+0000 7f8895c3d700 1 -- 172.21.15.189:0/2763117372 shutdown_connections 2020-11-20T12:17:38.801 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:38.791+0000 7f8895c3d700 1 -- 172.21.15.189:0/2763117372 wait complete. 2020-11-20T12:17:38.802 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 51 2020-11-20T12:17:38.813 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:17:40.815 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:40.941 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.935+0000 7efcdb26b700 1 Processor -- start 2020-11-20T12:17:40.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.935+0000 7efcdb26b700 1 -- start start 2020-11-20T12:17:40.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.935+0000 7efcdb26b700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efcd413a6e0 0x7efcd413ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:40.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.935+0000 7efcdb26b700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcd4067730 con 0x7efcd40ad000 2020-11-20T12:17:40.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.935+0000 7efcdb26b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcd413dd40 con 0x7efcd40ad3e0 2020-11-20T12:17:40.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.935+0000 7efcdb26b700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcd4066050 con 0x7efcd413a6e0 2020-11-20T12:17:40.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.935+0000 7efcd9a68700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7efcd40ad3e0 0x7efcd4066d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52922/0 (socket says 172.21.15.189:52922) 2020-11-20T12:17:40.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.935+0000 7efcd9a68700 1 -- 172.21.15.189:0/1136928204 learned_addr learned my addr 172.21.15.189:0/1136928204 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:17:40.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcdaa6a700 1 --2- 172.21.15.189:0/1136928204 >> v2:172.21.15.47:3300/0 conn(0x7efcd413a6e0 0x7efcd413ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:40.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcd9267700 1 -- 172.21.15.189:0/1136928204 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215694211 0 0) 0x7efcd413dd40 con 0x7efcd40ad3e0 2020-11-20T12:17:40.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcd9267700 1 -- 172.21.15.189:0/1136928204 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcbc003240 con 0x7efcd40ad3e0 2020-11-20T12:17:40.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcd9267700 1 -- 172.21.15.189:0/1136928204 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 517745142 0 0) 0x7efcbc003240 con 0x7efcd40ad3e0 2020-11-20T12:17:40.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcd9267700 1 -- 172.21.15.189:0/1136928204 >> v2:172.21.15.47:3300/0 conn(0x7efcd413a6e0 msgr2=0x7efcd413ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:40.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcd9267700 1 --2- 172.21.15.189:0/1136928204 >> v2:172.21.15.47:3300/0 conn(0x7efcd413a6e0 0x7efcd413ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:40.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcd9267700 1 -- 172.21.15.189:0/1136928204 >> v1:172.21.15.47:6789/0 conn(0x7efcd40ad000 legacy=0x7efcd413e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:40.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcd9267700 1 -- 172.21.15.189:0/1136928204 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd413dfe0 con 0x7efcd40ad3e0 2020-11-20T12:17:40.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcd9267700 1 -- 172.21.15.189:0/1136928204 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7efcc40014e0 con 0x7efcd40ad3e0 2020-11-20T12:17:40.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcd9267700 1 -- 172.21.15.189:0/1136928204 >> v1:172.21.15.189:6789/0 conn(0x7efcd40ad3e0 legacy=0x7efcd4066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:40.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcd9267700 1 --2- 172.21.15.189:0/1136928204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcbc003730 0x7efcbc003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:40.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcd9267700 1 --2- 172.21.15.189:0/1136928204 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcbc0044e0 0x7efcbc004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:40.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcd9267700 1 --2- 172.21.15.189:0/1136928204 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efcbc004e50 0x7efcbc00b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:40.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcd9267700 1 -- 172.21.15.189:0/1136928204 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc004260 con 0x7efcbc0044e0 2020-11-20T12:17:40.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcd9267700 1 -- 172.21.15.189:0/1136928204 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc00bc20 con 0x7efcbc003730 2020-11-20T12:17:40.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcd9267700 1 -- 172.21.15.189:0/1136928204 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc00bd60 con 0x7efcbc004e50 2020-11-20T12:17:40.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcda269700 1 --2- 172.21.15.189:0/1136928204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcbc003730 0x7efcbc003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:40.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcd9a68700 1 --2- 172.21.15.189:0/1136928204 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcbc0044e0 0x7efcbc004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:40.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcdaa6a700 1 --2- 172.21.15.189:0/1136928204 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efcbc004e50 0x7efcbc00b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:40.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcda269700 1 -- 172.21.15.189:0/1136928204 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efcbc004e50 msgr2=0x7efcbc00b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:40.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcda269700 1 --2- 172.21.15.189:0/1136928204 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efcbc004e50 0x7efcbc00b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:40.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcda269700 1 -- 172.21.15.189:0/1136928204 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcbc0044e0 msgr2=0x7efcbc004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:40.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcda269700 1 --2- 172.21.15.189:0/1136928204 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcbc0044e0 0x7efcbc004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:40.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcda269700 1 -- 172.21.15.189:0/1136928204 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcbc003240 con 0x7efcbc003730 2020-11-20T12:17:40.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcdaa6a700 1 -- 172.21.15.189:0/1136928204 reap_dead start 2020-11-20T12:17:40.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcda269700 1 --2- 172.21.15.189:0/1136928204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcbc003730 0x7efcbc003b50 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7efcd0001920 tx=0x7efcd0007b90).ready entity=mon.1 client_cookie=2535f204dc70ec44 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:40.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcd9267700 1 -- 172.21.15.189:0/1136928204 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd0007020 con 0x7efcbc003730 2020-11-20T12:17:40.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcd9267700 1 -- 172.21.15.189:0/1136928204 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd0007180 con 0x7efcbc003730 2020-11-20T12:17:40.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcd9267700 1 -- 172.21.15.189:0/1136928204 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd000c830 con 0x7efcbc003730 2020-11-20T12:17:40.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcdb26b700 1 -- 172.21.15.189:0/1136928204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcbc003730 msgr2=0x7efcbc003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:40.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcdb26b700 1 --2- 172.21.15.189:0/1136928204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcbc003730 0x7efcbc003b50 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7efcd0001920 tx=0x7efcd0007b90).stop 2020-11-20T12:17:40.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcdb26b700 1 -- 172.21.15.189:0/1136928204 shutdown_connections 2020-11-20T12:17:40.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcdb26b700 1 --2- 172.21.15.189:0/1136928204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcbc003730 0x7efcbc003b50 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:40.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcdb26b700 1 -- 172.21.15.189:0/1136928204 >> 172.21.15.189:0/1136928204 conn(0x7efcd4062080 msgr2=0x7efcd4062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:40.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcdb26b700 1 -- 172.21.15.189:0/1136928204 shutdown_connections 2020-11-20T12:17:40.954 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.939+0000 7efcdb26b700 1 -- 172.21.15.189:0/1136928204 wait complete. 2020-11-20T12:17:40.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.943+0000 7efcdb26b700 1 Processor -- start 2020-11-20T12:17:40.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.943+0000 7efcdb26b700 1 -- start start 2020-11-20T12:17:40.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.943+0000 7efcdb26b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcbc003730 0x7efcd40afdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:40.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.943+0000 7efcdb26b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcd414d6a0 0x7efcd414a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:40.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.943+0000 7efcdb26b700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efcd414a8b0 0x7efcd4136fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:40.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.943+0000 7efcdb26b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd40a9680 con 0x7efcd414d6a0 2020-11-20T12:17:40.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.943+0000 7efcdb26b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd414df90 con 0x7efcbc003730 2020-11-20T12:17:40.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.943+0000 7efcdb26b700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd4149f20 con 0x7efcd414a8b0 2020-11-20T12:17:40.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.943+0000 7efcda269700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcbc003730 0x7efcd40afdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:40.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.943+0000 7efcda269700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcbc003730 0x7efcd40afdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51312/0 (socket says 172.21.15.189:51312) 2020-11-20T12:17:40.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.943+0000 7efcda269700 1 -- 172.21.15.189:0/2973899400 learned_addr learned my addr 172.21.15.189:0/2973899400 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:17:40.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.947+0000 7efcd9a68700 1 --2- 172.21.15.189:0/2973899400 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcd414d6a0 0x7efcd414a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:40.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.947+0000 7efcdaa6a700 1 --2- 172.21.15.189:0/2973899400 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efcd414a8b0 0x7efcd4136fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:40.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.947+0000 7efcda269700 1 -- 172.21.15.189:0/2973899400 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efcd414a8b0 msgr2=0x7efcd4136fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:40.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.947+0000 7efcda269700 1 --2- 172.21.15.189:0/2973899400 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efcd414a8b0 0x7efcd4136fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:40.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.947+0000 7efcda269700 1 -- 172.21.15.189:0/2973899400 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcd414d6a0 msgr2=0x7efcd414a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:40.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.947+0000 7efcda269700 1 --2- 172.21.15.189:0/2973899400 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcd414d6a0 0x7efcd414a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:40.960 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.947+0000 7efcda269700 1 -- 172.21.15.189:0/2973899400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd4149c10 con 0x7efcbc003730 2020-11-20T12:17:40.960 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.947+0000 7efcd9a68700 1 --2- 172.21.15.189:0/2973899400 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcd414d6a0 0x7efcd414a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:17:40.960 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.947+0000 7efcda269700 1 --2- 172.21.15.189:0/2973899400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcbc003730 0x7efcd40afdd0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7efcd0008fd0 tx=0x7efcd0015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:40.960 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.947+0000 7efcc27fc700 1 -- 172.21.15.189:0/2973899400 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd0007560 con 0x7efcbc003730 2020-11-20T12:17:40.960 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.947+0000 7efcc27fc700 1 -- 172.21.15.189:0/2973899400 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd0013970 con 0x7efcbc003730 2020-11-20T12:17:40.961 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.947+0000 7efcdb26b700 1 -- 172.21.15.189:0/2973899400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcd40b0930 con 0x7efcbc003730 2020-11-20T12:17:40.961 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.947+0000 7efcc27fc700 1 -- 172.21.15.189:0/2973899400 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd0013d10 con 0x7efcbc003730 2020-11-20T12:17:40.961 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.947+0000 7efcdb26b700 1 -- 172.21.15.189:0/2973899400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcd414dcc0 con 0x7efcbc003730 2020-11-20T12:17:40.961 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.947+0000 7efcb3fff700 1 -- 172.21.15.189:0/2973899400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efca8004f90 con 0x7efcbc003730 2020-11-20T12:17:40.961 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.947+0000 7efcc27fc700 1 -- 172.21.15.189:0/2973899400 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7efcd0026020 con 0x7efcbc003730 2020-11-20T12:17:40.962 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.947+0000 7efcc27fc700 1 --2- 172.21.15.189:0/2973899400 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7efcac0557f0 0x7efcac057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:40.962 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.947+0000 7efcc27fc700 1 -- 172.21.15.189:0/2973899400 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6847+0+0 (secure 0 0 0) 0x7efcd006fec0 con 0x7efcbc003730 2020-11-20T12:17:40.962 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.947+0000 7efcd9a68700 1 --2- 172.21.15.189:0/2973899400 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7efcac0557f0 0x7efcac057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:40.962 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.951+0000 7efcd9a68700 1 --2- 172.21.15.189:0/2973899400 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7efcac0557f0 0x7efcac057cb0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7efcc40013b0 tx=0x7efcc4009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:40.963 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:40.951+0000 7efcc27fc700 1 -- 172.21.15.189:0/2973899400 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7efcd00458f0 con 0x7efcbc003730 2020-11-20T12:17:41.154 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:41.147+0000 7efcb3fff700 1 -- 172.21.15.189:0/2973899400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efca8001660 con 0x7efcbc003730 2020-11-20T12:17:41.155 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:41.147+0000 7efcc27fc700 1 -- 172.21.15.189:0/2973899400 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2134 (secure 0 0 0) 0x7efcd0048f10 con 0x7efcbc003730 2020-11-20T12:17:41.155 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:17:41.155 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5572,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/344330560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":344330560},{"type":"v1","addr":"172.21.15.189:6835","nonce":344330560}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":51}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:16:35.736803+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5075":{"gid":5075,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.47:6833/563711894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":563711894},{"type":"v1","addr":"172.21.15.47:6833","nonce":563711894}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:17:41.157 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:41.151+0000 7efcb3fff700 1 -- 172.21.15.189:0/2973899400 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7efcac0557f0 msgr2=0x7efcac057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:41.157 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:41.151+0000 7efcb3fff700 1 --2- 172.21.15.189:0/2973899400 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7efcac0557f0 0x7efcac057cb0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7efcc40013b0 tx=0x7efcc4009040).stop 2020-11-20T12:17:41.157 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:41.151+0000 7efcb3fff700 1 -- 172.21.15.189:0/2973899400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcbc003730 msgr2=0x7efcd40afdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:41.158 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:41.151+0000 7efcb3fff700 1 --2- 172.21.15.189:0/2973899400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcbc003730 0x7efcd40afdd0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7efcd0008fd0 tx=0x7efcd0015000).stop 2020-11-20T12:17:41.158 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:41.151+0000 7efcb3fff700 1 -- 172.21.15.189:0/2973899400 shutdown_connections 2020-11-20T12:17:41.158 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:41.151+0000 7efcb3fff700 1 --2- 172.21.15.189:0/2973899400 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efcd414a8b0 0x7efcd4136fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:41.158 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:41.151+0000 7efcb3fff700 1 --2- 172.21.15.189:0/2973899400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcbc003730 0x7efcd40afdd0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:41.158 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:41.151+0000 7efcb3fff700 1 --2- 172.21.15.189:0/2973899400 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7efcac0557f0 0x7efcac057cb0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:41.159 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:41.151+0000 7efcb3fff700 1 --2- 172.21.15.189:0/2973899400 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcd414d6a0 0x7efcd414a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:41.159 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:41.151+0000 7efcb3fff700 1 -- 172.21.15.189:0/2973899400 >> 172.21.15.189:0/2973899400 conn(0x7efcd4062080 msgr2=0x7efcd40aba30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:41.159 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:41.151+0000 7efcb3fff700 1 -- 172.21.15.189:0/2973899400 shutdown_connections 2020-11-20T12:17:41.159 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:41.151+0000 7efcb3fff700 1 -- 172.21.15.189:0/2973899400 wait complete. 2020-11-20T12:17:41.160 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 51 2020-11-20T12:17:41.170 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:17:43.171 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:43.303 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.299+0000 7f6d30ba2700 1 Processor -- start 2020-11-20T12:17:43.304 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.299+0000 7f6d30ba2700 1 -- start start 2020-11-20T12:17:43.305 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.299+0000 7f6d30ba2700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6d2c066de0 0x7f6d2c13ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:43.305 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.299+0000 7f6d30ba2700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d2c13b070 con 0x7f6d2c0671c0 2020-11-20T12:17:43.306 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.299+0000 7f6d30ba2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d2c13b300 con 0x7f6d2c13cea0 2020-11-20T12:17:43.306 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.299+0000 7f6d30ba2700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d2c13dd30 con 0x7f6d2c066de0 2020-11-20T12:17:43.306 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.299+0000 7f6d2a59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6d2c066de0 0x7f6d2c13ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:43.306 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.299+0000 7f6d2a59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6d2c066de0 0x7f6d2c13ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51322/0 (socket says 172.21.15.189:51322) 2020-11-20T12:17:43.306 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.299+0000 7f6d2ad9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6d2c13cea0 0x7f6d2c13d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52948/0 (socket says 172.21.15.189:52948) 2020-11-20T12:17:43.307 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.299+0000 7f6d2a59c700 1 -- 172.21.15.189:0/2916893971 learned_addr learned my addr 172.21.15.189:0/2916893971 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:17:43.307 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.299+0000 7f6d2a59c700 1 -- 172.21.15.189:0/2916893971 >> v1:172.21.15.189:6789/0 conn(0x7f6d2c13cea0 legacy=0x7f6d2c13d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:43.307 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.299+0000 7f6d2a59c700 1 -- 172.21.15.189:0/2916893971 >> v1:172.21.15.47:6789/0 conn(0x7f6d2c0671c0 legacy=0x7f6d2c13e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:43.307 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.299+0000 7f6d2a59c700 1 -- 172.21.15.189:0/2916893971 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d2c067650 con 0x7f6d2c066de0 2020-11-20T12:17:43.307 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.299+0000 7f6d2a59c700 1 --2- 172.21.15.189:0/2916893971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d2c066de0 0x7f6d2c13ab80 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f6d1c00ba20 tx=0x7f6d1c006eb0).ready entity=mon.1 client_cookie=620f760441ddd5fc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:43.308 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.299+0000 7f6d2959a700 1 -- 172.21.15.189:0/2916893971 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c009890 con 0x7f6d2c066de0 2020-11-20T12:17:43.308 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.299+0000 7f6d2959a700 1 -- 172.21.15.189:0/2916893971 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d1c0099f0 con 0x7f6d2c066de0 2020-11-20T12:17:43.308 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.299+0000 7f6d2959a700 1 -- 172.21.15.189:0/2916893971 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c009d70 con 0x7f6d2c066de0 2020-11-20T12:17:43.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.299+0000 7f6d30ba2700 1 -- 172.21.15.189:0/2916893971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d2c066de0 msgr2=0x7f6d2c13ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:43.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.299+0000 7f6d30ba2700 1 --2- 172.21.15.189:0/2916893971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d2c066de0 0x7f6d2c13ab80 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f6d1c00ba20 tx=0x7f6d1c006eb0).stop 2020-11-20T12:17:43.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.299+0000 7f6d30ba2700 1 -- 172.21.15.189:0/2916893971 shutdown_connections 2020-11-20T12:17:43.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.299+0000 7f6d30ba2700 1 --2- 172.21.15.189:0/2916893971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d2c066de0 0x7f6d2c13ab80 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:43.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.299+0000 7f6d30ba2700 1 -- 172.21.15.189:0/2916893971 >> 172.21.15.189:0/2916893971 conn(0x7f6d2c061f60 msgr2=0x7f6d2c062340 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:43.312 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.299+0000 7f6d30ba2700 1 -- 172.21.15.189:0/2916893971 shutdown_connections 2020-11-20T12:17:43.312 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.299+0000 7f6d30ba2700 1 -- 172.21.15.189:0/2916893971 wait complete. 2020-11-20T12:17:43.312 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.303+0000 7f6d30ba2700 1 Processor -- start 2020-11-20T12:17:43.312 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.303+0000 7f6d30ba2700 1 -- start start 2020-11-20T12:17:43.313 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.303+0000 7f6d30ba2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d2c066de0 0x7f6d2c0a8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:43.313 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.303+0000 7f6d30ba2700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6d2c0671c0 0x7f6d2c0a7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:43.313 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.303+0000 7f6d30ba2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d2c13cea0 0x7f6d2c14cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:43.313 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.303+0000 7f6d30ba2700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c0a6060 con 0x7f6d2c066de0 2020-11-20T12:17:43.313 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.303+0000 7f6d30ba2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c0a6300 con 0x7f6d2c13cea0 2020-11-20T12:17:43.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.303+0000 7f6d30ba2700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c0a65a0 con 0x7f6d2c0671c0 2020-11-20T12:17:43.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.303+0000 7f6d2ad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d2c13cea0 0x7f6d2c14cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:43.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.303+0000 7f6d2ad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d2c13cea0 0x7f6d2c14cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51332/0 (socket says 172.21.15.189:51332) 2020-11-20T12:17:43.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.303+0000 7f6d2a59c700 1 --2- 172.21.15.189:0/2550631610 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d2c066de0 0x7f6d2c0a8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:43.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.303+0000 7f6d2ad9d700 1 -- 172.21.15.189:0/2550631610 learned_addr learned my addr 172.21.15.189:0/2550631610 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:17:43.315 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.303+0000 7f6d29d9b700 1 --2- 172.21.15.189:0/2550631610 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6d2c0671c0 0x7f6d2c0a7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:43.315 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.303+0000 7f6d2ad9d700 1 -- 172.21.15.189:0/2550631610 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6d2c0671c0 msgr2=0x7f6d2c0a7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:43.315 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.303+0000 7f6d2ad9d700 1 --2- 172.21.15.189:0/2550631610 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6d2c0671c0 0x7f6d2c0a7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:43.315 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.303+0000 7f6d2ad9d700 1 -- 172.21.15.189:0/2550631610 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d2c066de0 msgr2=0x7f6d2c0a8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:43.315 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.303+0000 7f6d2ad9d700 1 --2- 172.21.15.189:0/2550631610 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d2c066de0 0x7f6d2c0a8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:43.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.303+0000 7f6d2ad9d700 1 -- 172.21.15.189:0/2550631610 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d2c0a7a00 con 0x7f6d2c13cea0 2020-11-20T12:17:43.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.303+0000 7f6d2ad9d700 1 --2- 172.21.15.189:0/2550631610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d2c13cea0 0x7f6d2c14cef0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7f6d2400ca90 tx=0x7f6d24001360).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:43.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.303+0000 7f6d227fc700 1 -- 172.21.15.189:0/2550631610 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2400ce60 con 0x7f6d2c13cea0 2020-11-20T12:17:43.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.303+0000 7f6d227fc700 1 -- 172.21.15.189:0/2550631610 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d24005ac0 con 0x7f6d2c13cea0 2020-11-20T12:17:43.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.303+0000 7f6d30ba2700 1 -- 172.21.15.189:0/2550631610 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d2c0a5d50 con 0x7f6d2c13cea0 2020-11-20T12:17:43.317 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.303+0000 7f6d227fc700 1 -- 172.21.15.189:0/2550631610 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d24005e40 con 0x7f6d2c13cea0 2020-11-20T12:17:43.317 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.303+0000 7f6d30ba2700 1 -- 172.21.15.189:0/2550631610 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d2c14d3e0 con 0x7f6d2c13cea0 2020-11-20T12:17:43.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.307+0000 7f6d30ba2700 1 -- 172.21.15.189:0/2550631610 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d2c0a6760 con 0x7f6d2c13cea0 2020-11-20T12:17:43.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.307+0000 7f6d227fc700 1 -- 172.21.15.189:0/2550631610 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f6d24010030 con 0x7f6d2c13cea0 2020-11-20T12:17:43.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.307+0000 7f6d227fc700 1 --2- 172.21.15.189:0/2550631610 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6d080557c0 0x7f6d08057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:43.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.307+0000 7f6d227fc700 1 -- 172.21.15.189:0/2550631610 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6847+0+0 (secure 0 0 0) 0x7f6d2406e760 con 0x7f6d2c13cea0 2020-11-20T12:17:43.319 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.307+0000 7f6d2a59c700 1 --2- 172.21.15.189:0/2550631610 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6d080557c0 0x7f6d08057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:43.319 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.311+0000 7f6d2a59c700 1 --2- 172.21.15.189:0/2550631610 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6d080557c0 0x7f6d08057c80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f6d2c0a8870 tx=0x7f6d1c015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:43.321 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.315+0000 7f6d227fc700 1 -- 172.21.15.189:0/2550631610 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6d24044300 con 0x7f6d2c13cea0 2020-11-20T12:17:43.525 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.519+0000 7f6d30ba2700 1 -- 172.21.15.189:0/2550631610 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d2c044f60 con 0x7f6d2c13cea0 2020-11-20T12:17:43.526 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.519+0000 7f6d227fc700 1 -- 172.21.15.189:0/2550631610 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f6d24047920 con 0x7f6d2c13cea0 2020-11-20T12:17:43.526 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:17:43.527 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5572,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/344330560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":344330560},{"type":"v1","addr":"172.21.15.189:6835","nonce":344330560}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":51}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:16:35.736803+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5075":{"gid":5075,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.47:6833/563711894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":563711894},{"type":"v1","addr":"172.21.15.47:6833","nonce":563711894}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:17:43.529 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.523+0000 7f6d30ba2700 1 -- 172.21.15.189:0/2550631610 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6d080557c0 msgr2=0x7f6d08057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:43.530 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.523+0000 7f6d30ba2700 1 --2- 172.21.15.189:0/2550631610 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6d080557c0 0x7f6d08057c80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f6d2c0a8870 tx=0x7f6d1c015000).stop 2020-11-20T12:17:43.530 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.523+0000 7f6d30ba2700 1 -- 172.21.15.189:0/2550631610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d2c13cea0 msgr2=0x7f6d2c14cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:43.530 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.523+0000 7f6d30ba2700 1 --2- 172.21.15.189:0/2550631610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d2c13cea0 0x7f6d2c14cef0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7f6d2400ca90 tx=0x7f6d24001360).stop 2020-11-20T12:17:43.530 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.523+0000 7f6d30ba2700 1 -- 172.21.15.189:0/2550631610 shutdown_connections 2020-11-20T12:17:43.531 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.523+0000 7f6d30ba2700 1 --2- 172.21.15.189:0/2550631610 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f6d080557c0 0x7f6d08057c80 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:43.531 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.523+0000 7f6d30ba2700 1 --2- 172.21.15.189:0/2550631610 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6d2c066de0 0x7f6d2c0a8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:43.531 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.523+0000 7f6d30ba2700 1 --2- 172.21.15.189:0/2550631610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d2c13cea0 0x7f6d2c14cef0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:43.531 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.523+0000 7f6d30ba2700 1 --2- 172.21.15.189:0/2550631610 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6d2c0671c0 0x7f6d2c0a7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:43.531 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.523+0000 7f6d30ba2700 1 -- 172.21.15.189:0/2550631610 >> 172.21.15.189:0/2550631610 conn(0x7f6d2c061f60 msgr2=0x7f6d2c0adc80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:43.532 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.523+0000 7f6d30ba2700 1 -- 172.21.15.189:0/2550631610 shutdown_connections 2020-11-20T12:17:43.532 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:43.523+0000 7f6d30ba2700 1 -- 172.21.15.189:0/2550631610 wait complete. 2020-11-20T12:17:43.532 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 51 2020-11-20T12:17:43.546 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:17:45.547 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:45.677 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.671+0000 7f96eb59e700 1 Processor -- start 2020-11-20T12:17:45.678 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.671+0000 7f96eb59e700 1 -- start start 2020-11-20T12:17:45.680 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.671+0000 7f96eb59e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f96ec1386b0 0x7f96ec0a9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:45.680 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.671+0000 7f96eb59e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f96ec135b60 0x7f96ec0a5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:45.681 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.671+0000 7f96eb59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ec0a9a60 con 0x7f96ec0a50e0 2020-11-20T12:17:45.681 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.671+0000 7f96eb59e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96ec135ff0 con 0x7f96ec1386b0 2020-11-20T12:17:45.682 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.671+0000 7f96eb59e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96ec0a5a50 con 0x7f96ec135b60 2020-11-20T12:17:45.682 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.671+0000 7f96ea59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f96ec1386b0 0x7f96ec0a9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:45.683 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.671+0000 7f96ea59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f96ec1386b0 0x7f96ec0a9410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51340/0 (socket says 172.21.15.189:51340) 2020-11-20T12:17:45.684 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.671+0000 7f96ea59c700 1 -- 172.21.15.189:0/908658639 learned_addr learned my addr 172.21.15.189:0/908658639 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:17:45.684 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.671+0000 7f96ead9d700 1 --2- 172.21.15.189:0/908658639 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f96ec135b60 0x7f96ec0a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:45.684 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.671+0000 7f96ea59c700 1 -- 172.21.15.189:0/908658639 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f96ec135b60 msgr2=0x7f96ec0a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:45.685 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.671+0000 7f96ea59c700 1 --2- 172.21.15.189:0/908658639 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f96ec135b60 0x7f96ec0a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:45.690 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.671+0000 7f96ea59c700 1 -- 172.21.15.189:0/908658639 >> v1:172.21.15.47:6789/0 conn(0x7f96ec0a50e0 legacy=0x7f96ec135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:45.690 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.671+0000 7f96ea59c700 1 -- 172.21.15.189:0/908658639 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ec0a9d00 con 0x7f96ec1386b0 2020-11-20T12:17:45.691 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.671+0000 7f96ea59c700 1 --2- 172.21.15.189:0/908658639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96ec1386b0 0x7f96ec0a9410 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7f96d4006fd0 tx=0x7f96d4006eb0).ready entity=mon.1 client_cookie=383dfa40f8fa56c7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:45.691 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.675+0000 7f96e959a700 1 -- 172.21.15.189:0/908658639 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d40098a0 con 0x7f96ec1386b0 2020-11-20T12:17:45.691 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.675+0000 7f96e959a700 1 -- 172.21.15.189:0/908658639 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96d4009a00 con 0x7f96ec1386b0 2020-11-20T12:17:45.692 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.675+0000 7f96e959a700 1 -- 172.21.15.189:0/908658639 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d4009d80 con 0x7f96ec1386b0 2020-11-20T12:17:45.692 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.675+0000 7f96eb59e700 1 -- 172.21.15.189:0/908658639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96ec1386b0 msgr2=0x7f96ec0a9410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:45.693 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.675+0000 7f96eb59e700 1 --2- 172.21.15.189:0/908658639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96ec1386b0 0x7f96ec0a9410 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7f96d4006fd0 tx=0x7f96d4006eb0).stop 2020-11-20T12:17:45.694 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.675+0000 7f96eb59e700 1 -- 172.21.15.189:0/908658639 shutdown_connections 2020-11-20T12:17:45.694 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.675+0000 7f96eb59e700 1 --2- 172.21.15.189:0/908658639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96ec1386b0 0x7f96ec0a9410 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:45.694 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.675+0000 7f96eb59e700 1 --2- 172.21.15.189:0/908658639 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f96ec135b60 0x7f96ec0a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:45.695 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.675+0000 7f96eb59e700 1 -- 172.21.15.189:0/908658639 >> 172.21.15.189:0/908658639 conn(0x7f96ec060170 msgr2=0x7f96ec060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:45.695 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.675+0000 7f96eb59e700 1 -- 172.21.15.189:0/908658639 shutdown_connections 2020-11-20T12:17:45.696 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.675+0000 7f96eb59e700 1 -- 172.21.15.189:0/908658639 wait complete. 2020-11-20T12:17:45.696 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.679+0000 7f96eb59e700 1 Processor -- start 2020-11-20T12:17:45.696 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.679+0000 7f96eb59e700 1 -- start start 2020-11-20T12:17:45.697 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.679+0000 7f96eb59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96ec0a50e0 0x7f96ec124ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:45.697 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.679+0000 7f96eb59e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96ec135b60 0x7f96ec0660e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:45.697 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.679+0000 7f96eb59e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f96ec1386b0 0x7f96ec0665d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:45.698 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.679+0000 7f96eb59e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ec066ac0 con 0x7f96ec135b60 2020-11-20T12:17:45.698 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.679+0000 7f96eb59e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ec13c480 con 0x7f96ec0a50e0 2020-11-20T12:17:45.699 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.679+0000 7f96eb59e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96ec13b2d0 con 0x7f96ec1386b0 2020-11-20T12:17:45.699 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.679+0000 7f96ea59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96ec0a50e0 0x7f96ec124ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:45.700 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.679+0000 7f96ea59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96ec0a50e0 0x7f96ec124ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51344/0 (socket says 172.21.15.189:51344) 2020-11-20T12:17:45.700 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.679+0000 7f96ea59c700 1 -- 172.21.15.189:0/2524763769 learned_addr learned my addr 172.21.15.189:0/2524763769 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:17:45.700 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.679+0000 7f96e9d9b700 1 --2- 172.21.15.189:0/2524763769 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96ec135b60 0x7f96ec0660e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:45.701 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.679+0000 7f96ead9d700 1 --2- 172.21.15.189:0/2524763769 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f96ec1386b0 0x7f96ec0665d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:45.701 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.679+0000 7f96ea59c700 1 -- 172.21.15.189:0/2524763769 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f96ec1386b0 msgr2=0x7f96ec0665d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:45.703 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.679+0000 7f96ea59c700 1 --2- 172.21.15.189:0/2524763769 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f96ec1386b0 0x7f96ec0665d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:45.703 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.679+0000 7f96ea59c700 1 -- 172.21.15.189:0/2524763769 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96ec135b60 msgr2=0x7f96ec0660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:45.703 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.679+0000 7f96ea59c700 1 --2- 172.21.15.189:0/2524763769 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96ec135b60 0x7f96ec0660e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:45.704 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.679+0000 7f96ea59c700 1 -- 172.21.15.189:0/2524763769 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ec065c40 con 0x7f96ec0a50e0 2020-11-20T12:17:45.704 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.679+0000 7f96ea59c700 1 --2- 172.21.15.189:0/2524763769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96ec0a50e0 0x7f96ec124ba0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f96d4006ef0 tx=0x7f96d4015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:45.704 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.679+0000 7f96e27fc700 1 -- 172.21.15.189:0/2524763769 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d4011d50 con 0x7f96ec0a50e0 2020-11-20T12:17:45.705 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.679+0000 7f96e27fc700 1 -- 172.21.15.189:0/2524763769 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96d401e460 con 0x7f96ec0a50e0 2020-11-20T12:17:45.705 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.679+0000 7f96eb59e700 1 -- 172.21.15.189:0/2524763769 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96ec134db0 con 0x7f96ec0a50e0 2020-11-20T12:17:45.705 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.679+0000 7f96e27fc700 1 -- 172.21.15.189:0/2524763769 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d401e7e0 con 0x7f96ec0a50e0 2020-11-20T12:17:45.706 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.679+0000 7f96eb59e700 1 -- 172.21.15.189:0/2524763769 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ec13af50 con 0x7f96ec0a50e0 2020-11-20T12:17:45.706 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.683+0000 7f96eb59e700 1 -- 172.21.15.189:0/2524763769 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96ec065dd0 con 0x7f96ec0a50e0 2020-11-20T12:17:45.707 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.683+0000 7f96e27fc700 1 -- 172.21.15.189:0/2524763769 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f96d4025030 con 0x7f96ec0a50e0 2020-11-20T12:17:45.707 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.683+0000 7f96e27fc700 1 --2- 172.21.15.189:0/2524763769 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f96c80557c0 0x7f96c8057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:45.707 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.683+0000 7f96e27fc700 1 -- 172.21.15.189:0/2524763769 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6847+0+0 (secure 0 0 0) 0x7f96d40704f0 con 0x7f96ec0a50e0 2020-11-20T12:17:45.708 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.683+0000 7f96e9d9b700 1 --2- 172.21.15.189:0/2524763769 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f96c80557c0 0x7f96c8057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:45.709 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.683+0000 7f96e9d9b700 1 --2- 172.21.15.189:0/2524763769 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f96c80557c0 0x7f96c8057c80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f96ec12a3c0 tx=0x7f96dc000b80).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:45.709 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.687+0000 7f96e27fc700 1 -- 172.21.15.189:0/2524763769 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f96d40461c0 con 0x7f96ec0a50e0 2020-11-20T12:17:45.895 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.891+0000 7f96eb59e700 1 -- 172.21.15.189:0/2524763769 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96ec13b490 con 0x7f96ec0a50e0 2020-11-20T12:17:45.896 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.891+0000 7f96e27fc700 1 -- 172.21.15.189:0/2524763769 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f96d4023020 con 0x7f96ec0a50e0 2020-11-20T12:17:45.896 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:17:45.897 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5572,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/344330560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":344330560},{"type":"v1","addr":"172.21.15.189:6835","nonce":344330560}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":51}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:16:35.736803+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5075":{"gid":5075,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.47:6833/563711894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":563711894},{"type":"v1","addr":"172.21.15.47:6833","nonce":563711894}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:17:45.898 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.891+0000 7f96eb59e700 1 -- 172.21.15.189:0/2524763769 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f96c80557c0 msgr2=0x7f96c8057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:45.898 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.891+0000 7f96eb59e700 1 --2- 172.21.15.189:0/2524763769 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f96c80557c0 0x7f96c8057c80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f96ec12a3c0 tx=0x7f96dc000b80).stop 2020-11-20T12:17:45.899 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.891+0000 7f96eb59e700 1 -- 172.21.15.189:0/2524763769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96ec0a50e0 msgr2=0x7f96ec124ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:45.899 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.891+0000 7f96eb59e700 1 --2- 172.21.15.189:0/2524763769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96ec0a50e0 0x7f96ec124ba0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f96d4006ef0 tx=0x7f96d4015000).stop 2020-11-20T12:17:45.899 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.891+0000 7f96eb59e700 1 -- 172.21.15.189:0/2524763769 shutdown_connections 2020-11-20T12:17:45.899 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.891+0000 7f96eb59e700 1 --2- 172.21.15.189:0/2524763769 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f96ec1386b0 0x7f96ec0665d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:45.899 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.891+0000 7f96eb59e700 1 --2- 172.21.15.189:0/2524763769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96ec0a50e0 0x7f96ec124ba0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:45.900 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.891+0000 7f96eb59e700 1 --2- 172.21.15.189:0/2524763769 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f96c80557c0 0x7f96c8057c80 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:45.900 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.891+0000 7f96eb59e700 1 --2- 172.21.15.189:0/2524763769 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f96ec135b60 0x7f96ec0660e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:45.900 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.891+0000 7f96eb59e700 1 -- 172.21.15.189:0/2524763769 >> 172.21.15.189:0/2524763769 conn(0x7f96ec060170 msgr2=0x7f96ec060550 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:45.901 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.891+0000 7f96eb59e700 1 -- 172.21.15.189:0/2524763769 shutdown_connections 2020-11-20T12:17:45.901 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:45.891+0000 7f96eb59e700 1 -- 172.21.15.189:0/2524763769 wait complete. 2020-11-20T12:17:45.902 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 51 2020-11-20T12:17:45.911 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:17:47.913 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:48.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.035+0000 7ff13b59e700 1 Processor -- start 2020-11-20T12:17:48.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.035+0000 7ff13b59e700 1 -- start start 2020-11-20T12:17:48.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.039+0000 7ff13b59e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff13c0a8840 0x7ff13c13eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:48.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.039+0000 7ff13b59e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff13c066ed0 0x7ff13c13f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:48.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.039+0000 7ff13b59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff13c066800 con 0x7ff13c142090 2020-11-20T12:17:48.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.039+0000 7ff13b59e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff13c066aa0 con 0x7ff13c0a8840 2020-11-20T12:17:48.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.039+0000 7ff13b59e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff13c067450 con 0x7ff13c066ed0 2020-11-20T12:17:48.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.039+0000 7ff13a59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff13c0a8840 0x7ff13c13eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:48.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.039+0000 7ff13a59c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff13c0a8840 0x7ff13c13eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51354/0 (socket says 172.21.15.189:51354) 2020-11-20T12:17:48.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.039+0000 7ff13a59c700 1 -- 172.21.15.189:0/174273109 learned_addr learned my addr 172.21.15.189:0/174273109 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:17:48.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.039+0000 7ff131d9b700 1 --2- 172.21.15.189:0/174273109 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff13c066ed0 0x7ff13c13f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:48.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.039+0000 7ff13a59c700 1 -- 172.21.15.189:0/174273109 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff13c066ed0 msgr2=0x7ff13c13f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:48.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.039+0000 7ff13a59c700 1 --2- 172.21.15.189:0/174273109 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff13c066ed0 0x7ff13c13f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:48.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.039+0000 7ff139d9b700 1 -- 172.21.15.189:0/174273109 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600803645 0 0) 0x7ff13c066800 con 0x7ff13c142090 2020-11-20T12:17:48.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.039+0000 7ff13a59c700 1 -- 172.21.15.189:0/174273109 >> v1:172.21.15.47:6789/0 conn(0x7ff13c142090 legacy=0x7ff13c142520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:48.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.039+0000 7ff13a59c700 1 -- 172.21.15.189:0/174273109 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff13c0676f0 con 0x7ff13c0a8840 2020-11-20T12:17:48.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.039+0000 7ff13a59c700 1 --2- 172.21.15.189:0/174273109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff13c0a8840 0x7ff13c13eb40 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7ff124006fd0 tx=0x7ff124006eb0).ready entity=mon.1 client_cookie=30cc38c91c17fa74 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:48.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.039+0000 7ff139d9b700 1 -- 172.21.15.189:0/174273109 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff124009810 con 0x7ff13c0a8840 2020-11-20T12:17:48.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.039+0000 7ff139d9b700 1 -- 172.21.15.189:0/174273109 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff124009970 con 0x7ff13c0a8840 2020-11-20T12:17:48.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.039+0000 7ff139d9b700 1 -- 172.21.15.189:0/174273109 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff124009cf0 con 0x7ff13c0a8840 2020-11-20T12:17:48.052 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.039+0000 7ff13b59e700 1 -- 172.21.15.189:0/174273109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff13c0a8840 msgr2=0x7ff13c13eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:48.053 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.039+0000 7ff13b59e700 1 --2- 172.21.15.189:0/174273109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff13c0a8840 0x7ff13c13eb40 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7ff124006fd0 tx=0x7ff124006eb0).stop 2020-11-20T12:17:48.053 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.039+0000 7ff13b59e700 1 -- 172.21.15.189:0/174273109 shutdown_connections 2020-11-20T12:17:48.053 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.039+0000 7ff13b59e700 1 --2- 172.21.15.189:0/174273109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff13c0a8840 0x7ff13c13eb40 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:48.054 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.039+0000 7ff13b59e700 1 --2- 172.21.15.189:0/174273109 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff13c066ed0 0x7ff13c13f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:48.054 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.039+0000 7ff13b59e700 1 -- 172.21.15.189:0/174273109 >> 172.21.15.189:0/174273109 conn(0x7ff13c060170 msgr2=0x7ff13c060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:48.054 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.039+0000 7ff13b59e700 1 -- 172.21.15.189:0/174273109 shutdown_connections 2020-11-20T12:17:48.055 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.039+0000 7ff13b59e700 1 -- 172.21.15.189:0/174273109 wait complete. 2020-11-20T12:17:48.055 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.043+0000 7ff13b59e700 1 Processor -- start 2020-11-20T12:17:48.055 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.043+0000 7ff13b59e700 1 -- start start 2020-11-20T12:17:48.055 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.043+0000 7ff13b59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff13c066ed0 0x7ff13c0a30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:48.055 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.043+0000 7ff13b59e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff13c0a8840 0x7ff13c1434c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:48.056 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.043+0000 7ff13b59e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c142090 0x7ff13c0a1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:48.056 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.043+0000 7ff13b59e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13c0a2700 con 0x7ff13c142090 2020-11-20T12:17:48.056 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.043+0000 7ff13b59e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13c0a29a0 con 0x7ff13c066ed0 2020-11-20T12:17:48.056 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.043+0000 7ff13b59e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff13c0a2c40 con 0x7ff13c0a8840 2020-11-20T12:17:48.057 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.043+0000 7ff13a59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff13c066ed0 0x7ff13c0a30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:48.057 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.043+0000 7ff13a59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff13c066ed0 0x7ff13c0a30e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51358/0 (socket says 172.21.15.189:51358) 2020-11-20T12:17:48.057 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.043+0000 7ff13a59c700 1 -- 172.21.15.189:0/2452948621 learned_addr learned my addr 172.21.15.189:0/2452948621 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:17:48.057 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.043+0000 7ff131d9b700 1 --2- 172.21.15.189:0/2452948621 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff13c0a8840 0x7ff13c1434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:48.058 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.043+0000 7ff13ad9d700 1 --2- 172.21.15.189:0/2452948621 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c142090 0x7ff13c0a1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:48.058 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.047+0000 7ff13a59c700 1 -- 172.21.15.189:0/2452948621 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff13c0a8840 msgr2=0x7ff13c1434c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:48.058 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.047+0000 7ff13a59c700 1 --2- 172.21.15.189:0/2452948621 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff13c0a8840 0x7ff13c1434c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:48.058 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.047+0000 7ff13a59c700 1 -- 172.21.15.189:0/2452948621 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c142090 msgr2=0x7ff13c0a1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:48.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.047+0000 7ff13a59c700 1 --2- 172.21.15.189:0/2452948621 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c142090 0x7ff13c0a1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:48.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.047+0000 7ff13a59c700 1 -- 172.21.15.189:0/2452948621 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff13c0a0d40 con 0x7ff13c066ed0 2020-11-20T12:17:48.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.047+0000 7ff13a59c700 1 --2- 172.21.15.189:0/2452948621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff13c066ed0 0x7ff13c0a30e0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7ff124006ef0 tx=0x7ff124015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:48.059 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.047+0000 7ff132ffd700 1 -- 172.21.15.189:0/2452948621 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff124011e50 con 0x7ff13c066ed0 2020-11-20T12:17:48.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.047+0000 7ff132ffd700 1 -- 172.21.15.189:0/2452948621 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff12401e460 con 0x7ff13c066ed0 2020-11-20T12:17:48.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.047+0000 7ff13b59e700 1 -- 172.21.15.189:0/2452948621 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff13c0a2250 con 0x7ff13c066ed0 2020-11-20T12:17:48.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.047+0000 7ff132ffd700 1 -- 172.21.15.189:0/2452948621 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12401e7e0 con 0x7ff13c066ed0 2020-11-20T12:17:48.060 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.047+0000 7ff13b59e700 1 -- 172.21.15.189:0/2452948621 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff13c151550 con 0x7ff13c066ed0 2020-11-20T12:17:48.061 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.047+0000 7ff13b59e700 1 -- 172.21.15.189:0/2452948621 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff13c0a2e00 con 0x7ff13c066ed0 2020-11-20T12:17:48.062 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.047+0000 7ff132ffd700 1 -- 172.21.15.189:0/2452948621 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7ff124025030 con 0x7ff13c066ed0 2020-11-20T12:17:48.062 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.047+0000 7ff132ffd700 1 --2- 172.21.15.189:0/2452948621 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff10c0557c0 0x7ff10c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:48.062 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.051+0000 7ff132ffd700 1 -- 172.21.15.189:0/2452948621 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6847+0+0 (secure 0 0 0) 0x7ff124070030 con 0x7ff13c066ed0 2020-11-20T12:17:48.062 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.051+0000 7ff131d9b700 1 --2- 172.21.15.189:0/2452948621 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff10c0557c0 0x7ff10c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:48.063 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.051+0000 7ff131d9b700 1 --2- 172.21.15.189:0/2452948621 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff10c0557c0 0x7ff10c057c80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7ff13c0a26b0 tx=0x7ff12c006df0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:48.063 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.055+0000 7ff132ffd700 1 -- 172.21.15.189:0/2452948621 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff124045c80 con 0x7ff13c066ed0 2020-11-20T12:17:48.262 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.255+0000 7ff13b59e700 1 -- 172.21.15.189:0/2452948621 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff13c0a3b90 con 0x7ff13c066ed0 2020-11-20T12:17:48.263 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.255+0000 7ff132ffd700 1 -- 172.21.15.189:0/2452948621 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2134 (secure 0 0 0) 0x7ff124023030 con 0x7ff13c066ed0 2020-11-20T12:17:48.263 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:17:48.263 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5572,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/344330560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":344330560},{"type":"v1","addr":"172.21.15.189:6835","nonce":344330560}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":51}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:16:35.736803+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5075":{"gid":5075,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.47:6833/563711894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":563711894},{"type":"v1","addr":"172.21.15.47:6833","nonce":563711894}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:17:48.264 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.259+0000 7ff13b59e700 1 -- 172.21.15.189:0/2452948621 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff10c0557c0 msgr2=0x7ff10c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:48.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.259+0000 7ff13b59e700 1 --2- 172.21.15.189:0/2452948621 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff10c0557c0 0x7ff10c057c80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7ff13c0a26b0 tx=0x7ff12c006df0).stop 2020-11-20T12:17:48.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.259+0000 7ff13b59e700 1 -- 172.21.15.189:0/2452948621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff13c066ed0 msgr2=0x7ff13c0a30e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:48.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.259+0000 7ff13b59e700 1 --2- 172.21.15.189:0/2452948621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff13c066ed0 0x7ff13c0a30e0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7ff124006ef0 tx=0x7ff124015000).stop 2020-11-20T12:17:48.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.259+0000 7ff13b59e700 1 -- 172.21.15.189:0/2452948621 shutdown_connections 2020-11-20T12:17:48.265 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.259+0000 7ff13b59e700 1 --2- 172.21.15.189:0/2452948621 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff10c0557c0 0x7ff10c057c80 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:48.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.259+0000 7ff13b59e700 1 --2- 172.21.15.189:0/2452948621 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff13c142090 0x7ff13c0a1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:48.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.259+0000 7ff13b59e700 1 --2- 172.21.15.189:0/2452948621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff13c066ed0 0x7ff13c0a30e0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:48.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.259+0000 7ff13b59e700 1 --2- 172.21.15.189:0/2452948621 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff13c0a8840 0x7ff13c1434c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:48.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.259+0000 7ff13b59e700 1 -- 172.21.15.189:0/2452948621 >> 172.21.15.189:0/2452948621 conn(0x7ff13c060170 msgr2=0x7ff13c05ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:48.266 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.259+0000 7ff13b59e700 1 -- 172.21.15.189:0/2452948621 shutdown_connections 2020-11-20T12:17:48.267 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:48.259+0000 7ff13b59e700 1 -- 172.21.15.189:0/2452948621 wait complete. 2020-11-20T12:17:48.267 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 51 2020-11-20T12:17:48.277 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:17:50.279 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:50.411 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa64b59e700 1 Processor -- start 2020-11-20T12:17:50.412 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa64b59e700 1 -- start start 2020-11-20T12:17:50.413 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa64b59e700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa64c064e80 0x7fa64c0aadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:50.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa64b59e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa64c0aeb80 0x7fa64c0a9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:50.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa64b59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64c0a9b80 con 0x7fa64c064830 2020-11-20T12:17:50.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa64b59e700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa64c0ab2c0 con 0x7fa64c064e80 2020-11-20T12:17:50.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa64b59e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa64c0af090 con 0x7fa64c0aeb80 2020-11-20T12:17:50.414 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa64ad9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa64c064830 0x7fa64c0ab670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:52992/0 (socket says 172.21.15.189:52992) 2020-11-20T12:17:50.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa64ad9d700 1 -- 172.21.15.189:0/2325262377 learned_addr learned my addr 172.21.15.189:0/2325262377 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:17:50.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa649d9b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa64c0aeb80 0x7fa64c0a9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:50.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa64a59c700 1 --2- 172.21.15.189:0/2325262377 >> v2:172.21.15.47:3300/0 conn(0x7fa64c064e80 0x7fa64c0aadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:50.415 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa64959a700 1 -- 172.21.15.189:0/2325262377 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137624581 0 0) 0x7fa64c0a9b80 con 0x7fa64c064830 2020-11-20T12:17:50.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa64959a700 1 -- 172.21.15.189:0/2325262377 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa62c003240 con 0x7fa64c064830 2020-11-20T12:17:50.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa649d9b700 1 -- 172.21.15.189:0/2325262377 >> v2:172.21.15.47:3300/0 conn(0x7fa64c064e80 msgr2=0x7fa64c0aadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:50.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa649d9b700 1 --2- 172.21.15.189:0/2325262377 >> v2:172.21.15.47:3300/0 conn(0x7fa64c064e80 0x7fa64c0aadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:50.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa649d9b700 1 -- 172.21.15.189:0/2325262377 >> v1:172.21.15.189:6789/0 conn(0x7fa64c064830 legacy=0x7fa64c0ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:50.416 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa649d9b700 1 -- 172.21.15.189:0/2325262377 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64c0af330 con 0x7fa64c0aeb80 2020-11-20T12:17:50.417 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa649d9b700 1 --2- 172.21.15.189:0/2325262377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa64c0aeb80 0x7fa64c0a9690 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7fa64c063d80 tx=0x7fa644006df0).ready entity=mon.1 client_cookie=41d021cf87783e96 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:50.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa64959a700 1 -- 172.21.15.189:0/2325262377 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6440096d0 con 0x7fa64c0aeb80 2020-11-20T12:17:50.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa64959a700 1 -- 172.21.15.189:0/2325262377 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa644009830 con 0x7fa64c0aeb80 2020-11-20T12:17:50.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa64959a700 1 -- 172.21.15.189:0/2325262377 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa644009bb0 con 0x7fa64c0aeb80 2020-11-20T12:17:50.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa64b59e700 1 -- 172.21.15.189:0/2325262377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa64c0aeb80 msgr2=0x7fa64c0a9690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:50.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa64b59e700 1 --2- 172.21.15.189:0/2325262377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa64c0aeb80 0x7fa64c0a9690 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7fa64c063d80 tx=0x7fa644006df0).stop 2020-11-20T12:17:50.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa64b59e700 1 -- 172.21.15.189:0/2325262377 shutdown_connections 2020-11-20T12:17:50.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa64b59e700 1 --2- 172.21.15.189:0/2325262377 >> v2:172.21.15.47:3300/0 conn(0x7fa64c064e80 0x7fa64c0aadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:50.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa64b59e700 1 --2- 172.21.15.189:0/2325262377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa64c0aeb80 0x7fa64c0a9690 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:50.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa64b59e700 1 -- 172.21.15.189:0/2325262377 >> 172.21.15.189:0/2325262377 conn(0x7fa64c062080 msgr2=0x7fa64c062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:50.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa64b59e700 1 -- 172.21.15.189:0/2325262377 shutdown_connections 2020-11-20T12:17:50.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.407+0000 7fa64b59e700 1 -- 172.21.15.189:0/2325262377 wait complete. 2020-11-20T12:17:50.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.411+0000 7fa64b59e700 1 Processor -- start 2020-11-20T12:17:50.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.411+0000 7fa64b59e700 1 -- start start 2020-11-20T12:17:50.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.411+0000 7fa64b59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa64c064830 0x7fa64c142220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:50.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.411+0000 7fa64b59e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa64c064e80 0x7fa64c0a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:50.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.411+0000 7fa64b59e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa64c0aeb80 0x7fa64c1406f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:50.423 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.411+0000 7fa64b59e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c13ffb0 con 0x7fa64c064e80 2020-11-20T12:17:50.423 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.411+0000 7fa64b59e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c140250 con 0x7fa64c064830 2020-11-20T12:17:50.423 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.411+0000 7fa64b59e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c0a57f0 con 0x7fa64c0aeb80 2020-11-20T12:17:50.423 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.411+0000 7fa64a59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa64c064830 0x7fa64c142220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:50.424 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.411+0000 7fa64a59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa64c064830 0x7fa64c142220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51372/0 (socket says 172.21.15.189:51372) 2020-11-20T12:17:50.424 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.411+0000 7fa64a59c700 1 -- 172.21.15.189:0/107080354 learned_addr learned my addr 172.21.15.189:0/107080354 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:17:50.424 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.411+0000 7fa649d9b700 1 --2- 172.21.15.189:0/107080354 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa64c064e80 0x7fa64c0a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:50.424 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.411+0000 7fa64ad9d700 1 --2- 172.21.15.189:0/107080354 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa64c0aeb80 0x7fa64c1406f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:50.424 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.411+0000 7fa64a59c700 1 -- 172.21.15.189:0/107080354 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa64c0aeb80 msgr2=0x7fa64c1406f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:50.425 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.411+0000 7fa64a59c700 1 --2- 172.21.15.189:0/107080354 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa64c0aeb80 0x7fa64c1406f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:50.425 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.411+0000 7fa64a59c700 1 -- 172.21.15.189:0/107080354 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa64c064e80 msgr2=0x7fa64c0a5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:50.425 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.411+0000 7fa64a59c700 1 --2- 172.21.15.189:0/107080354 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa64c064e80 0x7fa64c0a5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:50.425 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.411+0000 7fa64a59c700 1 -- 172.21.15.189:0/107080354 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64c140be0 con 0x7fa64c064830 2020-11-20T12:17:50.425 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.411+0000 7fa64a59c700 1 --2- 172.21.15.189:0/107080354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa64c064830 0x7fa64c142220 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7fa63400ef50 tx=0x7fa6340091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:50.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.411+0000 7fa6427fc700 1 -- 172.21.15.189:0/107080354 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63400ccc0 con 0x7fa64c064830 2020-11-20T12:17:50.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.411+0000 7fa6427fc700 1 -- 172.21.15.189:0/107080354 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa63400ce20 con 0x7fa64c064830 2020-11-20T12:17:50.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.411+0000 7fa64b59e700 1 -- 172.21.15.189:0/107080354 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa64c0a42b0 con 0x7fa64c064830 2020-11-20T12:17:50.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.411+0000 7fa6427fc700 1 -- 172.21.15.189:0/107080354 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa634012600 con 0x7fa64c064830 2020-11-20T12:17:50.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.411+0000 7fa64b59e700 1 -- 172.21.15.189:0/107080354 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa64c14ddf0 con 0x7fa64c064830 2020-11-20T12:17:50.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.415+0000 7fa6427fc700 1 -- 172.21.15.189:0/107080354 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fa634022070 con 0x7fa64c064830 2020-11-20T12:17:50.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.415+0000 7fa64b59e700 1 -- 172.21.15.189:0/107080354 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa64c1403c0 con 0x7fa64c064830 2020-11-20T12:17:50.428 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.415+0000 7fa6427fc700 1 --2- 172.21.15.189:0/107080354 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa6280557c0 0x7fa628057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:50.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.415+0000 7fa6427fc700 1 -- 172.21.15.189:0/107080354 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6847+0+0 (secure 0 0 0) 0x7fa63406f210 con 0x7fa64c064830 2020-11-20T12:17:50.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.415+0000 7fa649d9b700 1 --2- 172.21.15.189:0/107080354 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa6280557c0 0x7fa628057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:50.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.415+0000 7fa649d9b700 1 --2- 172.21.15.189:0/107080354 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa6280557c0 0x7fa628057c80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fa64400b150 tx=0x7fa644015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:50.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.423+0000 7fa6427fc700 1 -- 172.21.15.189:0/107080354 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa634044db0 con 0x7fa64c064830 2020-11-20T12:17:50.630 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.623+0000 7fa64b59e700 1 -- 172.21.15.189:0/107080354 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa64c0446d0 con 0x7fa64c064830 2020-11-20T12:17:50.634 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.627+0000 7fa6427fc700 1 -- 172.21.15.189:0/107080354 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2134 (secure 0 0 0) 0x7fa634016020 con 0x7fa64c064830 2020-11-20T12:17:50.634 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:17:50.635 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5572,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/344330560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":344330560},{"type":"v1","addr":"172.21.15.189:6835","nonce":344330560}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":51}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:16:35.736803+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5075},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5075":{"gid":5075,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":20,"addr":"172.21.15.47:6833/563711894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":563711894},{"type":"v1","addr":"172.21.15.47:6833","nonce":563711894}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:17:50.637 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.631+0000 7fa64b59e700 1 -- 172.21.15.189:0/107080354 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa6280557c0 msgr2=0x7fa628057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:50.638 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.631+0000 7fa64b59e700 1 --2- 172.21.15.189:0/107080354 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa6280557c0 0x7fa628057c80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fa64400b150 tx=0x7fa644015000).stop 2020-11-20T12:17:50.638 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.631+0000 7fa64b59e700 1 -- 172.21.15.189:0/107080354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa64c064830 msgr2=0x7fa64c142220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:50.638 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.631+0000 7fa64b59e700 1 --2- 172.21.15.189:0/107080354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa64c064830 0x7fa64c142220 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7fa63400ef50 tx=0x7fa6340091d0).stop 2020-11-20T12:17:50.638 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.631+0000 7fa64b59e700 1 -- 172.21.15.189:0/107080354 shutdown_connections 2020-11-20T12:17:50.639 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.631+0000 7fa64b59e700 1 --2- 172.21.15.189:0/107080354 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa64c0aeb80 0x7fa64c1406f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:50.639 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.631+0000 7fa64b59e700 1 --2- 172.21.15.189:0/107080354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa64c064830 0x7fa64c142220 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:50.639 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.631+0000 7fa64b59e700 1 --2- 172.21.15.189:0/107080354 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa6280557c0 0x7fa628057c80 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:50.639 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.631+0000 7fa64b59e700 1 --2- 172.21.15.189:0/107080354 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa64c064e80 0x7fa64c0a5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:50.639 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.631+0000 7fa64b59e700 1 -- 172.21.15.189:0/107080354 >> 172.21.15.189:0/107080354 conn(0x7fa64c062080 msgr2=0x7fa64c0abe10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:50.640 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.631+0000 7fa64b59e700 1 -- 172.21.15.189:0/107080354 shutdown_connections 2020-11-20T12:17:50.640 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:50.631+0000 7fa64b59e700 1 -- 172.21.15.189:0/107080354 wait complete. 2020-11-20T12:17:50.640 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 51 2020-11-20T12:17:50.652 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:17:52.654 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:52.783 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.779+0000 7f7f8355a700 1 Processor -- start 2020-11-20T12:17:52.816 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.779+0000 7f7f8355a700 1 -- start start 2020-11-20T12:17:52.818 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.779+0000 7f7f8355a700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7f7c067b50 0x7f7f7c136fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:52.819 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.779+0000 7f7f8355a700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7f7c1374a0 0x7f7f7c133a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:52.819 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.779+0000 7f7f8355a700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f7c068020 con 0x7f7f7c0612b0 2020-11-20T12:17:52.819 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.779+0000 7f7f8355a700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f7c0682c0 con 0x7f7f7c1374a0 2020-11-20T12:17:52.819 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.779+0000 7f7f8355a700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7c061690 con 0x7f7f7c067b50 2020-11-20T12:17:52.819 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.779+0000 7f7f81d57700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7f7c067b50 0x7f7f7c136fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:52.820 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.779+0000 7f7f82d59700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7f7c1374a0 0x7f7f7c133a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:52.820 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.779+0000 7f7f81d57700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7f7c067b50 0x7f7f7c136fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.189:44426/0 (socket says 172.21.15.189:44426) 2020-11-20T12:17:52.820 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.779+0000 7f7f82d59700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f7f7c1374a0 0x7f7f7c133a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.189:57618/0 (socket says 172.21.15.189:57618) 2020-11-20T12:17:52.820 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.779+0000 7f7f81d57700 1 -- 172.21.15.189:0/2167954322 learned_addr learned my addr 172.21.15.189:0/2167954322 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:17:52.820 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.779+0000 7f7f82558700 1 --1- 172.21.15.189:0/2167954322 >> v1:172.21.15.47:6789/0 conn(0x7f7f7c0612b0 0x7f7f7c0a0960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.189:47676/0 (socket says 172.21.15.189:47676) 2020-11-20T12:17:52.821 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.779+0000 7f7f81d57700 1 -- 172.21.15.189:0/2167954322 >> v2:172.21.15.47:3300/0 conn(0x7f7f7c1374a0 msgr2=0x7f7f7c133a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:52.821 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.779+0000 7f7f81d57700 1 --2- 172.21.15.189:0/2167954322 >> v2:172.21.15.47:3300/0 conn(0x7f7f7c1374a0 0x7f7f7c133a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:52.821 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.779+0000 7f7f81d57700 1 -- 172.21.15.189:0/2167954322 >> v1:172.21.15.47:6789/0 conn(0x7f7f7c0612b0 legacy=0x7f7f7c0a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:52.821 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.779+0000 7f7f81d57700 1 -- 172.21.15.189:0/2167954322 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f7c1378c0 con 0x7f7f7c067b50 2020-11-20T12:17:52.822 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.779+0000 7f7f81556700 1 -- 172.21.15.189:0/2167954322 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1301948372 0 0) 0x7f7f7c068020 con 0x7f7f7c0612b0 2020-11-20T12:17:52.822 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.779+0000 7f7f82d59700 1 --2- 172.21.15.189:0/2167954322 >> v2:172.21.15.47:3300/0 conn(0x7f7f7c1374a0 0x7f7f7c133a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:17:52.822 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.779+0000 7f7f81d57700 1 --2- 172.21.15.189:0/2167954322 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7f7c067b50 0x7f7f7c136fb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f7f7800dcf0 tx=0x7f7f78006e40).ready entity=mon.2 client_cookie=44ed768ef050bfa3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:52.822 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.779+0000 7f7f81556700 1 -- 172.21.15.189:0/2167954322 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7800bd60 con 0x7f7f7c067b50 2020-11-20T12:17:52.822 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.779+0000 7f7f81556700 1 -- 172.21.15.189:0/2167954322 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f780153f0 con 0x7f7f7c067b50 2020-11-20T12:17:52.823 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.779+0000 7f7f81556700 1 -- 172.21.15.189:0/2167954322 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f78015770 con 0x7f7f7c067b50 2020-11-20T12:17:52.823 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.783+0000 7f7f8355a700 1 -- 172.21.15.189:0/2167954322 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7f7c067b50 msgr2=0x7f7f7c136fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:52.823 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.783+0000 7f7f8355a700 1 --2- 172.21.15.189:0/2167954322 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7f7c067b50 0x7f7f7c136fb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f7f7800dcf0 tx=0x7f7f78006e40).stop 2020-11-20T12:17:52.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.783+0000 7f7f8355a700 1 -- 172.21.15.189:0/2167954322 shutdown_connections 2020-11-20T12:17:52.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.783+0000 7f7f8355a700 1 --2- 172.21.15.189:0/2167954322 >> v2:172.21.15.47:3300/0 conn(0x7f7f7c1374a0 0x7f7f7c133a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:52.827 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.783+0000 7f7f8355a700 1 --2- 172.21.15.189:0/2167954322 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7f7c067b50 0x7f7f7c136fb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:52.827 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.783+0000 7f7f8355a700 1 -- 172.21.15.189:0/2167954322 >> 172.21.15.189:0/2167954322 conn(0x7f7f7c063210 msgr2=0x7f7f7c060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:52.827 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.783+0000 7f7f8355a700 1 -- 172.21.15.189:0/2167954322 shutdown_connections 2020-11-20T12:17:52.827 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.783+0000 7f7f8355a700 1 -- 172.21.15.189:0/2167954322 wait complete. 2020-11-20T12:17:52.828 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.783+0000 7f7f8355a700 1 Processor -- start 2020-11-20T12:17:52.828 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.783+0000 7f7f8355a700 1 -- start start 2020-11-20T12:17:52.828 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.783+0000 7f7f8355a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f7c0612b0 0x7f7f7c12c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:52.829 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.783+0000 7f7f8355a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f7c067b50 0x7f7f7c1397c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:52.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.783+0000 7f7f8355a700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7f7c1374a0 0x7f7f7c14d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:52.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.783+0000 7f7f8355a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7c0abf00 con 0x7f7f7c067b50 2020-11-20T12:17:52.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.783+0000 7f7f8355a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7c0ac1a0 con 0x7f7f7c0612b0 2020-11-20T12:17:52.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.783+0000 7f7f8355a700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7c0ac440 con 0x7f7f7c1374a0 2020-11-20T12:17:52.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.783+0000 7f7f82558700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f7c0612b0 0x7f7f7c12c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:52.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.783+0000 7f7f81d57700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f7c067b50 0x7f7f7c1397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:52.832 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.783+0000 7f7f82558700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f7c0612b0 0x7f7f7c12c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51396/0 (socket says 172.21.15.189:51396) 2020-11-20T12:17:52.832 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.783+0000 7f7f81d57700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f7c067b50 0x7f7f7c1397c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.189:57622/0 (socket says 172.21.15.189:57622) 2020-11-20T12:17:52.832 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.783+0000 7f7f82558700 1 -- 172.21.15.189:0/3055593485 learned_addr learned my addr 172.21.15.189:0/3055593485 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:17:52.832 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.783+0000 7f7f82d59700 1 --2- 172.21.15.189:0/3055593485 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7f7c1374a0 0x7f7f7c14d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:52.833 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.787+0000 7f7f81d57700 1 -- 172.21.15.189:0/3055593485 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7f7c1374a0 msgr2=0x7f7f7c14d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:52.833 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.787+0000 7f7f81d57700 1 --2- 172.21.15.189:0/3055593485 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7f7c1374a0 0x7f7f7c14d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:52.833 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.787+0000 7f7f81d57700 1 -- 172.21.15.189:0/3055593485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f7c0612b0 msgr2=0x7f7f7c12c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:52.833 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.787+0000 7f7f81d57700 1 --2- 172.21.15.189:0/3055593485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f7c0612b0 0x7f7f7c12c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:52.834 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.787+0000 7f7f81d57700 1 -- 172.21.15.189:0/3055593485 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f7c0ac6e0 con 0x7f7f7c067b50 2020-11-20T12:17:52.834 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.787+0000 7f7f81d57700 1 --2- 172.21.15.189:0/3055593485 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f7c067b50 0x7f7f7c1397c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f7f7800dcf0 tx=0x7f7f78018000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:52.834 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.787+0000 7f7f6a7fc700 1 -- 172.21.15.189:0/3055593485 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f78021070 con 0x7f7f7c067b50 2020-11-20T12:17:52.835 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.787+0000 7f7f6a7fc700 1 -- 172.21.15.189:0/3055593485 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f7800a040 con 0x7f7f7c067b50 2020-11-20T12:17:52.835 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.787+0000 7f7f6a7fc700 1 -- 172.21.15.189:0/3055593485 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f78009030 con 0x7f7f7c067b50 2020-11-20T12:17:52.835 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.787+0000 7f7f8355a700 1 -- 172.21.15.189:0/3055593485 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f7c0ab1f0 con 0x7f7f7c067b50 2020-11-20T12:17:52.835 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.787+0000 7f7f8355a700 1 -- 172.21.15.189:0/3055593485 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f7c132a10 con 0x7f7f7c067b50 2020-11-20T12:17:52.836 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.787+0000 7f7f8355a700 1 -- 172.21.15.189:0/3055593485 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f7c0ac870 con 0x7f7f7c067b50 2020-11-20T12:17:52.836 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.787+0000 7f7f6a7fc700 1 -- 172.21.15.189:0/3055593485 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f7f7800cba0 con 0x7f7f7c067b50 2020-11-20T12:17:52.836 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.787+0000 7f7f6a7fc700 1 --2- 172.21.15.189:0/3055593485 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f7f580557c0 0x7f7f58057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:52.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.787+0000 7f7f6a7fc700 1 -- 172.21.15.189:0/3055593485 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6933+0+0 (secure 0 0 0) 0x7f7f780700f0 con 0x7f7f7c067b50 2020-11-20T12:17:52.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.787+0000 7f7f82558700 1 --2- 172.21.15.189:0/3055593485 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f7f580557c0 0x7f7f58057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:52.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.787+0000 7f7f82558700 1 --2- 172.21.15.189:0/3055593485 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f7f580557c0 0x7f7f58057c80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f7f6c0011a0 tx=0x7f7f6c007ae0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:52.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.795+0000 7f7f6a7fc700 1 -- 172.21.15.189:0/3055593485 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7f78045cb0 con 0x7f7f7c067b50 2020-11-20T12:17:52.998 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.991+0000 7f7f8355a700 1 -- 172.21.15.189:0/3055593485 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f7c0446d0 con 0x7f7f7c067b50 2020-11-20T12:17:52.998 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.991+0000 7f7f6a7fc700 1 -- 172.21.15.189:0/3055593485 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+1784 (secure 0 0 0) 0x7f7f7802c020 con 0x7f7f7c067b50 2020-11-20T12:17:52.999 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:17:52.999 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:17:52.741331+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5572":{"gid":5572,"name":"a","rank":0,"incarnation":52,"state":"up:replay","state_seq":1,"addr":"172.21.15.189:6835/344330560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":344330560},{"type":"v1","addr":"172.21.15.189:6835","nonce":344330560}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:17:53.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.995+0000 7f7f8355a700 1 -- 172.21.15.189:0/3055593485 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f7f580557c0 msgr2=0x7f7f58057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:53.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.995+0000 7f7f8355a700 1 --2- 172.21.15.189:0/3055593485 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f7f580557c0 0x7f7f58057c80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f7f6c0011a0 tx=0x7f7f6c007ae0).stop 2020-11-20T12:17:53.003 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.995+0000 7f7f8355a700 1 -- 172.21.15.189:0/3055593485 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f7c067b50 msgr2=0x7f7f7c1397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:53.003 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.995+0000 7f7f8355a700 1 --2- 172.21.15.189:0/3055593485 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f7c067b50 0x7f7f7c1397c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f7f7800dcf0 tx=0x7f7f78018000).stop 2020-11-20T12:17:53.003 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.995+0000 7f7f8355a700 1 -- 172.21.15.189:0/3055593485 shutdown_connections 2020-11-20T12:17:53.003 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.995+0000 7f7f8355a700 1 --2- 172.21.15.189:0/3055593485 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f7f7c1374a0 0x7f7f7c14d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:53.003 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.995+0000 7f7f8355a700 1 --2- 172.21.15.189:0/3055593485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f7c0612b0 0x7f7f7c12c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:53.004 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.995+0000 7f7f8355a700 1 --2- 172.21.15.189:0/3055593485 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f7f580557c0 0x7f7f58057c80 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:53.004 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.995+0000 7f7f8355a700 1 --2- 172.21.15.189:0/3055593485 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f7f7c067b50 0x7f7f7c1397c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:53.004 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.995+0000 7f7f8355a700 1 -- 172.21.15.189:0/3055593485 >> 172.21.15.189:0/3055593485 conn(0x7f7f7c063210 msgr2=0x7f7f7c0adae0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:53.004 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.995+0000 7f7f8355a700 1 -- 172.21.15.189:0/3055593485 shutdown_connections 2020-11-20T12:17:53.004 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:52.995+0000 7f7f8355a700 1 -- 172.21.15.189:0/3055593485 wait complete. 2020-11-20T12:17:53.005 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 52 2020-11-20T12:17:53.014 INFO:tasks.mds_thrash.fs.[None]:mds.b down, removed from mdsmap 2020-11-20T12:17:53.014 INFO:tasks.mds_thrash.fs.[None]:waiting for mds cluster to stabilize... 2020-11-20T12:17:53.015 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:53.128 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.123+0000 7ff2a49d8700 1 Processor -- start 2020-11-20T12:17:53.129 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.123+0000 7ff2a49d8700 1 -- start start 2020-11-20T12:17:53.130 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.123+0000 7ff2a49d8700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff2a00ad000 0x7ff2a013e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:53.130 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.123+0000 7ff2a49d8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff2a013a6e0 0x7ff2a013ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:53.131 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.123+0000 7ff2a49d8700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a0067730 con 0x7ff2a00ad3e0 2020-11-20T12:17:53.131 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.123+0000 7ff2a49d8700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2a0138190 con 0x7ff2a00ad000 2020-11-20T12:17:53.131 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.123+0000 7ff2a49d8700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2a013dd40 con 0x7ff2a013a6e0 2020-11-20T12:17:53.131 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.123+0000 7ff29f59e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff2a013a6e0 0x7ff2a013ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:53.131 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.123+0000 7ff29f59e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff2a013a6e0 0x7ff2a013ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51406/0 (socket says 172.21.15.189:51406) 2020-11-20T12:17:53.132 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.123+0000 7ff29f59e700 1 -- 172.21.15.189:0/139862929 learned_addr learned my addr 172.21.15.189:0/139862929 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:17:53.132 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.123+0000 7ff29e59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff2a00ad3e0 0x7ff2a0066d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53030/0 (socket says 172.21.15.189:53030) 2020-11-20T12:17:53.132 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.123+0000 7ff29ed9d700 1 --2- 172.21.15.189:0/139862929 >> v2:172.21.15.47:3300/0 conn(0x7ff2a00ad000 0x7ff2a013e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:53.132 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.123+0000 7ff29f59e700 1 -- 172.21.15.189:0/139862929 >> v2:172.21.15.47:3300/0 conn(0x7ff2a00ad000 msgr2=0x7ff2a013e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:53.132 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.123+0000 7ff29f59e700 1 --2- 172.21.15.189:0/139862929 >> v2:172.21.15.47:3300/0 conn(0x7ff2a00ad000 0x7ff2a013e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:53.133 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.123+0000 7ff29f59e700 1 -- 172.21.15.189:0/139862929 >> v1:172.21.15.189:6789/0 conn(0x7ff2a00ad3e0 legacy=0x7ff2a0066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:53.133 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.123+0000 7ff29dd9b700 1 -- 172.21.15.189:0/139862929 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071440074 0 0) 0x7ff2a0067730 con 0x7ff2a00ad3e0 2020-11-20T12:17:53.133 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.123+0000 7ff29f59e700 1 -- 172.21.15.189:0/139862929 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a013dfe0 con 0x7ff2a013a6e0 2020-11-20T12:17:53.133 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.123+0000 7ff29f59e700 1 --2- 172.21.15.189:0/139862929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2a013a6e0 0x7ff2a013ad00 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7ff2a0068780 tx=0x7ff298006eb0).ready entity=mon.1 client_cookie=8f5597533b2f958f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:53.136 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.123+0000 7ff29dd9b700 1 -- 172.21.15.189:0/139862929 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff298009810 con 0x7ff2a013a6e0 2020-11-20T12:17:53.136 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.123+0000 7ff29dd9b700 1 -- 172.21.15.189:0/139862929 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff298009970 con 0x7ff2a013a6e0 2020-11-20T12:17:53.136 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.123+0000 7ff29dd9b700 1 -- 172.21.15.189:0/139862929 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff298009cf0 con 0x7ff2a013a6e0 2020-11-20T12:17:53.136 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff2a49d8700 1 -- 172.21.15.189:0/139862929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2a013a6e0 msgr2=0x7ff2a013ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:53.137 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff2a49d8700 1 --2- 172.21.15.189:0/139862929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2a013a6e0 0x7ff2a013ad00 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7ff2a0068780 tx=0x7ff298006eb0).stop 2020-11-20T12:17:53.137 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff2a49d8700 1 -- 172.21.15.189:0/139862929 shutdown_connections 2020-11-20T12:17:53.137 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff2a49d8700 1 --2- 172.21.15.189:0/139862929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2a013a6e0 0x7ff2a013ad00 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:53.137 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff2a49d8700 1 --2- 172.21.15.189:0/139862929 >> v2:172.21.15.47:3300/0 conn(0x7ff2a00ad000 0x7ff2a013e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:53.137 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff2a49d8700 1 -- 172.21.15.189:0/139862929 >> 172.21.15.189:0/139862929 conn(0x7ff2a0062080 msgr2=0x7ff2a0062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:53.138 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff2a49d8700 1 -- 172.21.15.189:0/139862929 shutdown_connections 2020-11-20T12:17:53.138 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff2a49d8700 1 -- 172.21.15.189:0/139862929 wait complete. 2020-11-20T12:17:53.138 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff2a49d8700 1 Processor -- start 2020-11-20T12:17:53.138 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff2a49d8700 1 -- start start 2020-11-20T12:17:53.138 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff2a49d8700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2a00ad000 0x7ff2a00a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:53.139 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff2a49d8700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff2a00ad3e0 0x7ff2a00a6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:53.139 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff2a49d8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2a00a41e0 0x7ff2a00a4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:53.139 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff2a49d8700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a00a6ce0 con 0x7ff2a00ad000 2020-11-20T12:17:53.139 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff2a49d8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a00a6f10 con 0x7ff2a00a41e0 2020-11-20T12:17:53.139 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff2a49d8700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a00a71b0 con 0x7ff2a00ad3e0 2020-11-20T12:17:53.140 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff29f59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2a00a41e0 0x7ff2a00a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:53.140 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff29f59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2a00a41e0 0x7ff2a00a4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51412/0 (socket says 172.21.15.189:51412) 2020-11-20T12:17:53.140 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff29f59e700 1 -- 172.21.15.189:0/1615250115 learned_addr learned my addr 172.21.15.189:0/1615250115 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:17:53.140 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff29ed9d700 1 --2- 172.21.15.189:0/1615250115 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2a00ad000 0x7ff2a00a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:53.140 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff29e59c700 1 --2- 172.21.15.189:0/1615250115 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff2a00ad3e0 0x7ff2a00a6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:53.141 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff29f59e700 1 -- 172.21.15.189:0/1615250115 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff2a00ad3e0 msgr2=0x7ff2a00a6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:53.141 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff29f59e700 1 --2- 172.21.15.189:0/1615250115 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff2a00ad3e0 0x7ff2a00a6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:53.141 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff29f59e700 1 -- 172.21.15.189:0/1615250115 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2a00ad000 msgr2=0x7ff2a00a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:53.141 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff29f59e700 1 --2- 172.21.15.189:0/1615250115 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2a00ad000 0x7ff2a00a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:53.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff29f59e700 1 -- 172.21.15.189:0/1615250115 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a00a7450 con 0x7ff2a00a41e0 2020-11-20T12:17:53.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff29f59e700 1 --2- 172.21.15.189:0/1615250115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2a00a41e0 0x7ff2a00a4620 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7ff298006ef0 tx=0x7ff298015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:53.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.127+0000 7ff286ffd700 1 -- 172.21.15.189:0/1615250115 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2980093a0 con 0x7ff2a00a41e0 2020-11-20T12:17:53.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.131+0000 7ff286ffd700 1 -- 172.21.15.189:0/1615250115 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff29801d5d0 con 0x7ff2a00a41e0 2020-11-20T12:17:53.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.131+0000 7ff286ffd700 1 -- 172.21.15.189:0/1615250115 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29801d950 con 0x7ff2a00a41e0 2020-11-20T12:17:53.143 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.131+0000 7ff2a49d8700 1 -- 172.21.15.189:0/1615250115 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2a00a5050 con 0x7ff2a00a41e0 2020-11-20T12:17:53.143 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.131+0000 7ff2a49d8700 1 -- 172.21.15.189:0/1615250115 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2a01493e0 con 0x7ff2a00a41e0 2020-11-20T12:17:53.144 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.131+0000 7ff2a49d8700 1 -- 172.21.15.189:0/1615250115 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2a00a6820 con 0x7ff2a00a41e0 2020-11-20T12:17:53.144 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.131+0000 7ff286ffd700 1 -- 172.21.15.189:0/1615250115 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7ff29801a040 con 0x7ff2a00a41e0 2020-11-20T12:17:53.144 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.131+0000 7ff286ffd700 1 --2- 172.21.15.189:0/1615250115 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff27c0557c0 0x7ff27c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:53.144 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.131+0000 7ff29ed9d700 1 --2- 172.21.15.189:0/1615250115 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff27c0557c0 0x7ff27c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:53.145 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.131+0000 7ff286ffd700 1 -- 172.21.15.189:0/1615250115 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6933+0+0 (secure 0 0 0) 0x7ff298070110 con 0x7ff2a00a41e0 2020-11-20T12:17:53.145 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.131+0000 7ff29ed9d700 1 --2- 172.21.15.189:0/1615250115 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff27c0557c0 0x7ff27c057c80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7ff290009d90 tx=0x7ff290008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:53.145 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.139+0000 7ff286ffd700 1 -- 172.21.15.189:0/1615250115 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff298045b90 con 0x7ff2a00a41e0 2020-11-20T12:17:53.349 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.339+0000 7ff2a49d8700 1 -- 172.21.15.189:0/1615250115 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2a00446d0 con 0x7ff2a00a41e0 2020-11-20T12:17:53.350 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.339+0000 7ff286ffd700 1 -- 172.21.15.189:0/1615250115 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+1784 (secure 0 0 0) 0x7ff298023020 con 0x7ff2a00a41e0 2020-11-20T12:17:53.350 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.343+0000 7ff2a49d8700 1 -- 172.21.15.189:0/1615250115 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff27c0557c0 msgr2=0x7ff27c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:53.351 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.343+0000 7ff2a49d8700 1 --2- 172.21.15.189:0/1615250115 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff27c0557c0 0x7ff27c057c80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7ff290009d90 tx=0x7ff290008040).stop 2020-11-20T12:17:53.351 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.343+0000 7ff2a49d8700 1 -- 172.21.15.189:0/1615250115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2a00a41e0 msgr2=0x7ff2a00a4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:53.351 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.343+0000 7ff2a49d8700 1 --2- 172.21.15.189:0/1615250115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2a00a41e0 0x7ff2a00a4620 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7ff298006ef0 tx=0x7ff298015000).stop 2020-11-20T12:17:53.352 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.343+0000 7ff2a49d8700 1 -- 172.21.15.189:0/1615250115 shutdown_connections 2020-11-20T12:17:53.352 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.343+0000 7ff2a49d8700 1 --2- 172.21.15.189:0/1615250115 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff27c0557c0 0x7ff27c057c80 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:53.352 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.343+0000 7ff2a49d8700 1 --2- 172.21.15.189:0/1615250115 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff2a00ad000 0x7ff2a00a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:53.353 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.343+0000 7ff2a49d8700 1 --2- 172.21.15.189:0/1615250115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2a00a41e0 0x7ff2a00a4620 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:53.353 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.343+0000 7ff2a49d8700 1 --2- 172.21.15.189:0/1615250115 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff2a00ad3e0 0x7ff2a00a6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:53.353 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.343+0000 7ff2a49d8700 1 -- 172.21.15.189:0/1615250115 >> 172.21.15.189:0/1615250115 conn(0x7ff2a0062080 msgr2=0x7ff2a00ad860 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:53.354 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.343+0000 7ff2a49d8700 1 -- 172.21.15.189:0/1615250115 shutdown_connections 2020-11-20T12:17:53.354 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:53.343+0000 7ff2a49d8700 1 -- 172.21.15.189:0/1615250115 wait complete. 2020-11-20T12:17:53.355 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:17:53.355 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:17:52.741331+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5572":{"gid":5572,"name":"a","rank":0,"incarnation":52,"state":"up:replay","state_seq":1,"addr":"172.21.15.189:6835/344330560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":344330560},{"type":"v1","addr":"172.21.15.189:6835","nonce":344330560}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:17:53.356 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 52 2020-11-20T12:17:53.365 INFO:tasks.mds_thrash.fs.[None]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 52, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2020-11-20T12:04:54.012110+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 52, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5572": { "addr": "172.21.15.189:6835/344330560", "addrs": { "addrvec": [ { "addr": "172.21.15.189:6834", "nonce": 344330560, "type": "v2" }, { "addr": "172.21.15.189:6835", "nonce": 344330560, "type": "v1" } ] }, "export_targets": [], "features": 4540138297136906239, "flags": 0, "gid": 5572, "incarnation": 52, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 32, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2020-11-20T12:17:52.741331+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5572 } } } ], "standbys": [] } 2020-11-20T12:17:55.367 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:55.377 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:17:55.379 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:17:55.498 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.490+0000 7f976f21d700 1 Processor -- start 2020-11-20T12:17:55.498 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.490+0000 7f976f21d700 1 -- start start 2020-11-20T12:17:55.502 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.490+0000 7f976f21d700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f97680ad000 0x7f976813e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:55.502 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.490+0000 7f976f21d700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9768067730 con 0x7f976813a6e0 2020-11-20T12:17:55.502 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.490+0000 7f976f21d700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f976813dd40 con 0x7f97680ad3e0 2020-11-20T12:17:55.503 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.490+0000 7f976f21d700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9768138190 con 0x7f97680ad000 2020-11-20T12:17:55.503 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.490+0000 7f976da1a700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f97680ad3e0 0x7f9768066d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53044/0 (socket says 172.21.15.189:53044) 2020-11-20T12:17:55.503 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.490+0000 7f976da1a700 1 -- 172.21.15.189:0/1590776380 learned_addr learned my addr 172.21.15.189:0/1590776380 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:17:55.503 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.490+0000 7f976e21b700 1 --2- 172.21.15.189:0/1590776380 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f97680ad000 0x7f976813e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:55.503 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976d219700 1 -- 172.21.15.189:0/1590776380 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820757765 0 0) 0x7f976813dd40 con 0x7f97680ad3e0 2020-11-20T12:17:55.504 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976d219700 1 -- 172.21.15.189:0/1590776380 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9750003240 con 0x7f97680ad3e0 2020-11-20T12:17:55.504 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976d219700 1 -- 172.21.15.189:0/1590776380 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626372484 0 0) 0x7f9768067730 con 0x7f976813a6e0 2020-11-20T12:17:55.504 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976d219700 1 -- 172.21.15.189:0/1590776380 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f976813dd40 con 0x7f976813a6e0 2020-11-20T12:17:55.504 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976d219700 1 -- 172.21.15.189:0/1590776380 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3965575119 0 0) 0x7f9750003240 con 0x7f97680ad3e0 2020-11-20T12:17:55.504 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976d219700 1 -- 172.21.15.189:0/1590776380 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f97680ad000 msgr2=0x7f976813e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:55.505 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976d219700 1 --2- 172.21.15.189:0/1590776380 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f97680ad000 0x7f976813e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:55.505 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976d219700 1 -- 172.21.15.189:0/1590776380 >> v1:172.21.15.47:6789/0 conn(0x7f976813a6e0 legacy=0x7f976813ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:55.505 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976d219700 1 -- 172.21.15.189:0/1590776380 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f976813dfe0 con 0x7f97680ad3e0 2020-11-20T12:17:55.505 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976d219700 1 -- 172.21.15.189:0/1590776380 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f9754001860 con 0x7f97680ad3e0 2020-11-20T12:17:55.505 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976d219700 1 -- 172.21.15.189:0/1590776380 >> v1:172.21.15.189:6789/0 conn(0x7f97680ad3e0 legacy=0x7f9768066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:55.506 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976d219700 1 --2- 172.21.15.189:0/1590776380 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f97680ad000 0x7f97500038c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:55.506 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976d219700 1 --2- 172.21.15.189:0/1590776380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9750003db0 0x7f97500044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:55.506 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976d219700 1 --2- 172.21.15.189:0/1590776380 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97500049d0 0x7f9750004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:55.506 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976d219700 1 -- 172.21.15.189:0/1590776380 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9768138190 con 0x7f97500049d0 2020-11-20T12:17:55.506 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976d219700 1 -- 172.21.15.189:0/1590776380 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9750005340 con 0x7f9750003db0 2020-11-20T12:17:55.507 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976d219700 1 -- 172.21.15.189:0/1590776380 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f975000b760 con 0x7f97680ad000 2020-11-20T12:17:55.507 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976da1a700 1 --2- 172.21.15.189:0/1590776380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9750003db0 0x7f97500044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:55.507 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976e21b700 1 --2- 172.21.15.189:0/1590776380 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f97680ad000 0x7f97500038c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:55.507 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976ea1c700 1 --2- 172.21.15.189:0/1590776380 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97500049d0 0x7f9750004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:55.507 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976da1a700 1 -- 172.21.15.189:0/1590776380 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f97680ad000 msgr2=0x7f97500038c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:55.508 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976da1a700 1 --2- 172.21.15.189:0/1590776380 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f97680ad000 0x7f97500038c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:55.508 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976da1a700 1 -- 172.21.15.189:0/1590776380 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97500049d0 msgr2=0x7f9750004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:17:55.508 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976da1a700 1 --2- 172.21.15.189:0/1590776380 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97500049d0 0x7f9750004e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:55.508 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976da1a700 1 -- 172.21.15.189:0/1590776380 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9750003240 con 0x7f9750003db0 2020-11-20T12:17:55.508 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976da1a700 1 --2- 172.21.15.189:0/1590776380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9750003db0 0x7f97500044e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7f9768068950 tx=0x7f976813e170).ready entity=mon.1 client_cookie=c56c596e8318c070 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:55.509 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976d219700 1 -- 172.21.15.189:0/1590776380 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9754005b80 con 0x7f9750003db0 2020-11-20T12:17:55.509 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976d219700 1 -- 172.21.15.189:0/1590776380 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9754005ce0 con 0x7f9750003db0 2020-11-20T12:17:55.509 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976d219700 1 -- 172.21.15.189:0/1590776380 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97540072e0 con 0x7f9750003db0 2020-11-20T12:17:55.509 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976f21d700 1 -- 172.21.15.189:0/1590776380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9750003db0 msgr2=0x7f97500044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:55.510 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976f21d700 1 --2- 172.21.15.189:0/1590776380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9750003db0 0x7f97500044e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7f9768068950 tx=0x7f976813e170).stop 2020-11-20T12:17:55.510 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976ea1c700 1 -- 172.21.15.189:0/1590776380 reap_dead start 2020-11-20T12:17:55.510 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976f21d700 1 -- 172.21.15.189:0/1590776380 shutdown_connections 2020-11-20T12:17:55.510 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976f21d700 1 -- 172.21.15.189:0/1590776380 >> 172.21.15.189:0/1590776380 conn(0x7f9768062080 msgr2=0x7f9768062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:55.510 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976f21d700 1 -- 172.21.15.189:0/1590776380 shutdown_connections 2020-11-20T12:17:55.510 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976f21d700 1 -- 172.21.15.189:0/1590776380 wait complete. 2020-11-20T12:17:55.513 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976f21d700 1 Processor -- start 2020-11-20T12:17:55.513 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.494+0000 7f976f21d700 1 -- start start 2020-11-20T12:17:55.513 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.498+0000 7f976f21d700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97680a2770 0x7f97680a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:55.514 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.498+0000 7f976f21d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97680a2b50 0x7f9768149cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:55.514 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.498+0000 7f976f21d700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f976814a1a0 0x7f97680a5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:55.514 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.498+0000 7f976f21d700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97680a1c50 con 0x7f97680a2770 2020-11-20T12:17:55.514 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.498+0000 7f976f21d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97680a7cd0 con 0x7f97680a2b50 2020-11-20T12:17:55.514 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.498+0000 7f976f21d700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97680a72f0 con 0x7f976814a1a0 2020-11-20T12:17:55.515 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.498+0000 7f976da1a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97680a2b50 0x7f9768149cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:55.515 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.498+0000 7f976da1a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97680a2b50 0x7f9768149cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51434/0 (socket says 172.21.15.189:51434) 2020-11-20T12:17:55.515 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.498+0000 7f976da1a700 1 -- 172.21.15.189:0/4044426059 learned_addr learned my addr 172.21.15.189:0/4044426059 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:17:55.515 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.498+0000 7f976e21b700 1 --2- 172.21.15.189:0/4044426059 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97680a2770 0x7f97680a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:55.515 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.498+0000 7f976ea1c700 1 --2- 172.21.15.189:0/4044426059 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f976814a1a0 0x7f97680a5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:55.516 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.498+0000 7f976da1a700 1 -- 172.21.15.189:0/4044426059 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f976814a1a0 msgr2=0x7f97680a5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:55.516 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.498+0000 7f976da1a700 1 --2- 172.21.15.189:0/4044426059 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f976814a1a0 0x7f97680a5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:55.516 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.498+0000 7f976da1a700 1 -- 172.21.15.189:0/4044426059 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97680a2770 msgr2=0x7f97680a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:55.516 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.498+0000 7f976da1a700 1 --2- 172.21.15.189:0/4044426059 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97680a2770 0x7f97680a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:55.516 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.498+0000 7f976da1a700 1 -- 172.21.15.189:0/4044426059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97680a7590 con 0x7f97680a2b50 2020-11-20T12:17:55.517 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.498+0000 7f976ea1c700 1 --2- 172.21.15.189:0/4044426059 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f976814a1a0 0x7f97680a5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:17:55.517 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.498+0000 7f976da1a700 1 --2- 172.21.15.189:0/4044426059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97680a2b50 0x7f9768149cb0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7f975000b730 tx=0x7f9754010000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:55.517 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.498+0000 7f97667fc700 1 -- 172.21.15.189:0/4044426059 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97540072e0 con 0x7f97680a2b50 2020-11-20T12:17:55.517 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.498+0000 7f97667fc700 1 -- 172.21.15.189:0/4044426059 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9754001320 con 0x7f97680a2b50 2020-11-20T12:17:55.518 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.498+0000 7f976f21d700 1 -- 172.21.15.189:0/4044426059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97680a6f70 con 0x7f97680a2b50 2020-11-20T12:17:55.518 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.498+0000 7f97667fc700 1 -- 172.21.15.189:0/4044426059 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9754007860 con 0x7f97680a2b50 2020-11-20T12:17:55.518 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.498+0000 7f976f21d700 1 -- 172.21.15.189:0/4044426059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97680a6360 con 0x7f97680a2b50 2020-11-20T12:17:55.518 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.498+0000 7f976f21d700 1 -- 172.21.15.189:0/4044426059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97680a7160 con 0x7f97680a2b50 2020-11-20T12:17:55.518 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.502+0000 7f97667fc700 1 -- 172.21.15.189:0/4044426059 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f975401c070 con 0x7f97680a2b50 2020-11-20T12:17:55.519 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.502+0000 7f97667fc700 1 --2- 172.21.15.189:0/4044426059 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f973c0557c0 0x7f973c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:55.519 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.502+0000 7f976e21b700 1 --2- 172.21.15.189:0/4044426059 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f973c0557c0 0x7f973c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:55.519 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.502+0000 7f97667fc700 1 -- 172.21.15.189:0/4044426059 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6933+0+0 (secure 0 0 0) 0x7f975406fd40 con 0x7f97680a2b50 2020-11-20T12:17:55.519 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.502+0000 7f976e21b700 1 --2- 172.21.15.189:0/4044426059 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f973c0557c0 0x7f973c057c80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f9760006fd0 tx=0x7f9760008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:55.520 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.506+0000 7f97667fc700 1 -- 172.21.15.189:0/4044426059 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f97540457b0 con 0x7f97680a2b50 2020-11-20T12:17:55.714 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.706+0000 7f976f21d700 1 -- 172.21.15.189:0/4044426059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f976813b8c0 con 0x7f97680a2b50 2020-11-20T12:17:55.714 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.706+0000 7f97667fc700 1 -- 172.21.15.189:0/4044426059 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+1785 (secure 0 0 0) 0x7f976813b8c0 con 0x7f97680a2b50 2020-11-20T12:17:55.715 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:17:55.716 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:17:54.760842+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5572":{"gid":5572,"name":"a","rank":0,"incarnation":52,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.189:6835/344330560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":344330560},{"type":"v1","addr":"172.21.15.189:6835","nonce":344330560}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:17:55.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.710+0000 7f976f21d700 1 -- 172.21.15.189:0/4044426059 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f973c0557c0 msgr2=0x7f973c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:55.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.710+0000 7f976f21d700 1 --2- 172.21.15.189:0/4044426059 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f973c0557c0 0x7f973c057c80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f9760006fd0 tx=0x7f9760008040).stop 2020-11-20T12:17:55.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.710+0000 7f976f21d700 1 -- 172.21.15.189:0/4044426059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97680a2b50 msgr2=0x7f9768149cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:55.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.710+0000 7f976f21d700 1 --2- 172.21.15.189:0/4044426059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97680a2b50 0x7f9768149cb0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7f975000b730 tx=0x7f9754010000).stop 2020-11-20T12:17:55.717 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.710+0000 7f976f21d700 1 -- 172.21.15.189:0/4044426059 shutdown_connections 2020-11-20T12:17:55.718 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.710+0000 7f976f21d700 1 --2- 172.21.15.189:0/4044426059 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f973c0557c0 0x7f973c057c80 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:55.718 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.710+0000 7f976f21d700 1 --2- 172.21.15.189:0/4044426059 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f97680a2770 0x7f97680a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:55.718 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.710+0000 7f976f21d700 1 --2- 172.21.15.189:0/4044426059 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f976814a1a0 0x7f97680a5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:55.718 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.710+0000 7f976f21d700 1 --2- 172.21.15.189:0/4044426059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97680a2b50 0x7f9768149cb0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:55.718 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.710+0000 7f976f21d700 1 -- 172.21.15.189:0/4044426059 >> 172.21.15.189:0/4044426059 conn(0x7f9768062080 msgr2=0x7f976813aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:55.719 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.710+0000 7f976f21d700 1 -- 172.21.15.189:0/4044426059 shutdown_connections 2020-11-20T12:17:55.719 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:55.710+0000 7f976f21d700 1 -- 172.21.15.189:0/4044426059 wait complete. 2020-11-20T12:17:55.719 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 54 2020-11-20T12:17:55.735 INFO:tasks.mds_thrash.fs.[None]:no change 2020-11-20T12:17:57.736 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:17:57.878 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.870+0000 7fc5415cd700 1 Processor -- start 2020-11-20T12:17:57.878 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.870+0000 7fc5415cd700 1 -- start start 2020-11-20T12:17:57.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.874+0000 7fc5415cd700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fc53c0612b0 0x7fc53c0a0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:57.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.874+0000 7fc5415cd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc53c1376c0 0x7fc53c133a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:57.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.874+0000 7fc5415cd700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc53c068020 con 0x7fc53c067b50 2020-11-20T12:17:57.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.874+0000 7fc5415cd700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc53c0682c0 con 0x7fc53c0612b0 2020-11-20T12:17:57.880 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.874+0000 7fc5415cd700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc53c061690 con 0x7fc53c1376c0 2020-11-20T12:17:57.881 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.874+0000 7fc53b7fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc53c067b50 0x7fc53c136fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53066/0 (socket says 172.21.15.189:53066) 2020-11-20T12:17:57.881 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.874+0000 7fc53b7fe700 1 -- 172.21.15.189:0/299018076 learned_addr learned my addr 172.21.15.189:0/299018076 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:17:57.882 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.874+0000 7fc540dcc700 1 --2- 172.21.15.189:0/299018076 >> v2:172.21.15.189:3300/0 conn(0x7fc53c1376c0 0x7fc53c133a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:57.883 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.874+0000 7fc53affd700 1 -- 172.21.15.189:0/299018076 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3109490364 0 0) 0x7fc53c068020 con 0x7fc53c067b50 2020-11-20T12:17:57.884 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.874+0000 7fc53affd700 1 -- 172.21.15.189:0/299018076 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc528003240 con 0x7fc53c067b50 2020-11-20T12:17:57.884 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.874+0000 7fc540dcc700 1 -- 172.21.15.189:0/299018076 >> v2:172.21.15.47:3300/0 conn(0x7fc53c0612b0 msgr2=0x7fc53c0a0960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2020-11-20T12:17:57.885 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.874+0000 7fc540dcc700 1 --2- 172.21.15.189:0/299018076 >> v2:172.21.15.47:3300/0 conn(0x7fc53c0612b0 0x7fc53c0a0960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:57.885 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.874+0000 7fc540dcc700 1 -- 172.21.15.189:0/299018076 >> v1:172.21.15.189:6789/0 conn(0x7fc53c067b50 legacy=0x7fc53c136fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:57.885 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.874+0000 7fc540dcc700 1 -- 172.21.15.189:0/299018076 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc53c136aa0 con 0x7fc53c1376c0 2020-11-20T12:17:57.886 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.874+0000 7fc540dcc700 1 --2- 172.21.15.189:0/299018076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc53c1376c0 0x7fc53c133a10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7fc53c0ad260 tx=0x7fc530006df0).ready entity=mon.1 client_cookie=a490792dffc8474a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:57.886 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.874+0000 7fc53affd700 1 -- 172.21.15.189:0/299018076 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5300097b0 con 0x7fc53c1376c0 2020-11-20T12:17:57.887 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.874+0000 7fc53affd700 1 -- 172.21.15.189:0/299018076 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc530009910 con 0x7fc53c1376c0 2020-11-20T12:17:57.887 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.874+0000 7fc53affd700 1 -- 172.21.15.189:0/299018076 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc530009c90 con 0x7fc53c1376c0 2020-11-20T12:17:57.887 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.874+0000 7fc5415cd700 1 -- 172.21.15.189:0/299018076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc53c1376c0 msgr2=0x7fc53c133a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:57.888 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.874+0000 7fc5415cd700 1 --2- 172.21.15.189:0/299018076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc53c1376c0 0x7fc53c133a10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7fc53c0ad260 tx=0x7fc530006df0).stop 2020-11-20T12:17:57.888 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.874+0000 7fc5415cd700 1 -- 172.21.15.189:0/299018076 shutdown_connections 2020-11-20T12:17:57.888 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.874+0000 7fc5415cd700 1 --2- 172.21.15.189:0/299018076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc53c1376c0 0x7fc53c133a10 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:57.889 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.874+0000 7fc5415cd700 1 --2- 172.21.15.189:0/299018076 >> v2:172.21.15.47:3300/0 conn(0x7fc53c0612b0 0x7fc53c0a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:17:57.889 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.874+0000 7fc5415cd700 1 -- 172.21.15.189:0/299018076 >> 172.21.15.189:0/299018076 conn(0x7fc53c063210 msgr2=0x7fc53c060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:57.889 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc5415cd700 1 -- 172.21.15.189:0/299018076 shutdown_connections 2020-11-20T12:17:57.890 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc5415cd700 1 -- 172.21.15.189:0/299018076 wait complete. 2020-11-20T12:17:57.894 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc5415cd700 1 Processor -- start 2020-11-20T12:17:57.895 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc5415cd700 1 -- start start 2020-11-20T12:17:57.895 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc5415cd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc53c0612b0 0x7fc53c132c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:57.895 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc5415cd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc53c067b50 0x7fc53c0a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:57.896 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc5415cd700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc53c1376c0 0x7fc53c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:57.896 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc5415cd700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c0a5320 con 0x7fc53c067b50 2020-11-20T12:17:57.896 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc5415cd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c0a5930 con 0x7fc53c0612b0 2020-11-20T12:17:57.897 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc5415cd700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c0a5bd0 con 0x7fc53c1376c0 2020-11-20T12:17:57.897 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc53bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc53c0612b0 0x7fc53c132c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:57.898 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc53bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc53c0612b0 0x7fc53c132c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51448/0 (socket says 172.21.15.189:51448) 2020-11-20T12:17:57.898 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc53bfff700 1 -- 172.21.15.189:0/4085616882 learned_addr learned my addr 172.21.15.189:0/4085616882 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:17:57.898 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc53b7fe700 1 --2- 172.21.15.189:0/4085616882 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc53c067b50 0x7fc53c0a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:57.899 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc540dcc700 1 --2- 172.21.15.189:0/4085616882 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc53c1376c0 0x7fc53c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:57.899 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc53bfff700 1 -- 172.21.15.189:0/4085616882 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc53c1376c0 msgr2=0x7fc53c0a4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:57.899 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc53bfff700 1 --2- 172.21.15.189:0/4085616882 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc53c1376c0 0x7fc53c0a4e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:57.900 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc53bfff700 1 -- 172.21.15.189:0/4085616882 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc53c067b50 msgr2=0x7fc53c0a4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:57.900 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc53bfff700 1 --2- 172.21.15.189:0/4085616882 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc53c067b50 0x7fc53c0a4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:57.900 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc53bfff700 1 -- 172.21.15.189:0/4085616882 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc53c0a55b0 con 0x7fc53c0612b0 2020-11-20T12:17:57.901 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc53b7fe700 1 --2- 172.21.15.189:0/4085616882 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc53c067b50 0x7fc53c0a4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:17:57.901 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc53bfff700 1 --2- 172.21.15.189:0/4085616882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc53c0612b0 0x7fc53c132c10 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7fc52c009710 tx=0x7fc52c006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:57.902 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc51bfff700 1 -- 172.21.15.189:0/4085616882 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c00d650 con 0x7fc53c0612b0 2020-11-20T12:17:57.902 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc51bfff700 1 -- 172.21.15.189:0/4085616882 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc52c00d7b0 con 0x7fc53c0612b0 2020-11-20T12:17:57.902 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc51bfff700 1 -- 172.21.15.189:0/4085616882 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c00db70 con 0x7fc53c0612b0 2020-11-20T12:17:57.903 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc5415cd700 1 -- 172.21.15.189:0/4085616882 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc53c0a10f0 con 0x7fc53c0612b0 2020-11-20T12:17:57.903 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.878+0000 7fc5415cd700 1 -- 172.21.15.189:0/4085616882 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc53c0acf30 con 0x7fc53c0612b0 2020-11-20T12:17:57.903 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.882+0000 7fc5415cd700 1 -- 172.21.15.189:0/4085616882 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc53c0a57a0 con 0x7fc53c0612b0 2020-11-20T12:17:57.904 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.882+0000 7fc51bfff700 1 -- 172.21.15.189:0/4085616882 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fc52c017030 con 0x7fc53c0612b0 2020-11-20T12:17:57.904 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.882+0000 7fc51bfff700 1 --2- 172.21.15.189:0/4085616882 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc5140557c0 0x7fc514057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:17:57.905 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.882+0000 7fc51bfff700 1 -- 172.21.15.189:0/4085616882 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6933+0+0 (secure 0 0 0) 0x7fc52c06dc80 con 0x7fc53c0612b0 2020-11-20T12:17:57.905 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.882+0000 7fc53b7fe700 1 --2- 172.21.15.189:0/4085616882 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc5140557c0 0x7fc514057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:17:57.905 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.882+0000 7fc53b7fe700 1 --2- 172.21.15.189:0/4085616882 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc5140557c0 0x7fc514057c80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fc534005920 tx=0x7fc534009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:17:57.906 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:57.886+0000 7fc51bfff700 1 -- 172.21.15.189:0/4085616882 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc52c043870 con 0x7fc53c0612b0 2020-11-20T12:17:58.135 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:58.130+0000 7fc5415cd700 1 -- 172.21.15.189:0/4085616882 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc53c0a1280 con 0x7fc53c0612b0 2020-11-20T12:17:58.136 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:58.130+0000 7fc51bfff700 1 -- 172.21.15.189:0/4085616882 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+1785 (secure 0 0 0) 0x7fc53c0a1280 con 0x7fc53c0612b0 2020-11-20T12:17:58.136 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:17:58.136 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:17:55.763199+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5572":{"gid":5572,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/344330560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":344330560},{"type":"v1","addr":"172.21.15.189:6835","nonce":344330560}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:17:58.137 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:58.130+0000 7fc5415cd700 1 -- 172.21.15.189:0/4085616882 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc5140557c0 msgr2=0x7fc514057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:58.138 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:58.130+0000 7fc5415cd700 1 --2- 172.21.15.189:0/4085616882 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc5140557c0 0x7fc514057c80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fc534005920 tx=0x7fc534009040).stop 2020-11-20T12:17:58.138 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:58.130+0000 7fc5415cd700 1 -- 172.21.15.189:0/4085616882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc53c0612b0 msgr2=0x7fc53c132c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:17:58.138 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:58.130+0000 7fc5415cd700 1 --2- 172.21.15.189:0/4085616882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc53c0612b0 0x7fc53c132c10 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7fc52c009710 tx=0x7fc52c006df0).stop 2020-11-20T12:17:58.138 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:58.130+0000 7fc5415cd700 1 -- 172.21.15.189:0/4085616882 shutdown_connections 2020-11-20T12:17:58.139 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:58.130+0000 7fc5415cd700 1 --2- 172.21.15.189:0/4085616882 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fc53c1376c0 0x7fc53c0a4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:58.139 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:58.130+0000 7fc5415cd700 1 --2- 172.21.15.189:0/4085616882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc53c0612b0 0x7fc53c132c10 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:58.139 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:58.130+0000 7fc5415cd700 1 --2- 172.21.15.189:0/4085616882 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fc5140557c0 0x7fc514057c80 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:58.139 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:58.130+0000 7fc5415cd700 1 --2- 172.21.15.189:0/4085616882 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fc53c067b50 0x7fc53c0a4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:17:58.139 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:58.130+0000 7fc5415cd700 1 -- 172.21.15.189:0/4085616882 >> 172.21.15.189:0/4085616882 conn(0x7fc53c063210 msgr2=0x7fc53c0ad2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:17:58.140 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:58.130+0000 7fc5415cd700 1 -- 172.21.15.189:0/4085616882 shutdown_connections 2020-11-20T12:17:58.140 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:17:58.130+0000 7fc5415cd700 1 -- 172.21.15.189:0/4085616882 wait complete. 2020-11-20T12:17:58.140 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 55 2020-11-20T12:17:58.150 INFO:tasks.mds_thrash.fs.[None]:mds.a has gained rank=0, replacing gid=5075 2020-11-20T12:17:58.150 INFO:tasks.mds_thrash.fs.[None]:waiting for 9 secs before reviving mds.b 2020-11-20T12:18:00.647 INFO:tasks.ceph.mds.a.smithi189.stderr:2020-11-20T12:18:00.638+0000 7f8000fce700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:18:07.153 INFO:tasks.mds_thrash.fs.[None]:reviving mds.b 2020-11-20T12:18:07.153 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:18:07.154 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:18:07.174 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:18:07.176 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:18:07.248 INFO:tasks.ceph.mds.b.smithi047.stdout:starting mds.b at 2020-11-20T12:18:07.343 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.314+0000 7fde17fff700 1 Processor -- start 2020-11-20T12:18:07.343 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.314+0000 7fde17fff700 1 -- start start 2020-11-20T12:18:07.345 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.318+0000 7fde17fff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fde180aebd0 0x7fde180aa620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:07.345 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.318+0000 7fde17fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fde180aefb0 0x7fde180a9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:07.346 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.318+0000 7fde17fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde180a9680 con 0x7fde180a9fd0 2020-11-20T12:18:07.346 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.318+0000 7fde17fff700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde180aae50 con 0x7fde180aefb0 2020-11-20T12:18:07.346 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.318+0000 7fde17fff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1813ba10 con 0x7fde180aebd0 2020-11-20T12:18:07.346 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.318+0000 7fde0e7fc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fde180aefb0 0x7fde180a9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:07.347 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.318+0000 7fde177fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fde180a9fd0 0x7fde1813be70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53082/0 (socket says 172.21.15.189:53082) 2020-11-20T12:18:07.347 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.318+0000 7fde177fe700 1 -- 172.21.15.189:0/3623900971 learned_addr learned my addr 172.21.15.189:0/3623900971 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:18:07.347 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.318+0000 7fde16ffd700 1 --2- 172.21.15.189:0/3623900971 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fde180aebd0 0x7fde180aa620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:07.347 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.318+0000 7fde0e7fc700 1 -- 172.21.15.189:0/3623900971 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fde180aebd0 msgr2=0x7fde180aa620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:07.347 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.318+0000 7fde0e7fc700 1 --2- 172.21.15.189:0/3623900971 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fde180aebd0 0x7fde180aa620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:07.348 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.318+0000 7fde0e7fc700 1 -- 172.21.15.189:0/3623900971 >> v1:172.21.15.189:6789/0 conn(0x7fde180a9fd0 legacy=0x7fde1813be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:07.348 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.318+0000 7fde167fc700 1 -- 172.21.15.189:0/3623900971 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3682880098 0 0) 0x7fde180a9680 con 0x7fde180a9fd0 2020-11-20T12:18:07.348 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.318+0000 7fde0e7fc700 1 -- 172.21.15.189:0/3623900971 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde180aab10 con 0x7fde180aefb0 2020-11-20T12:18:07.348 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.318+0000 7fde0e7fc700 1 --2- 172.21.15.189:0/3623900971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde180aefb0 0x7fde180a9ae0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7fde0000b9a0 tx=0x7fde00006df0).ready entity=mon.1 client_cookie=d52a0ff9a458b5c3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:07.348 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.318+0000 7fde167fc700 1 -- 172.21.15.189:0/3623900971 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde000097b0 con 0x7fde180aefb0 2020-11-20T12:18:07.349 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.318+0000 7fde167fc700 1 -- 172.21.15.189:0/3623900971 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde00009910 con 0x7fde180aefb0 2020-11-20T12:18:07.349 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.318+0000 7fde167fc700 1 -- 172.21.15.189:0/3623900971 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde00009c90 con 0x7fde180aefb0 2020-11-20T12:18:07.349 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.318+0000 7fde17fff700 1 -- 172.21.15.189:0/3623900971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde180aefb0 msgr2=0x7fde180a9ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:07.350 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.318+0000 7fde17fff700 1 --2- 172.21.15.189:0/3623900971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde180aefb0 0x7fde180a9ae0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7fde0000b9a0 tx=0x7fde00006df0).stop 2020-11-20T12:18:07.350 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.318+0000 7fde17fff700 1 -- 172.21.15.189:0/3623900971 shutdown_connections 2020-11-20T12:18:07.350 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.318+0000 7fde17fff700 1 --2- 172.21.15.189:0/3623900971 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fde180aebd0 0x7fde180aa620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:07.350 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.318+0000 7fde17fff700 1 --2- 172.21.15.189:0/3623900971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde180aefb0 0x7fde180a9ae0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:07.350 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.318+0000 7fde17fff700 1 -- 172.21.15.189:0/3623900971 >> 172.21.15.189:0/3623900971 conn(0x7fde18062080 msgr2=0x7fde18062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:07.351 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.318+0000 7fde17fff700 1 -- 172.21.15.189:0/3623900971 shutdown_connections 2020-11-20T12:18:07.351 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.318+0000 7fde17fff700 1 -- 172.21.15.189:0/3623900971 wait complete. 2020-11-20T12:18:07.351 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.322+0000 7fde17fff700 1 Processor -- start 2020-11-20T12:18:07.351 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.322+0000 7fde17fff700 1 -- start start 2020-11-20T12:18:07.351 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.322+0000 7fde17fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde180a9fd0 0x7fde18132340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:07.352 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.322+0000 7fde17fff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fde180aebd0 0x7fde180aca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:07.352 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.322+0000 7fde17fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde180aefb0 0x7fde1813add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:07.352 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.322+0000 7fde17fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1813bd20 con 0x7fde180aefb0 2020-11-20T12:18:07.352 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.322+0000 7fde17fff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1813bf10 con 0x7fde180a9fd0 2020-11-20T12:18:07.352 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.322+0000 7fde17fff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1813c1b0 con 0x7fde180aebd0 2020-11-20T12:18:07.352 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.322+0000 7fde16ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde180a9fd0 0x7fde18132340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:07.353 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.322+0000 7fde16ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde180a9fd0 0x7fde18132340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51462/0 (socket says 172.21.15.189:51462) 2020-11-20T12:18:07.353 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.322+0000 7fde16ffd700 1 -- 172.21.15.189:0/3256028122 learned_addr learned my addr 172.21.15.189:0/3256028122 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:18:07.353 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.322+0000 7fde0e7fc700 1 --2- 172.21.15.189:0/3256028122 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fde180aebd0 0x7fde180aca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:07.353 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.322+0000 7fde177fe700 1 --2- 172.21.15.189:0/3256028122 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde180aefb0 0x7fde1813add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:07.354 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.322+0000 7fde16ffd700 1 -- 172.21.15.189:0/3256028122 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fde180aebd0 msgr2=0x7fde180aca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:07.354 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.322+0000 7fde16ffd700 1 --2- 172.21.15.189:0/3256028122 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fde180aebd0 0x7fde180aca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:07.355 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.322+0000 7fde16ffd700 1 -- 172.21.15.189:0/3256028122 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde180aefb0 msgr2=0x7fde1813add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:07.356 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.322+0000 7fde16ffd700 1 --2- 172.21.15.189:0/3256028122 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde180aefb0 0x7fde1813add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:07.358 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.322+0000 7fde16ffd700 1 -- 172.21.15.189:0/3256028122 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1813c450 con 0x7fde180a9fd0 2020-11-20T12:18:07.358 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.322+0000 7fde177fe700 1 --2- 172.21.15.189:0/3256028122 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde180aefb0 0x7fde1813add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:18:07.358 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.322+0000 7fde16ffd700 1 --2- 172.21.15.189:0/3256028122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde180a9fd0 0x7fde18132340 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7fde0800ec00 tx=0x7fde08006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:07.358 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.322+0000 7fde0f7fe700 1 -- 172.21.15.189:0/3256028122 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0800c9b0 con 0x7fde180a9fd0 2020-11-20T12:18:07.359 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.322+0000 7fde0f7fe700 1 -- 172.21.15.189:0/3256028122 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0800cb10 con 0x7fde180a9fd0 2020-11-20T12:18:07.359 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.322+0000 7fde0f7fe700 1 -- 172.21.15.189:0/3256028122 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0800ceb0 con 0x7fde180a9fd0 2020-11-20T12:18:07.359 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.322+0000 7fde17fff700 1 -- 172.21.15.189:0/3256028122 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde180acf40 con 0x7fde180a9fd0 2020-11-20T12:18:07.359 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.322+0000 7fde17fff700 1 -- 172.21.15.189:0/3256028122 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde1813b460 con 0x7fde180a9fd0 2020-11-20T12:18:07.359 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.326+0000 7fde17fff700 1 -- 172.21.15.189:0/3256028122 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde1813ac40 con 0x7fde180a9fd0 2020-11-20T12:18:07.360 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.326+0000 7fde0f7fe700 1 -- 172.21.15.189:0/3256028122 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fde08012600 con 0x7fde180a9fd0 2020-11-20T12:18:07.360 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.326+0000 7fde0f7fe700 1 --2- 172.21.15.189:0/3256028122 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fddf40557c0 0x7fddf4057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:07.360 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.326+0000 7fde0f7fe700 1 -- 172.21.15.189:0/3256028122 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6933+0+0 (secure 0 0 0) 0x7fde0806ee20 con 0x7fde180a9fd0 2020-11-20T12:18:07.360 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.326+0000 7fde0e7fc700 1 --2- 172.21.15.189:0/3256028122 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fddf40557c0 0x7fddf4057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:07.360 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.326+0000 7fde0e7fc700 1 --2- 172.21.15.189:0/3256028122 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fddf40557c0 0x7fddf4057c80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fde180a91d0 tx=0x7fde00015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:07.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.334+0000 7fde0f7fe700 1 -- 172.21.15.189:0/3256028122 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fde08044960 con 0x7fde180a9fd0 2020-11-20T12:18:07.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.562+0000 7fde17fff700 1 -- 172.21.15.189:0/3256028122 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde180ad130 con 0x7fde180a9fd0 2020-11-20T12:18:07.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.562+0000 7fde0f7fe700 1 -- 172.21.15.189:0/3256028122 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+1785 (secure 0 0 0) 0x7fde08047f80 con 0x7fde180a9fd0 2020-11-20T12:18:07.569 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:18:07.569 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:17:55.763199+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5572":{"gid":5572,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/344330560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":344330560},{"type":"v1","addr":"172.21.15.189:6835","nonce":344330560}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:18:07.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.562+0000 7fde17fff700 1 -- 172.21.15.189:0/3256028122 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fddf40557c0 msgr2=0x7fddf4057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:07.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.562+0000 7fde17fff700 1 --2- 172.21.15.189:0/3256028122 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fddf40557c0 0x7fddf4057c80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fde180a91d0 tx=0x7fde00015000).stop 2020-11-20T12:18:07.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.562+0000 7fde17fff700 1 -- 172.21.15.189:0/3256028122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde180a9fd0 msgr2=0x7fde18132340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:07.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.562+0000 7fde17fff700 1 --2- 172.21.15.189:0/3256028122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde180a9fd0 0x7fde18132340 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7fde0800ec00 tx=0x7fde08006df0).stop 2020-11-20T12:18:07.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.562+0000 7fde17fff700 1 -- 172.21.15.189:0/3256028122 shutdown_connections 2020-11-20T12:18:07.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.562+0000 7fde17fff700 1 --2- 172.21.15.189:0/3256028122 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fddf40557c0 0x7fddf4057c80 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:07.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.562+0000 7fde17fff700 1 --2- 172.21.15.189:0/3256028122 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fde180aefb0 0x7fde1813add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:07.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.562+0000 7fde17fff700 1 --2- 172.21.15.189:0/3256028122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde180a9fd0 0x7fde18132340 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:07.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.562+0000 7fde17fff700 1 --2- 172.21.15.189:0/3256028122 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fde180aebd0 0x7fde180aca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:07.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.562+0000 7fde17fff700 1 -- 172.21.15.189:0/3256028122 >> 172.21.15.189:0/3256028122 conn(0x7fde18062080 msgr2=0x7fde180608c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:07.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.562+0000 7fde17fff700 1 -- 172.21.15.189:0/3256028122 shutdown_connections 2020-11-20T12:18:07.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:07.562+0000 7fde17fff700 1 -- 172.21.15.189:0/3256028122 wait complete. 2020-11-20T12:18:07.574 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 55 2020-11-20T12:18:07.586 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2020-11-20T12:18:09.587 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:18:09.727 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.714+0000 7f0dbe3ff700 1 Processor -- start 2020-11-20T12:18:09.728 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.714+0000 7f0dbe3ff700 1 -- start start 2020-11-20T12:18:09.728 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.718+0000 7f0dbe3ff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0db80623c0 0x7f0db8138290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:09.728 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.718+0000 7f0dbe3ff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0db8061040 0x7f0db813bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:09.728 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.718+0000 7f0dbe3ff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db8065ca0 con 0x7f0db8060bc0 2020-11-20T12:18:09.728 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.718+0000 7f0dbe3ff700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0db80687e0 con 0x7f0db80623c0 2020-11-20T12:18:09.729 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.718+0000 7f0dbd3fd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0db80623c0 0x7f0db8138290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:09.729 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.718+0000 7f0dbe3ff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db8138ba0 con 0x7f0db8061040 2020-11-20T12:18:09.729 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.718+0000 7f0dbd3fd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0db80623c0 0x7f0db8138290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51470/0 (socket says 172.21.15.189:51470) 2020-11-20T12:18:09.729 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.718+0000 7f0dbd3fd700 1 -- 172.21.15.189:0/2645943074 learned_addr learned my addr 172.21.15.189:0/2645943074 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:18:09.729 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.718+0000 7f0dbdbfe700 1 --2- 172.21.15.189:0/2645943074 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0db8061040 0x7f0db813bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:09.730 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.718+0000 7f0dbd3fd700 1 -- 172.21.15.189:0/2645943074 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0db8061040 msgr2=0x7f0db813bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:18:09.730 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.718+0000 7f0dbd3fd700 1 --2- 172.21.15.189:0/2645943074 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0db8061040 0x7f0db813bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:09.730 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.718+0000 7f0dbd3fd700 1 -- 172.21.15.189:0/2645943074 >> v1:172.21.15.47:6789/0 conn(0x7f0db8060bc0 legacy=0x7f0db813b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:09.730 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.718+0000 7f0dbd3fd700 1 -- 172.21.15.189:0/2645943074 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db8138e40 con 0x7f0db80623c0 2020-11-20T12:18:09.731 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.718+0000 7f0dbd3fd700 1 --2- 172.21.15.189:0/2645943074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0db80623c0 0x7f0db8138290 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7f0db4006fd0 tx=0x7f0db4006eb0).ready entity=mon.1 client_cookie=ee570157eada320d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:09.731 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.718+0000 7f0da7fff700 1 -- 172.21.15.189:0/2645943074 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db4009960 con 0x7f0db80623c0 2020-11-20T12:18:09.731 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.718+0000 7f0da7fff700 1 -- 172.21.15.189:0/2645943074 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db4009ac0 con 0x7f0db80623c0 2020-11-20T12:18:09.731 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.718+0000 7f0da7fff700 1 -- 172.21.15.189:0/2645943074 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db4009e40 con 0x7f0db80623c0 2020-11-20T12:18:09.731 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.718+0000 7f0dbe3ff700 1 -- 172.21.15.189:0/2645943074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0db80623c0 msgr2=0x7f0db8138290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:09.732 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.718+0000 7f0dbe3ff700 1 --2- 172.21.15.189:0/2645943074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0db80623c0 0x7f0db8138290 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7f0db4006fd0 tx=0x7f0db4006eb0).stop 2020-11-20T12:18:09.732 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.718+0000 7f0dbe3ff700 1 -- 172.21.15.189:0/2645943074 shutdown_connections 2020-11-20T12:18:09.732 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.718+0000 7f0dbe3ff700 1 --2- 172.21.15.189:0/2645943074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0db80623c0 0x7f0db8138290 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:09.732 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.718+0000 7f0dbe3ff700 1 --2- 172.21.15.189:0/2645943074 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0db8061040 0x7f0db813bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:09.732 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.718+0000 7f0dbe3ff700 1 -- 172.21.15.189:0/2645943074 >> 172.21.15.189:0/2645943074 conn(0x7f0db8065fe0 msgr2=0x7f0db8061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:09.733 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.718+0000 7f0dbe3ff700 1 -- 172.21.15.189:0/2645943074 shutdown_connections 2020-11-20T12:18:09.733 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.718+0000 7f0dbe3ff700 1 -- 172.21.15.189:0/2645943074 wait complete. 2020-11-20T12:18:09.733 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.722+0000 7f0dbe3ff700 1 Processor -- start 2020-11-20T12:18:09.734 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.722+0000 7f0dbe3ff700 1 -- start start 2020-11-20T12:18:09.737 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.722+0000 7f0dbe3ff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0db8060bc0 0x7f0db809d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:09.737 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.722+0000 7f0dbe3ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0db8061040 0x7f0db80aa9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:09.737 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.722+0000 7f0dbe3ff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0db80623c0 0x7f0db80a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:09.737 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.722+0000 7f0dbe3ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db80a3600 con 0x7f0db8061040 2020-11-20T12:18:09.738 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.722+0000 7f0dbe3ff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db80a1480 con 0x7f0db8060bc0 2020-11-20T12:18:09.738 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.722+0000 7f0dbe3ff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db80a2c20 con 0x7f0db80623c0 2020-11-20T12:18:09.738 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.722+0000 7f0dbd3fd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0db8060bc0 0x7f0db809d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:09.738 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.722+0000 7f0dbd3fd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0db8060bc0 0x7f0db809d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51476/0 (socket says 172.21.15.189:51476) 2020-11-20T12:18:09.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.722+0000 7f0dbd3fd700 1 -- 172.21.15.189:0/3019050017 learned_addr learned my addr 172.21.15.189:0/3019050017 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:18:09.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.722+0000 7f0dbcbfc700 1 --2- 172.21.15.189:0/3019050017 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0db8061040 0x7f0db80aa9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:09.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.722+0000 7f0dbdbfe700 1 --2- 172.21.15.189:0/3019050017 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0db80623c0 0x7f0db80a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:09.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.722+0000 7f0dbd3fd700 1 -- 172.21.15.189:0/3019050017 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0db80623c0 msgr2=0x7f0db80a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:09.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.722+0000 7f0dbd3fd700 1 --2- 172.21.15.189:0/3019050017 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0db80623c0 0x7f0db80a3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:09.740 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.722+0000 7f0dbd3fd700 1 -- 172.21.15.189:0/3019050017 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0db8061040 msgr2=0x7f0db80aa9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:09.740 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.722+0000 7f0dbd3fd700 1 --2- 172.21.15.189:0/3019050017 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0db8061040 0x7f0db80aa9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:09.740 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.722+0000 7f0dbd3fd700 1 -- 172.21.15.189:0/3019050017 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db80a2ec0 con 0x7f0db8060bc0 2020-11-20T12:18:09.740 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.722+0000 7f0dbd3fd700 1 --2- 172.21.15.189:0/3019050017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0db8060bc0 0x7f0db809d390 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f0db80ab3f0 tx=0x7f0db4015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:09.740 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.722+0000 7f0da57fa700 1 -- 172.21.15.189:0/3019050017 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db4006890 con 0x7f0db8060bc0 2020-11-20T12:18:09.741 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.722+0000 7f0da57fa700 1 -- 172.21.15.189:0/3019050017 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db401e460 con 0x7f0db8060bc0 2020-11-20T12:18:09.741 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.722+0000 7f0dbe3ff700 1 -- 172.21.15.189:0/3019050017 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db80a31c0 con 0x7f0db8060bc0 2020-11-20T12:18:09.741 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.722+0000 7f0da57fa700 1 -- 172.21.15.189:0/3019050017 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db401e800 con 0x7f0db8060bc0 2020-11-20T12:18:09.741 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.722+0000 7f0dbe3ff700 1 -- 172.21.15.189:0/3019050017 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db80aae90 con 0x7f0db8060bc0 2020-11-20T12:18:09.741 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.722+0000 7f0dbe3ff700 1 -- 172.21.15.189:0/3019050017 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db80a2a40 con 0x7f0db8060bc0 2020-11-20T12:18:09.742 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.722+0000 7f0da57fa700 1 -- 172.21.15.189:0/3019050017 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f0db4025030 con 0x7f0db8060bc0 2020-11-20T12:18:09.742 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.726+0000 7f0da57fa700 1 --2- 172.21.15.189:0/3019050017 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0d900557c0 0x7f0d90057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:18:09.742 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.726+0000 7f0da57fa700 1 -- 172.21.15.189:0/3019050017 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6933+0+0 (secure 0 0 0) 0x7f0db40701a0 con 0x7f0db8060bc0 2020-11-20T12:18:09.742 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.726+0000 7f0dbcbfc700 1 --2- 172.21.15.189:0/3019050017 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0d900557c0 0x7f0d90057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:18:09.742 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.726+0000 7f0dbcbfc700 1 --2- 172.21.15.189:0/3019050017 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0d900557c0 0x7f0d90057c80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f0db80a27c0 tx=0x7f0da8009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:18:09.743 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.730+0000 7f0da57fa700 1 -- 172.21.15.189:0/3019050017 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0db4045ce0 con 0x7f0db8060bc0 2020-11-20T12:18:09.972 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.966+0000 7f0dbe3ff700 1 -- 172.21.15.189:0/3019050017 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0db80663c0 con 0x7f0db8060bc0 2020-11-20T12:18:09.973 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.966+0000 7f0da57fa700 1 -- 172.21.15.189:0/3019050017 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f0db4023020 con 0x7f0db8060bc0 2020-11-20T12:18:09.973 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:18:09.973 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5520,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/201662157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":201662157},{"type":"v1","addr":"172.21.15.47:6833","nonce":201662157}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:17:55.763199+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5572":{"gid":5572,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/344330560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":344330560},{"type":"v1","addr":"172.21.15.189:6835","nonce":344330560}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:18:09.977 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.970+0000 7f0dbe3ff700 1 -- 172.21.15.189:0/3019050017 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0d900557c0 msgr2=0x7f0d90057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:09.977 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.970+0000 7f0dbe3ff700 1 --2- 172.21.15.189:0/3019050017 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0d900557c0 0x7f0d90057c80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f0db80a27c0 tx=0x7f0da8009040).stop 2020-11-20T12:18:09.977 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.970+0000 7f0dbe3ff700 1 -- 172.21.15.189:0/3019050017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0db8060bc0 msgr2=0x7f0db809d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:18:09.977 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.970+0000 7f0dbe3ff700 1 --2- 172.21.15.189:0/3019050017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0db8060bc0 0x7f0db809d390 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f0db80ab3f0 tx=0x7f0db4015000).stop 2020-11-20T12:18:09.978 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.970+0000 7f0dbe3ff700 1 -- 172.21.15.189:0/3019050017 shutdown_connections 2020-11-20T12:18:09.978 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.970+0000 7f0dbe3ff700 1 --2- 172.21.15.189:0/3019050017 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0db80623c0 0x7f0db80a3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:09.978 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.970+0000 7f0dbe3ff700 1 --2- 172.21.15.189:0/3019050017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0db8060bc0 0x7f0db809d390 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:09.978 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.970+0000 7f0dbe3ff700 1 --2- 172.21.15.189:0/3019050017 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0d900557c0 0x7f0d90057c80 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:09.979 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.970+0000 7f0dbe3ff700 1 --2- 172.21.15.189:0/3019050017 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0db8061040 0x7f0db80aa9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:18:09.979 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.970+0000 7f0dbe3ff700 1 -- 172.21.15.189:0/3019050017 >> 172.21.15.189:0/3019050017 conn(0x7f0db8065fe0 msgr2=0x7f0db80615e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:18:09.979 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.970+0000 7f0dbe3ff700 1 -- 172.21.15.189:0/3019050017 shutdown_connections 2020-11-20T12:18:09.979 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:18:09.970+0000 7f0dbe3ff700 1 -- 172.21.15.189:0/3019050017 wait complete. 2020-11-20T12:18:09.980 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 56 2020-11-20T12:18:09.991 INFO:tasks.mds_thrash.fs.[None]:mds.b reported in up:standby state 2020-11-20T12:18:09.992 INFO:tasks.mds_thrash.fs.[None]:waiting for 100 secs before thrashing 2020-11-20T12:18:25.450 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:18:25.453 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:18:55.885 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:18:55.888 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:19:25.936 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:19:25.971 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:19:49.993 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:19:50.125 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.120+0000 7f5b9359e700 1 Processor -- start 2020-11-20T12:19:50.125 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.120+0000 7f5b9359e700 1 -- start start 2020-11-20T12:19:50.135 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.120+0000 7f5b9359e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5b94060310 0x7f5b940a1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:50.135 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.120+0000 7f5b9359e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5b940a8450 0x7f5b940a9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:50.136 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.120+0000 7f5b9359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b940a8fa0 con 0x7f5b940a2440 2020-11-20T12:19:50.136 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.120+0000 7f5b9359e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b940618b0 con 0x7f5b94060310 2020-11-20T12:19:50.136 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.120+0000 7f5b9359e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b940a9240 con 0x7f5b940a8450 2020-11-20T12:19:50.137 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.120+0000 7f5b9259c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5b94060310 0x7f5b940a1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:50.137 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.120+0000 7f5b8bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5b940a2440 0x7f5b940a7d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53140/0 (socket says 172.21.15.189:53140) 2020-11-20T12:19:50.138 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.120+0000 7f5b8bfff700 1 -- 172.21.15.189:0/911294209 learned_addr learned my addr 172.21.15.189:0/911294209 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:19:50.138 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.120+0000 7f5b92d9d700 1 --2- 172.21.15.189:0/911294209 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5b940a8450 0x7f5b940a9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:50.139 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.120+0000 7f5b9259c700 1 -- 172.21.15.189:0/911294209 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5b940a8450 msgr2=0x7f5b940a9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:50.139 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.120+0000 7f5b9259c700 1 --2- 172.21.15.189:0/911294209 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5b940a8450 0x7f5b940a9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.139 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.120+0000 7f5b9259c700 1 -- 172.21.15.189:0/911294209 >> v1:172.21.15.189:6789/0 conn(0x7f5b940a2440 legacy=0x7f5b940a7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:50.140 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.120+0000 7f5b91d9b700 1 -- 172.21.15.189:0/911294209 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2036015420 0 0) 0x7f5b940a8fa0 con 0x7f5b940a2440 2020-11-20T12:19:50.140 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.120+0000 7f5b9259c700 1 -- 172.21.15.189:0/911294209 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b940a94e0 con 0x7f5b94060310 2020-11-20T12:19:50.141 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.120+0000 7f5b92d9d700 1 --2- 172.21.15.189:0/911294209 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5b940a8450 0x7f5b940a9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:19:50.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.120+0000 7f5b9259c700 1 --2- 172.21.15.189:0/911294209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b94060310 0x7f5b940a1f50 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7f5b8400b9a0 tx=0x7f5b84006df0).ready entity=mon.1 client_cookie=d473f5cac04f5818 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:50.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.124+0000 7f5b91d9b700 1 -- 172.21.15.189:0/911294209 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b84009840 con 0x7f5b94060310 2020-11-20T12:19:50.142 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.124+0000 7f5b91d9b700 1 -- 172.21.15.189:0/911294209 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b840099a0 con 0x7f5b94060310 2020-11-20T12:19:50.143 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.124+0000 7f5b91d9b700 1 -- 172.21.15.189:0/911294209 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b84009d20 con 0x7f5b94060310 2020-11-20T12:19:50.143 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.124+0000 7f5b9359e700 1 -- 172.21.15.189:0/911294209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b94060310 msgr2=0x7f5b940a1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:50.143 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.124+0000 7f5b9359e700 1 --2- 172.21.15.189:0/911294209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b94060310 0x7f5b940a1f50 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7f5b8400b9a0 tx=0x7f5b84006df0).stop 2020-11-20T12:19:50.143 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.124+0000 7f5b9359e700 1 -- 172.21.15.189:0/911294209 shutdown_connections 2020-11-20T12:19:50.143 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.124+0000 7f5b9359e700 1 --2- 172.21.15.189:0/911294209 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5b940a8450 0x7f5b940a9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.144 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.124+0000 7f5b9359e700 1 --2- 172.21.15.189:0/911294209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b94060310 0x7f5b940a1f50 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.144 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.124+0000 7f5b9359e700 1 -- 172.21.15.189:0/911294209 >> 172.21.15.189:0/911294209 conn(0x7f5b94063210 msgr2=0x7f5b94060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:50.144 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.124+0000 7f5b9359e700 1 -- 172.21.15.189:0/911294209 shutdown_connections 2020-11-20T12:19:50.144 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.124+0000 7f5b9359e700 1 -- 172.21.15.189:0/911294209 wait complete. 2020-11-20T12:19:50.144 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.124+0000 7f5b9359e700 1 Processor -- start 2020-11-20T12:19:50.145 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.124+0000 7f5b9359e700 1 -- start start 2020-11-20T12:19:50.145 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.124+0000 7f5b9359e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b94060310 0x7f5b94137790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:50.145 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.124+0000 7f5b9359e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5b940a2440 0x7f5b9409c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:50.145 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.124+0000 7f5b9359e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b940a8450 0x7f5b94136be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:50.146 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.124+0000 7f5b9359e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b94137db0 con 0x7f5b94060310 2020-11-20T12:19:50.146 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.124+0000 7f5b9359e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b94138050 con 0x7f5b940a8450 2020-11-20T12:19:50.146 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.124+0000 7f5b9359e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b94147700 con 0x7f5b940a2440 2020-11-20T12:19:50.146 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.124+0000 7f5b92d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b940a8450 0x7f5b94136be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:50.146 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.124+0000 7f5b92d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b940a8450 0x7f5b94136be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51526/0 (socket says 172.21.15.189:51526) 2020-11-20T12:19:50.147 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.124+0000 7f5b92d9d700 1 -- 172.21.15.189:0/4068842230 learned_addr learned my addr 172.21.15.189:0/4068842230 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:19:50.147 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.124+0000 7f5b8bfff700 1 --2- 172.21.15.189:0/4068842230 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5b940a2440 0x7f5b9409c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:50.147 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.124+0000 7f5b9259c700 1 --2- 172.21.15.189:0/4068842230 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b94060310 0x7f5b94137790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:50.147 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.128+0000 7f5b92d9d700 1 -- 172.21.15.189:0/4068842230 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5b940a2440 msgr2=0x7f5b9409c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:50.147 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.128+0000 7f5b92d9d700 1 --2- 172.21.15.189:0/4068842230 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5b940a2440 0x7f5b9409c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.148 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.128+0000 7f5b92d9d700 1 -- 172.21.15.189:0/4068842230 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b94060310 msgr2=0x7f5b94137790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:50.148 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.128+0000 7f5b92d9d700 1 --2- 172.21.15.189:0/4068842230 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b94060310 0x7f5b94137790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.148 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.128+0000 7f5b92d9d700 1 -- 172.21.15.189:0/4068842230 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b941382f0 con 0x7f5b940a8450 2020-11-20T12:19:50.148 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.128+0000 7f5b9259c700 1 --2- 172.21.15.189:0/4068842230 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b94060310 0x7f5b94137790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:19:50.149 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.128+0000 7f5b8bfff700 1 --2- 172.21.15.189:0/4068842230 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5b940a2440 0x7f5b9409c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:19:50.149 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.128+0000 7f5b92d9d700 1 --2- 172.21.15.189:0/4068842230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b940a8450 0x7f5b94136be0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7f5b8c00db30 tx=0x7f5b8c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:50.149 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.128+0000 7f5b8a7fc700 1 -- 172.21.15.189:0/4068842230 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c00b870 con 0x7f5b940a8450 2020-11-20T12:19:50.149 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.128+0000 7f5b8a7fc700 1 -- 172.21.15.189:0/4068842230 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b8c00b9f0 con 0x7f5b940a8450 2020-11-20T12:19:50.149 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.128+0000 7f5b9359e700 1 -- 172.21.15.189:0/4068842230 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b940a0640 con 0x7f5b940a8450 2020-11-20T12:19:50.150 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.128+0000 7f5b8a7fc700 1 -- 172.21.15.189:0/4068842230 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c00bdb0 con 0x7f5b940a8450 2020-11-20T12:19:50.150 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.128+0000 7f5b9359e700 1 -- 172.21.15.189:0/4068842230 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b94137270 con 0x7f5b940a8450 2020-11-20T12:19:50.150 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.128+0000 7f5b9359e700 1 -- 172.21.15.189:0/4068842230 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b941366c0 con 0x7f5b940a8450 2020-11-20T12:19:50.150 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.128+0000 7f5b8a7fc700 1 -- 172.21.15.189:0/4068842230 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f5b8c012630 con 0x7f5b940a8450 2020-11-20T12:19:50.150 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.128+0000 7f5b8a7fc700 1 --2- 172.21.15.189:0/4068842230 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5b700557c0 0x7f5b70057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:50.151 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.128+0000 7f5b8a7fc700 1 -- 172.21.15.189:0/4068842230 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6933+0+0 (secure 0 0 0) 0x7f5b8c06e800 con 0x7f5b940a8450 2020-11-20T12:19:50.151 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.128+0000 7f5b9259c700 1 --2- 172.21.15.189:0/4068842230 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5b700557c0 0x7f5b70057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:50.151 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.128+0000 7f5b9259c700 1 --2- 172.21.15.189:0/4068842230 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5b700557c0 0x7f5b70057c80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f5b8400b150 tx=0x7f5b84015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:50.152 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.132+0000 7f5b8a7fc700 1 -- 172.21.15.189:0/4068842230 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5b941366c0 con 0x7f5b940a8450 2020-11-20T12:19:50.362 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.356+0000 7f5b9359e700 1 -- 172.21.15.189:0/4068842230 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b941366c0 con 0x7f5b940a8450 2020-11-20T12:19:50.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.356+0000 7f5b8a7fc700 1 -- 172.21.15.189:0/4068842230 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f5b941366c0 con 0x7f5b940a8450 2020-11-20T12:19:50.363 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:19:50.364 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5520,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/201662157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":201662157},{"type":"v1","addr":"172.21.15.47:6833","nonce":201662157}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:17:55.763199+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5572":{"gid":5572,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/344330560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":344330560},{"type":"v1","addr":"172.21.15.189:6835","nonce":344330560}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:19:50.366 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.360+0000 7f5b9359e700 1 -- 172.21.15.189:0/4068842230 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5b700557c0 msgr2=0x7f5b70057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:50.366 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.360+0000 7f5b9359e700 1 --2- 172.21.15.189:0/4068842230 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5b700557c0 0x7f5b70057c80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f5b8400b150 tx=0x7f5b84015000).stop 2020-11-20T12:19:50.366 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.360+0000 7f5b9359e700 1 -- 172.21.15.189:0/4068842230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b940a8450 msgr2=0x7f5b94136be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:50.367 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.360+0000 7f5b9359e700 1 --2- 172.21.15.189:0/4068842230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b940a8450 0x7f5b94136be0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7f5b8c00db30 tx=0x7f5b8c008040).stop 2020-11-20T12:19:50.367 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.360+0000 7f5b9359e700 1 -- 172.21.15.189:0/4068842230 shutdown_connections 2020-11-20T12:19:50.367 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.360+0000 7f5b9359e700 1 --2- 172.21.15.189:0/4068842230 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f5b700557c0 0x7f5b70057c80 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.367 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.360+0000 7f5b9359e700 1 --2- 172.21.15.189:0/4068842230 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f5b94060310 0x7f5b94137790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.367 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.360+0000 7f5b9359e700 1 --2- 172.21.15.189:0/4068842230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b940a8450 0x7f5b94136be0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.368 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.360+0000 7f5b9359e700 1 --2- 172.21.15.189:0/4068842230 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f5b940a2440 0x7f5b9409c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.368 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.360+0000 7f5b9359e700 1 -- 172.21.15.189:0/4068842230 >> 172.21.15.189:0/4068842230 conn(0x7f5b94063210 msgr2=0x7f5b940aa460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:50.368 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.360+0000 7f5b9359e700 1 -- 172.21.15.189:0/4068842230 shutdown_connections 2020-11-20T12:19:50.368 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.360+0000 7f5b9359e700 1 -- 172.21.15.189:0/4068842230 wait complete. 2020-11-20T12:19:50.369 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 56 2020-11-20T12:19:50.380 INFO:tasks.mds_thrash.fs.[None]:kill mds.a (rank=0) 2020-11-20T12:19:50.381 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T12:19:50.381 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:19:50.394 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T12:19:50.394 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:19:50.559 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.552+0000 7fe882e0f700 1 Processor -- start 2020-11-20T12:19:50.560 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.552+0000 7fe882e0f700 1 -- start start 2020-11-20T12:19:50.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe882e0f700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe87c0ad000 0x7fe87c13e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:50.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe882e0f700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe87c0ad3e0 0x7fe87c066d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:50.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe882e0f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe87c067730 con 0x7fe87c13a6e0 2020-11-20T12:19:50.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe882e0f700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe87c138190 con 0x7fe87c0ad000 2020-11-20T12:19:50.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe882e0f700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c13dd40 con 0x7fe87c0ad3e0 2020-11-20T12:19:50.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe88260e700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe87c13a6e0 0x7fe87c13ad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53158/0 (socket says 172.21.15.189:53158) 2020-11-20T12:19:50.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe88260e700 1 -- 172.21.15.189:0/2650560202 learned_addr learned my addr 172.21.15.189:0/2650560202 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:19:50.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe881e0d700 1 --2- 172.21.15.189:0/2650560202 >> v2:172.21.15.47:3300/0 conn(0x7fe87c0ad000 0x7fe87c13e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:50.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe88160c700 1 --2- 172.21.15.189:0/2650560202 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe87c0ad3e0 0x7fe87c066d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:50.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe880e0b700 1 -- 172.21.15.189:0/2650560202 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316836894 0 0) 0x7fe87c067730 con 0x7fe87c13a6e0 2020-11-20T12:19:50.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe880e0b700 1 -- 172.21.15.189:0/2650560202 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe864003240 con 0x7fe87c13a6e0 2020-11-20T12:19:50.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe880e0b700 1 -- 172.21.15.189:0/2650560202 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 506394008 0 0) 0x7fe864003240 con 0x7fe87c13a6e0 2020-11-20T12:19:50.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe881e0d700 1 -- 172.21.15.189:0/2650560202 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe87c0ad3e0 msgr2=0x7fe87c066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:50.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe881e0d700 1 --2- 172.21.15.189:0/2650560202 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe87c0ad3e0 0x7fe87c066d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe881e0d700 1 -- 172.21.15.189:0/2650560202 >> v1:172.21.15.189:6789/0 conn(0x7fe87c13a6e0 legacy=0x7fe87c13ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:50.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe881e0d700 1 -- 172.21.15.189:0/2650560202 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe87c13dfe0 con 0x7fe87c0ad000 2020-11-20T12:19:50.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe881e0d700 1 --2- 172.21.15.189:0/2650560202 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe87c0ad000 0x7fe87c13e480 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fe878009c90 tx=0x7fe8780091d0).ready entity=mon.0 client_cookie=789a222fbc0e35d4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:50.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe880e0b700 1 -- 172.21.15.189:0/2650560202 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8740014e0 con 0x7fe87c0ad000 2020-11-20T12:19:50.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe880e0b700 1 -- 172.21.15.189:0/2650560202 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe87800b6b0 con 0x7fe87c0ad000 2020-11-20T12:19:50.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe880e0b700 1 -- 172.21.15.189:0/2650560202 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87800ba30 con 0x7fe87c0ad000 2020-11-20T12:19:50.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe882e0f700 1 -- 172.21.15.189:0/2650560202 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe87c0ad000 msgr2=0x7fe87c13e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:50.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe882e0f700 1 --2- 172.21.15.189:0/2650560202 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe87c0ad000 0x7fe87c13e480 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fe878009c90 tx=0x7fe8780091d0).stop 2020-11-20T12:19:50.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe882e0f700 1 -- 172.21.15.189:0/2650560202 shutdown_connections 2020-11-20T12:19:50.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe882e0f700 1 --2- 172.21.15.189:0/2650560202 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe87c0ad000 0x7fe87c13e480 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe882e0f700 1 --2- 172.21.15.189:0/2650560202 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe87c0ad3e0 0x7fe87c066d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe882e0f700 1 -- 172.21.15.189:0/2650560202 >> 172.21.15.189:0/2650560202 conn(0x7fe87c062080 msgr2=0x7fe87c062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:50.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe882e0f700 1 -- 172.21.15.189:0/2650560202 shutdown_connections 2020-11-20T12:19:50.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.556+0000 7fe882e0f700 1 -- 172.21.15.189:0/2650560202 wait complete. 2020-11-20T12:19:50.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe882e0f700 1 Processor -- start 2020-11-20T12:19:50.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe882e0f700 1 -- start start 2020-11-20T12:19:50.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe882e0f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe87c0ad000 0x7fe87c14cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:50.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe882e0f700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe87c0ad3e0 0x7fe87c140930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:50.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe882e0f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe87c0a6430 0x7fe87c0aa9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:50.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe882e0f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c141800 con 0x7fe87c0a6430 2020-11-20T12:19:50.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe882e0f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c14d570 con 0x7fe87c0ad000 2020-11-20T12:19:50.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe882e0f700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c14d810 con 0x7fe87c0ad3e0 2020-11-20T12:19:50.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe881e0d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe87c0ad000 0x7fe87c14cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:50.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe881e0d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe87c0ad000 0x7fe87c14cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51538/0 (socket says 172.21.15.189:51538) 2020-11-20T12:19:50.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe881e0d700 1 -- 172.21.15.189:0/253153557 learned_addr learned my addr 172.21.15.189:0/253153557 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:19:50.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe88160c700 1 --2- 172.21.15.189:0/253153557 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe87c0ad3e0 0x7fe87c140930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:50.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe88260e700 1 --2- 172.21.15.189:0/253153557 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe87c0a6430 0x7fe87c0aa9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:50.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe881e0d700 1 -- 172.21.15.189:0/253153557 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe87c0ad3e0 msgr2=0x7fe87c140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:50.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe881e0d700 1 --2- 172.21.15.189:0/253153557 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe87c0ad3e0 0x7fe87c140930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe881e0d700 1 -- 172.21.15.189:0/253153557 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe87c0a6430 msgr2=0x7fe87c0aa9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:50.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe881e0d700 1 --2- 172.21.15.189:0/253153557 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe87c0a6430 0x7fe87c0aa9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe881e0d700 1 -- 172.21.15.189:0/253153557 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe87c14dab0 con 0x7fe87c0ad000 2020-11-20T12:19:50.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe88260e700 1 --2- 172.21.15.189:0/253153557 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe87c0a6430 0x7fe87c0aa9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:19:50.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe881e0d700 1 --2- 172.21.15.189:0/253153557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe87c0ad000 0x7fe87c14cf40 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7fe87800bb50 tx=0x7fe878016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:50.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe869ffb700 1 -- 172.21.15.189:0/253153557 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe878011df0 con 0x7fe87c0ad000 2020-11-20T12:19:50.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe869ffb700 1 -- 172.21.15.189:0/253153557 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe87801d450 con 0x7fe87c0ad000 2020-11-20T12:19:50.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe869ffb700 1 -- 172.21.15.189:0/253153557 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87801d7f0 con 0x7fe87c0ad000 2020-11-20T12:19:50.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe882e0f700 1 -- 172.21.15.189:0/253153557 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe87c0aaea0 con 0x7fe87c0ad000 2020-11-20T12:19:50.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe882e0f700 1 -- 172.21.15.189:0/253153557 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe87c149970 con 0x7fe87c0ad000 2020-11-20T12:19:50.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe869ffb700 1 -- 172.21.15.189:0/253153557 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fe878017040 con 0x7fe87c0ad000 2020-11-20T12:19:50.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.560+0000 7fe882e0f700 1 -- 172.21.15.189:0/253153557 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe87c0b0e10 con 0x7fe87c0ad000 2020-11-20T12:19:50.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.564+0000 7fe869ffb700 1 --2- 172.21.15.189:0/253153557 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe8540557c0 0x7fe854057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:50.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.564+0000 7fe869ffb700 1 -- 172.21.15.189:0/253153557 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6933+0+0 (secure 0 0 0) 0x7fe878070860 con 0x7fe87c0ad000 2020-11-20T12:19:50.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.564+0000 7fe88160c700 1 --2- 172.21.15.189:0/253153557 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe8540557c0 0x7fe854057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:50.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.564+0000 7fe88160c700 1 --2- 172.21.15.189:0/253153557 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe8540557c0 0x7fe854057c80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fe87c0a5a00 tx=0x7fe86c008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:50.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.568+0000 7fe869ffb700 1 -- 172.21.15.189:0/253153557 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe87c0b0e10 con 0x7fe87c0ad000 2020-11-20T12:19:50.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.800+0000 7fe882e0f700 1 -- 172.21.15.189:0/253153557 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe87c0b0e10 con 0x7fe87c0ad000 2020-11-20T12:19:50.806 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.800+0000 7fe869ffb700 1 -- 172.21.15.189:0/253153557 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2134 (secure 0 0 0) 0x7fe87c0b0e10 con 0x7fe87c0ad000 2020-11-20T12:19:50.807 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:19:50.807 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5520,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/201662157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":201662157},{"type":"v1","addr":"172.21.15.47:6833","nonce":201662157}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:17:55.763199+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5572":{"gid":5572,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/344330560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":344330560},{"type":"v1","addr":"172.21.15.189:6835","nonce":344330560}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:19:50.809 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.804+0000 7fe882e0f700 1 -- 172.21.15.189:0/253153557 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe8540557c0 msgr2=0x7fe854057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:50.809 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.804+0000 7fe882e0f700 1 --2- 172.21.15.189:0/253153557 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe8540557c0 0x7fe854057c80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fe87c0a5a00 tx=0x7fe86c008040).stop 2020-11-20T12:19:50.809 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.804+0000 7fe882e0f700 1 -- 172.21.15.189:0/253153557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe87c0ad000 msgr2=0x7fe87c14cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:50.810 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.804+0000 7fe882e0f700 1 --2- 172.21.15.189:0/253153557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe87c0ad000 0x7fe87c14cf40 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7fe87800bb50 tx=0x7fe878016000).stop 2020-11-20T12:19:50.810 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.804+0000 7fe882e0f700 1 -- 172.21.15.189:0/253153557 shutdown_connections 2020-11-20T12:19:50.810 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.804+0000 7fe882e0f700 1 --2- 172.21.15.189:0/253153557 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe8540557c0 0x7fe854057c80 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.810 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.804+0000 7fe882e0f700 1 --2- 172.21.15.189:0/253153557 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe87c0a6430 0x7fe87c0aa9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.810 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.804+0000 7fe882e0f700 1 --2- 172.21.15.189:0/253153557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe87c0ad000 0x7fe87c14cf40 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.811 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.804+0000 7fe882e0f700 1 --2- 172.21.15.189:0/253153557 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe87c0ad3e0 0x7fe87c140930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:50.811 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.804+0000 7fe882e0f700 1 -- 172.21.15.189:0/253153557 >> 172.21.15.189:0/253153557 conn(0x7fe87c062080 msgr2=0x7fe87c0ac440 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:50.811 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.804+0000 7fe882e0f700 1 -- 172.21.15.189:0/253153557 shutdown_connections 2020-11-20T12:19:50.811 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:50.804+0000 7fe882e0f700 1 -- 172.21.15.189:0/253153557 wait complete. 2020-11-20T12:19:50.812 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 56 2020-11-20T12:19:50.825 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:19:52.826 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:19:52.955 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.948+0000 7f56a9446700 1 Processor -- start 2020-11-20T12:19:52.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.952+0000 7f56a9446700 1 -- start start 2020-11-20T12:19:52.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.952+0000 7f56a9446700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f56a4064500 0x7f56a40aba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:52.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.952+0000 7f56a9446700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f56a40aa980 0x7f56a4060880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:52.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.952+0000 7f56a9446700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a40a27c0 con 0x7f56a40aa4d0 2020-11-20T12:19:52.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.952+0000 7f56a9446700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56a40a97f0 con 0x7f56a40aa980 2020-11-20T12:19:52.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.952+0000 7f56a9446700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56a40a9a90 con 0x7f56a4064500 2020-11-20T12:19:52.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.952+0000 7f56a2ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f56a4064500 0x7f56a40aba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:52.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.952+0000 7f56a2ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f56a4064500 0x7f56a40aba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51554/0 (socket says 172.21.15.189:51554) 2020-11-20T12:19:52.960 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.952+0000 7f56a2ffd700 1 -- 172.21.15.189:0/2408432314 learned_addr learned my addr 172.21.15.189:0/2408432314 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:19:52.960 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.952+0000 7f56a37fe700 1 --2- 172.21.15.189:0/2408432314 >> v2:172.21.15.47:3300/0 conn(0x7f56a40aa980 0x7f56a4060880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:52.960 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.952+0000 7f56a2ffd700 1 -- 172.21.15.189:0/2408432314 >> v2:172.21.15.47:3300/0 conn(0x7f56a40aa980 msgr2=0x7f56a4060880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:52.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.952+0000 7f56a2ffd700 1 --2- 172.21.15.189:0/2408432314 >> v2:172.21.15.47:3300/0 conn(0x7f56a40aa980 0x7f56a4060880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:52.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.952+0000 7f56a2ffd700 1 -- 172.21.15.189:0/2408432314 >> v1:172.21.15.47:6789/0 conn(0x7f56a40aa4d0 legacy=0x7f56a4060170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:52.965 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.952+0000 7f56a2ffd700 1 -- 172.21.15.189:0/2408432314 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a40a9d30 con 0x7f56a4064500 2020-11-20T12:19:52.965 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.952+0000 7f56a2ffd700 1 --2- 172.21.15.189:0/2408432314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56a4064500 0x7f56a40aba30 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7f56a40afbc0 tx=0x7f5694006eb0).ready entity=mon.1 client_cookie=1940f4b509ffb6e0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:52.965 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.952+0000 7f56a1ffb700 1 -- 172.21.15.189:0/2408432314 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5694009810 con 0x7f56a4064500 2020-11-20T12:19:52.966 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.952+0000 7f56a1ffb700 1 -- 172.21.15.189:0/2408432314 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5694009970 con 0x7f56a4064500 2020-11-20T12:19:52.966 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.952+0000 7f56a1ffb700 1 -- 172.21.15.189:0/2408432314 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5694009cf0 con 0x7f56a4064500 2020-11-20T12:19:52.966 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.952+0000 7f56a9446700 1 -- 172.21.15.189:0/2408432314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56a4064500 msgr2=0x7f56a40aba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:52.967 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.952+0000 7f56a9446700 1 --2- 172.21.15.189:0/2408432314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56a4064500 0x7f56a40aba30 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7f56a40afbc0 tx=0x7f5694006eb0).stop 2020-11-20T12:19:52.967 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.952+0000 7f56a9446700 1 -- 172.21.15.189:0/2408432314 shutdown_connections 2020-11-20T12:19:52.967 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.952+0000 7f56a9446700 1 --2- 172.21.15.189:0/2408432314 >> v2:172.21.15.47:3300/0 conn(0x7f56a40aa980 0x7f56a4060880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:52.968 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.952+0000 7f56a9446700 1 --2- 172.21.15.189:0/2408432314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56a4064500 0x7f56a40aba30 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:52.968 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.952+0000 7f56a9446700 1 -- 172.21.15.189:0/2408432314 >> 172.21.15.189:0/2408432314 conn(0x7f56a4061f60 msgr2=0x7f56a4062340 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:52.968 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.952+0000 7f56a9446700 1 -- 172.21.15.189:0/2408432314 shutdown_connections 2020-11-20T12:19:52.969 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.952+0000 7f56a9446700 1 -- 172.21.15.189:0/2408432314 wait complete. 2020-11-20T12:19:52.969 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.956+0000 7f56a9446700 1 Processor -- start 2020-11-20T12:19:52.969 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.956+0000 7f56a9446700 1 -- start start 2020-11-20T12:19:52.970 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.956+0000 7f56a9446700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f56a4064500 0x7f56a409fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:52.970 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.956+0000 7f56a9446700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56a40aa4d0 0x7f56a40a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:52.970 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.956+0000 7f56a9446700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56a40aa980 0x7f56a413a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:52.971 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.956+0000 7f56a9446700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a409e7e0 con 0x7f56a40aa4d0 2020-11-20T12:19:52.971 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.956+0000 7f56a9446700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a409de00 con 0x7f56a40aa980 2020-11-20T12:19:52.971 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.956+0000 7f56a9446700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56a409e0a0 con 0x7f56a4064500 2020-11-20T12:19:52.972 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.956+0000 7f56a37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56a40aa980 0x7f56a413a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:52.972 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.956+0000 7f56a37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56a40aa980 0x7f56a413a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51560/0 (socket says 172.21.15.189:51560) 2020-11-20T12:19:52.972 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.956+0000 7f56a37fe700 1 -- 172.21.15.189:0/206814951 learned_addr learned my addr 172.21.15.189:0/206814951 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:19:52.973 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.956+0000 7f56a2ffd700 1 --2- 172.21.15.189:0/206814951 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f56a4064500 0x7f56a409fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:52.973 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.956+0000 7f56a27fc700 1 --2- 172.21.15.189:0/206814951 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56a40aa4d0 0x7f56a40a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:52.973 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.956+0000 7f56a37fe700 1 -- 172.21.15.189:0/206814951 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f56a4064500 msgr2=0x7f56a409fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:52.974 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.956+0000 7f56a37fe700 1 --2- 172.21.15.189:0/206814951 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f56a4064500 0x7f56a409fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:52.974 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.956+0000 7f56a37fe700 1 -- 172.21.15.189:0/206814951 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56a40aa4d0 msgr2=0x7f56a40a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:52.975 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.956+0000 7f56a37fe700 1 --2- 172.21.15.189:0/206814951 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56a40aa4d0 0x7f56a40a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:52.975 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.956+0000 7f56a37fe700 1 -- 172.21.15.189:0/206814951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a409e340 con 0x7f56a40aa980 2020-11-20T12:19:52.975 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.960+0000 7f56a37fe700 1 --2- 172.21.15.189:0/206814951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56a40aa980 0x7f56a413a6f0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7f569c00ef80 tx=0x7f569c0091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:52.976 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.960+0000 7f568affd700 1 -- 172.21.15.189:0/206814951 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c00cbc0 con 0x7f56a40aa980 2020-11-20T12:19:52.976 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.960+0000 7f568affd700 1 -- 172.21.15.189:0/206814951 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f569c00cd20 con 0x7f56a40aa980 2020-11-20T12:19:52.976 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.960+0000 7f568affd700 1 -- 172.21.15.189:0/206814951 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c012620 con 0x7f56a40aa980 2020-11-20T12:19:52.977 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.960+0000 7f56a9446700 1 -- 172.21.15.189:0/206814951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56a4144950 con 0x7f56a40aa980 2020-11-20T12:19:52.977 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.960+0000 7f56a9446700 1 -- 172.21.15.189:0/206814951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a40a0ca0 con 0x7f56a40aa980 2020-11-20T12:19:52.979 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.960+0000 7f56a9446700 1 -- 172.21.15.189:0/206814951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56a409e4d0 con 0x7f56a40aa980 2020-11-20T12:19:52.979 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.960+0000 7f568affd700 1 -- 172.21.15.189:0/206814951 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f569c022070 con 0x7f56a40aa980 2020-11-20T12:19:52.979 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.960+0000 7f568affd700 1 --2- 172.21.15.189:0/206814951 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f56800557c0 0x7f5680057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:52.980 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.960+0000 7f568affd700 1 -- 172.21.15.189:0/206814951 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6933+0+0 (secure 0 0 0) 0x7f569c06f150 con 0x7f56a40aa980 2020-11-20T12:19:52.980 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.960+0000 7f56a2ffd700 1 --2- 172.21.15.189:0/206814951 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f56800557c0 0x7f5680057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:52.980 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.964+0000 7f56a2ffd700 1 --2- 172.21.15.189:0/206814951 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f56800557c0 0x7f5680057c80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f5694006ef0 tx=0x7f5694015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:52.981 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:52.968+0000 7f568affd700 1 -- 172.21.15.189:0/206814951 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f569c044c90 con 0x7f56a40aa980 2020-11-20T12:19:53.173 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:53.168+0000 7f56a9446700 1 -- 172.21.15.189:0/206814951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56a4044f60 con 0x7f56a40aa980 2020-11-20T12:19:53.174 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:53.168+0000 7f568affd700 1 -- 172.21.15.189:0/206814951 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f569c016070 con 0x7f56a40aa980 2020-11-20T12:19:53.174 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:19:53.175 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5520,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/201662157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":201662157},{"type":"v1","addr":"172.21.15.47:6833","nonce":201662157}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:17:55.763199+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5572":{"gid":5572,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/344330560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":344330560},{"type":"v1","addr":"172.21.15.189:6835","nonce":344330560}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:19:53.175 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:53.172+0000 7f56a9446700 1 -- 172.21.15.189:0/206814951 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f56800557c0 msgr2=0x7f5680057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:53.176 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:53.172+0000 7f56a9446700 1 --2- 172.21.15.189:0/206814951 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f56800557c0 0x7f5680057c80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f5694006ef0 tx=0x7f5694015000).stop 2020-11-20T12:19:53.177 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:53.172+0000 7f56a9446700 1 -- 172.21.15.189:0/206814951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56a40aa980 msgr2=0x7f56a413a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:53.177 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:53.172+0000 7f56a9446700 1 --2- 172.21.15.189:0/206814951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56a40aa980 0x7f56a413a6f0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7f569c00ef80 tx=0x7f569c0091d0).stop 2020-11-20T12:19:53.177 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:53.172+0000 7f56a9446700 1 -- 172.21.15.189:0/206814951 shutdown_connections 2020-11-20T12:19:53.177 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:53.172+0000 7f56a9446700 1 --2- 172.21.15.189:0/206814951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56a40aa980 0x7f56a413a6f0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:53.177 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:53.172+0000 7f56a9446700 1 --2- 172.21.15.189:0/206814951 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f56a4064500 0x7f56a409fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:53.178 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:53.172+0000 7f56a9446700 1 --2- 172.21.15.189:0/206814951 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f56800557c0 0x7f5680057c80 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:53.178 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:53.172+0000 7f56a9446700 1 --2- 172.21.15.189:0/206814951 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f56a40aa4d0 0x7f56a40a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:53.178 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:53.172+0000 7f56a9446700 1 -- 172.21.15.189:0/206814951 >> 172.21.15.189:0/206814951 conn(0x7f56a4061f60 msgr2=0x7f56a4062340 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:53.178 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:53.172+0000 7f56a9446700 1 -- 172.21.15.189:0/206814951 shutdown_connections 2020-11-20T12:19:53.178 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:53.172+0000 7f56a9446700 1 -- 172.21.15.189:0/206814951 wait complete. 2020-11-20T12:19:53.179 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 56 2020-11-20T12:19:53.188 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:19:55.189 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:19:55.561 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.556+0000 7f852721f700 1 Processor -- start 2020-11-20T12:19:55.561 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.556+0000 7f852721f700 1 -- start start 2020-11-20T12:19:55.563 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.556+0000 7f852721f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f85200a9a40 0x7f852013e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:55.563 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.556+0000 7f852721f700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8520060f90 con 0x7f85200a7760 2020-11-20T12:19:55.563 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.556+0000 7f852721f700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f852013dc10 con 0x7f85200a9660 2020-11-20T12:19:55.563 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.556+0000 7f852721f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f852013deb0 con 0x7f85200a9a40 2020-11-20T12:19:55.563 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.556+0000 7f8517fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f85200a9a40 0x7f852013e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:55.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.556+0000 7f8517fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f85200a9a40 0x7f852013e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51568/0 (socket says 172.21.15.189:51568) 2020-11-20T12:19:55.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.556+0000 7f8517fff700 1 -- 172.21.15.189:0/2316155544 learned_addr learned my addr 172.21.15.189:0/2316155544 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:19:55.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.556+0000 7f8517fff700 1 -- 172.21.15.189:0/2316155544 >> v1:172.21.15.189:6789/0 conn(0x7f85200a9660 legacy=0x7f8520060170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:55.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.556+0000 7f8517fff700 1 -- 172.21.15.189:0/2316155544 >> v1:172.21.15.47:6789/0 conn(0x7f85200a7760 legacy=0x7f8520060880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:55.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.556+0000 7f8517fff700 1 -- 172.21.15.189:0/2316155544 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852013e150 con 0x7f85200a9a40 2020-11-20T12:19:55.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.556+0000 7f8517fff700 1 --2- 172.21.15.189:0/2316155544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85200a9a40 0x7f852013e5f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7f8510006fd0 tx=0x7f8510006eb0).ready entity=mon.1 client_cookie=2e6d133e3c60fcd7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:55.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.556+0000 7f85177fe700 1 -- 172.21.15.189:0/2316155544 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8510009890 con 0x7f85200a9a40 2020-11-20T12:19:55.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.556+0000 7f85177fe700 1 -- 172.21.15.189:0/2316155544 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85100099f0 con 0x7f85200a9a40 2020-11-20T12:19:55.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.556+0000 7f85177fe700 1 -- 172.21.15.189:0/2316155544 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8510009d70 con 0x7f85200a9a40 2020-11-20T12:19:55.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f852721f700 1 -- 172.21.15.189:0/2316155544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85200a9a40 msgr2=0x7f852013e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:55.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f852721f700 1 --2- 172.21.15.189:0/2316155544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85200a9a40 0x7f852013e5f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7f8510006fd0 tx=0x7f8510006eb0).stop 2020-11-20T12:19:55.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f852721f700 1 -- 172.21.15.189:0/2316155544 shutdown_connections 2020-11-20T12:19:55.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f852721f700 1 --2- 172.21.15.189:0/2316155544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85200a9a40 0x7f852013e5f0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:55.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f852721f700 1 -- 172.21.15.189:0/2316155544 >> 172.21.15.189:0/2316155544 conn(0x7f8520061f60 msgr2=0x7f8520062370 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:55.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f852721f700 1 -- 172.21.15.189:0/2316155544 shutdown_connections 2020-11-20T12:19:55.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f852721f700 1 -- 172.21.15.189:0/2316155544 wait complete. 2020-11-20T12:19:55.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f852721f700 1 Processor -- start 2020-11-20T12:19:55.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f852721f700 1 -- start start 2020-11-20T12:19:55.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f852721f700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f85200a7760 0x7f852013a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:55.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f852721f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85200a9660 0x7f85200a67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:55.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f852721f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f85200a9a40 0x7f85200a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:55.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f852721f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85200a6ca0 con 0x7f85200a9a40 2020-11-20T12:19:55.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f852721f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852009db40 con 0x7f85200a9660 2020-11-20T12:19:55.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f852721f700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f852009dde0 con 0x7f85200a7760 2020-11-20T12:19:55.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f8517fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85200a9660 0x7f85200a67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:55.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f8517fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85200a9660 0x7f85200a67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51574/0 (socket says 172.21.15.189:51574) 2020-11-20T12:19:55.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f8517fff700 1 -- 172.21.15.189:0/3268210775 learned_addr learned my addr 172.21.15.189:0/3268210775 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:19:55.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f85257bc700 1 --2- 172.21.15.189:0/3268210775 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f85200a9a40 0x7f85200a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:55.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f8524fbb700 1 --2- 172.21.15.189:0/3268210775 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f85200a7760 0x7f852013a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:55.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f8517fff700 1 -- 172.21.15.189:0/3268210775 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f85200a7760 msgr2=0x7f852013a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:55.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f8517fff700 1 --2- 172.21.15.189:0/3268210775 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f85200a7760 0x7f852013a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:55.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f8517fff700 1 -- 172.21.15.189:0/3268210775 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f85200a9a40 msgr2=0x7f85200a5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:55.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f8517fff700 1 --2- 172.21.15.189:0/3268210775 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f85200a9a40 0x7f85200a5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:55.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f8517fff700 1 -- 172.21.15.189:0/3268210775 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852009d630 con 0x7f85200a9660 2020-11-20T12:19:55.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f8524fbb700 1 --2- 172.21.15.189:0/3268210775 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f85200a7760 0x7f852013a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:19:55.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f85257bc700 1 --2- 172.21.15.189:0/3268210775 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f85200a9a40 0x7f85200a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:19:55.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.560+0000 7f8517fff700 1 --2- 172.21.15.189:0/3268210775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85200a9660 0x7f85200a67b0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7f8510006ef0 tx=0x7f8510015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:55.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.564+0000 7f8514ff9700 1 -- 172.21.15.189:0/3268210775 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851001e460 con 0x7f85200a9660 2020-11-20T12:19:55.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.564+0000 7f8514ff9700 1 -- 172.21.15.189:0/3268210775 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f851001e5c0 con 0x7f85200a9660 2020-11-20T12:19:55.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.564+0000 7f852721f700 1 -- 172.21.15.189:0/3268210775 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f852009d880 con 0x7f85200a9660 2020-11-20T12:19:55.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.564+0000 7f8514ff9700 1 -- 172.21.15.189:0/3268210775 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851001e940 con 0x7f85200a9660 2020-11-20T12:19:55.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.564+0000 7f852721f700 1 -- 172.21.15.189:0/3268210775 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85200a5c00 con 0x7f85200a9660 2020-11-20T12:19:55.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.564+0000 7f8514ff9700 1 -- 172.21.15.189:0/3268210775 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f8510025030 con 0x7f85200a9660 2020-11-20T12:19:55.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.564+0000 7f852721f700 1 -- 172.21.15.189:0/3268210775 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8520044f60 con 0x7f85200a9660 2020-11-20T12:19:55.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.564+0000 7f8514ff9700 1 --2- 172.21.15.189:0/3268210775 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f84f80557c0 0x7f84f8057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:55.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.564+0000 7f8514ff9700 1 -- 172.21.15.189:0/3268210775 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6933+0+0 (secure 0 0 0) 0x7f85100702d0 con 0x7f85200a9660 2020-11-20T12:19:55.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.564+0000 7f8524fbb700 1 --2- 172.21.15.189:0/3268210775 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f84f80557c0 0x7f84f8057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:55.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.568+0000 7f8524fbb700 1 --2- 172.21.15.189:0/3268210775 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f84f80557c0 0x7f84f8057c80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f851c001550 tx=0x7f851c009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:55.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.572+0000 7f8514ff9700 1 -- 172.21.15.189:0/3268210775 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8510045ec0 con 0x7f85200a9660 2020-11-20T12:19:55.779 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.772+0000 7f852721f700 1 -- 172.21.15.189:0/3268210775 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85200a3b40 con 0x7f85200a9660 2020-11-20T12:19:55.779 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.776+0000 7f8514ff9700 1 -- 172.21.15.189:0/3268210775 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f8510023070 con 0x7f85200a9660 2020-11-20T12:19:55.780 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:19:55.780 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5520,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/201662157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":201662157},{"type":"v1","addr":"172.21.15.47:6833","nonce":201662157}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:17:55.763199+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5572":{"gid":5572,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/344330560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":344330560},{"type":"v1","addr":"172.21.15.189:6835","nonce":344330560}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:19:55.782 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.776+0000 7f852721f700 1 -- 172.21.15.189:0/3268210775 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f84f80557c0 msgr2=0x7f84f8057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:55.782 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.776+0000 7f852721f700 1 --2- 172.21.15.189:0/3268210775 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f84f80557c0 0x7f84f8057c80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f851c001550 tx=0x7f851c009040).stop 2020-11-20T12:19:55.782 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.776+0000 7f852721f700 1 -- 172.21.15.189:0/3268210775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85200a9660 msgr2=0x7f85200a67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:55.783 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.776+0000 7f852721f700 1 --2- 172.21.15.189:0/3268210775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85200a9660 0x7f85200a67b0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7f8510006ef0 tx=0x7f8510015000).stop 2020-11-20T12:19:55.783 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.776+0000 7f852721f700 1 -- 172.21.15.189:0/3268210775 shutdown_connections 2020-11-20T12:19:55.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.776+0000 7f852721f700 1 --2- 172.21.15.189:0/3268210775 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f84f80557c0 0x7f84f8057c80 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:55.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.776+0000 7f852721f700 1 --2- 172.21.15.189:0/3268210775 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f85200a9a40 0x7f85200a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:55.785 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.776+0000 7f852721f700 1 --2- 172.21.15.189:0/3268210775 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f85200a7760 0x7f852013a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:55.785 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.776+0000 7f852721f700 1 --2- 172.21.15.189:0/3268210775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85200a9660 0x7f85200a67b0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:55.785 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.776+0000 7f852721f700 1 -- 172.21.15.189:0/3268210775 >> 172.21.15.189:0/3268210775 conn(0x7f8520061f60 msgr2=0x7f85200624d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:55.785 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.776+0000 7f852721f700 1 -- 172.21.15.189:0/3268210775 shutdown_connections 2020-11-20T12:19:55.785 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:55.776+0000 7f852721f700 1 -- 172.21.15.189:0/3268210775 wait complete. 2020-11-20T12:19:55.786 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 56 2020-11-20T12:19:55.794 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:19:56.023 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:19:56.028 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:19:57.796 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:19:57.928 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03f171a700 1 Processor -- start 2020-11-20T12:19:57.929 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03f171a700 1 -- start start 2020-11-20T12:19:57.932 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03f171a700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f03ec0ad000 0x7f03ec13e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:57.932 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03f171a700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f03ec0ad3e0 0x7f03ec066d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:57.932 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03f171a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f03ec13a6e0 0x7f03ec13ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:57.933 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03f171a700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03ec138190 con 0x7f03ec0ad000 2020-11-20T12:19:57.933 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03f171a700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03ec067730 con 0x7f03ec13a6e0 2020-11-20T12:19:57.933 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03f171a700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03ec13dd40 con 0x7f03ec0ad3e0 2020-11-20T12:19:57.933 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03f0f19700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f03ec13a6e0 0x7f03ec13ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:57.933 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03f0f19700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f03ec13a6e0 0x7f03ec13ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51586/0 (socket says 172.21.15.189:51586) 2020-11-20T12:19:57.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03f0f19700 1 -- 172.21.15.189:0/4020499875 learned_addr learned my addr 172.21.15.189:0/4020499875 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:19:57.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03e37fe700 1 --2- 172.21.15.189:0/4020499875 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f03ec0ad3e0 0x7f03ec066d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:57.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03ebfff700 1 --2- 172.21.15.189:0/4020499875 >> v2:172.21.15.47:3300/0 conn(0x7f03ec0ad000 0x7f03ec13e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:57.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03f0f19700 1 -- 172.21.15.189:0/4020499875 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f03ec0ad3e0 msgr2=0x7f03ec066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:57.935 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03f0f19700 1 --2- 172.21.15.189:0/4020499875 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f03ec0ad3e0 0x7f03ec066d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:57.935 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03f0f19700 1 -- 172.21.15.189:0/4020499875 >> v2:172.21.15.47:3300/0 conn(0x7f03ec0ad000 msgr2=0x7f03ec13e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:19:57.935 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03f0f19700 1 --2- 172.21.15.189:0/4020499875 >> v2:172.21.15.47:3300/0 conn(0x7f03ec0ad000 0x7f03ec13e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:57.935 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03f0f19700 1 -- 172.21.15.189:0/4020499875 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ec13dfe0 con 0x7f03ec13a6e0 2020-11-20T12:19:57.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03f0f19700 1 --2- 172.21.15.189:0/4020499875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ec13a6e0 0x7f03ec13ad00 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7f03ec065df0 tx=0x7f03e4009290).ready entity=mon.1 client_cookie=9734a52425bbd0b2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:57.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03eb7fe700 1 -- 172.21.15.189:0/4020499875 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e400bbc0 con 0x7f03ec13a6e0 2020-11-20T12:19:57.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03eb7fe700 1 -- 172.21.15.189:0/4020499875 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e400bd20 con 0x7f03ec13a6e0 2020-11-20T12:19:57.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03eb7fe700 1 -- 172.21.15.189:0/4020499875 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e4011610 con 0x7f03ec13a6e0 2020-11-20T12:19:57.937 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03f171a700 1 -- 172.21.15.189:0/4020499875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ec13a6e0 msgr2=0x7f03ec13ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:57.937 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03f171a700 1 --2- 172.21.15.189:0/4020499875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ec13a6e0 0x7f03ec13ad00 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7f03ec065df0 tx=0x7f03e4009290).stop 2020-11-20T12:19:57.937 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03f171a700 1 -- 172.21.15.189:0/4020499875 shutdown_connections 2020-11-20T12:19:57.937 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03f171a700 1 --2- 172.21.15.189:0/4020499875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ec13a6e0 0x7f03ec13ad00 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:57.938 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03f171a700 1 --2- 172.21.15.189:0/4020499875 >> v2:172.21.15.47:3300/0 conn(0x7f03ec0ad000 0x7f03ec13e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:57.938 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03f171a700 1 --2- 172.21.15.189:0/4020499875 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f03ec0ad3e0 0x7f03ec066d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:57.938 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03f171a700 1 -- 172.21.15.189:0/4020499875 >> 172.21.15.189:0/4020499875 conn(0x7f03ec062080 msgr2=0x7f03ec062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:57.938 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03f171a700 1 -- 172.21.15.189:0/4020499875 shutdown_connections 2020-11-20T12:19:57.939 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.924+0000 7f03f171a700 1 -- 172.21.15.189:0/4020499875 wait complete. 2020-11-20T12:19:57.939 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.928+0000 7f03f171a700 1 Processor -- start 2020-11-20T12:19:57.939 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.928+0000 7f03f171a700 1 -- start start 2020-11-20T12:19:57.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.928+0000 7f03f171a700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f03ec0ad000 0x7f03ec140fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:57.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.928+0000 7f03f171a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ec0ad3e0 0x7f03ec0aa950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:57.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.928+0000 7f03f171a700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f03ec0a2080 0x7f03ec14d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:57.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.928+0000 7f03f171a700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ec0aae40 con 0x7f03ec0ad000 2020-11-20T12:19:57.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.928+0000 7f03f171a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ec0a9680 con 0x7f03ec0ad3e0 2020-11-20T12:19:57.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.928+0000 7f03f171a700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03ec14ce80 con 0x7f03ec0a2080 2020-11-20T12:19:57.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.928+0000 7f03e37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ec0ad3e0 0x7f03ec0aa950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:57.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.928+0000 7f03e37fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ec0ad3e0 0x7f03ec0aa950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51588/0 (socket says 172.21.15.189:51588) 2020-11-20T12:19:57.943 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.928+0000 7f03e37fe700 1 -- 172.21.15.189:0/360025314 learned_addr learned my addr 172.21.15.189:0/360025314 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:19:57.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.928+0000 7f03f0f19700 1 --2- 172.21.15.189:0/360025314 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f03ec0a2080 0x7f03ec14d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:57.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.928+0000 7f03ebfff700 1 --2- 172.21.15.189:0/360025314 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f03ec0ad000 0x7f03ec140fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:57.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.928+0000 7f03e37fe700 1 -- 172.21.15.189:0/360025314 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f03ec0a2080 msgr2=0x7f03ec14d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:57.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.928+0000 7f03e37fe700 1 --2- 172.21.15.189:0/360025314 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f03ec0a2080 0x7f03ec14d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:57.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.928+0000 7f03e37fe700 1 -- 172.21.15.189:0/360025314 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f03ec0ad000 msgr2=0x7f03ec140fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:57.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.928+0000 7f03e37fe700 1 --2- 172.21.15.189:0/360025314 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f03ec0ad000 0x7f03ec140fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:57.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.928+0000 7f03e37fe700 1 -- 172.21.15.189:0/360025314 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ec0ab060 con 0x7f03ec0ad3e0 2020-11-20T12:19:57.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.928+0000 7f03e37fe700 1 --2- 172.21.15.189:0/360025314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ec0ad3e0 0x7f03ec0aa950 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f03dc00db60 tx=0x7f03dc008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:57.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.928+0000 7f03e8ff9700 1 -- 172.21.15.189:0/360025314 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03dc009790 con 0x7f03ec0ad3e0 2020-11-20T12:19:57.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.928+0000 7f03e8ff9700 1 -- 172.21.15.189:0/360025314 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03dc0098f0 con 0x7f03ec0ad3e0 2020-11-20T12:19:57.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.928+0000 7f03e8ff9700 1 -- 172.21.15.189:0/360025314 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03dc009c90 con 0x7f03ec0ad3e0 2020-11-20T12:19:57.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.928+0000 7f03f171a700 1 -- 172.21.15.189:0/360025314 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03ec14d0b0 con 0x7f03ec0ad3e0 2020-11-20T12:19:57.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.928+0000 7f03f171a700 1 -- 172.21.15.189:0/360025314 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03ec14dc60 con 0x7f03ec0ad3e0 2020-11-20T12:19:57.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.932+0000 7f03f171a700 1 -- 172.21.15.189:0/360025314 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03ec14d240 con 0x7f03ec0ad3e0 2020-11-20T12:19:57.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.932+0000 7f03e8ff9700 1 -- 172.21.15.189:0/360025314 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f03dc013480 con 0x7f03ec0ad3e0 2020-11-20T12:19:57.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.932+0000 7f03e8ff9700 1 --2- 172.21.15.189:0/360025314 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f03c80557c0 0x7f03c8057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:19:57.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.932+0000 7f03ebfff700 1 --2- 172.21.15.189:0/360025314 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f03c80557c0 0x7f03c8057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:19:57.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.932+0000 7f03e8ff9700 1 -- 172.21.15.189:0/360025314 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6933+0+0 (secure 0 0 0) 0x7f03dc06f0f0 con 0x7f03ec0ad3e0 2020-11-20T12:19:57.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.932+0000 7f03ebfff700 1 --2- 172.21.15.189:0/360025314 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f03c80557c0 0x7f03c8057c80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f03ec13a6b0 tx=0x7f03d4008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:19:57.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:57.944+0000 7f03e8ff9700 1 -- 172.21.15.189:0/360025314 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f03dc044ac0 con 0x7f03ec0ad3e0 2020-11-20T12:19:58.148 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:58.144+0000 7f03f171a700 1 -- 172.21.15.189:0/360025314 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03ec0b18e0 con 0x7f03ec0ad3e0 2020-11-20T12:19:58.150 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:58.144+0000 7f03e8ff9700 1 -- 172.21.15.189:0/360025314 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f03dc0480e0 con 0x7f03ec0ad3e0 2020-11-20T12:19:58.150 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:19:58.151 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5520,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/201662157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":201662157},{"type":"v1","addr":"172.21.15.47:6833","nonce":201662157}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:17:55.763199+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5572":{"gid":5572,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/344330560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":344330560},{"type":"v1","addr":"172.21.15.189:6835","nonce":344330560}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:19:58.152 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:58.144+0000 7f03f171a700 1 -- 172.21.15.189:0/360025314 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f03c80557c0 msgr2=0x7f03c8057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:58.152 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:58.144+0000 7f03f171a700 1 --2- 172.21.15.189:0/360025314 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f03c80557c0 0x7f03c8057c80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f03ec13a6b0 tx=0x7f03d4008040).stop 2020-11-20T12:19:58.153 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:58.144+0000 7f03f171a700 1 -- 172.21.15.189:0/360025314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ec0ad3e0 msgr2=0x7f03ec0aa950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:19:58.153 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:58.144+0000 7f03f171a700 1 --2- 172.21.15.189:0/360025314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ec0ad3e0 0x7f03ec0aa950 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f03dc00db60 tx=0x7f03dc008040).stop 2020-11-20T12:19:58.153 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:58.144+0000 7f03f171a700 1 -- 172.21.15.189:0/360025314 shutdown_connections 2020-11-20T12:19:58.154 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:58.148+0000 7f03f171a700 1 --2- 172.21.15.189:0/360025314 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f03c80557c0 0x7f03c8057c80 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:58.154 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:58.148+0000 7f03f171a700 1 --2- 172.21.15.189:0/360025314 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f03ec0ad000 0x7f03ec140fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:58.154 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:58.148+0000 7f03f171a700 1 --2- 172.21.15.189:0/360025314 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f03ec0a2080 0x7f03ec14d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:58.155 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:58.148+0000 7f03f171a700 1 --2- 172.21.15.189:0/360025314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03ec0ad3e0 0x7f03ec0aa950 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:19:58.155 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:58.148+0000 7f03f171a700 1 -- 172.21.15.189:0/360025314 >> 172.21.15.189:0/360025314 conn(0x7f03ec062080 msgr2=0x7f03ec13aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:19:58.155 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:58.148+0000 7f03f171a700 1 -- 172.21.15.189:0/360025314 shutdown_connections 2020-11-20T12:19:58.156 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:19:58.148+0000 7f03f171a700 1 -- 172.21.15.189:0/360025314 wait complete. 2020-11-20T12:19:58.157 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 56 2020-11-20T12:19:58.164 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:20:00.165 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:00.821 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.816+0000 7f069747f700 1 Processor -- start 2020-11-20T12:20:00.821 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.816+0000 7f069747f700 1 -- start start 2020-11-20T12:20:00.823 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.816+0000 7f069747f700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0690062220 0x7f069013a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:00.823 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.816+0000 7f069747f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0690062600 0x7f0690066fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:00.823 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.816+0000 7f069747f700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0690137750 0x7f0690137ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:00.823 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.816+0000 7f069747f700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06900674c0 con 0x7f0690137750 2020-11-20T12:20:00.824 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.816+0000 7f069747f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06900a9bb0 con 0x7f0690062600 2020-11-20T12:20:00.824 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.816+0000 7f069747f700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f069013ab40 con 0x7f0690062220 2020-11-20T12:20:00.824 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.816+0000 7f0695c7c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0690062600 0x7f0690066fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:00.824 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.816+0000 7f0695c7c700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0690062600 0x7f0690066fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51600/0 (socket says 172.21.15.189:51600) 2020-11-20T12:20:00.824 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.816+0000 7f0695c7c700 1 -- 172.21.15.189:0/3545949729 learned_addr learned my addr 172.21.15.189:0/3545949729 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:20:00.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.816+0000 7f0696c7e700 1 --2- 172.21.15.189:0/3545949729 >> v2:172.21.15.47:3300/0 conn(0x7f0690137750 0x7f0690137ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:00.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.816+0000 7f069647d700 1 --2- 172.21.15.189:0/3545949729 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0690062220 0x7f069013a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:00.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.816+0000 7f0695c7c700 1 -- 172.21.15.189:0/3545949729 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0690062220 msgr2=0x7f069013a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:00.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.816+0000 7f0695c7c700 1 --2- 172.21.15.189:0/3545949729 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0690062220 0x7f069013a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:00.828 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.816+0000 7f0695c7c700 1 -- 172.21.15.189:0/3545949729 >> v2:172.21.15.47:3300/0 conn(0x7f0690137750 msgr2=0x7f0690137ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:00.828 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.816+0000 7f0695c7c700 1 --2- 172.21.15.189:0/3545949729 >> v2:172.21.15.47:3300/0 conn(0x7f0690137750 0x7f0690137ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:00.828 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.816+0000 7f0695c7c700 1 -- 172.21.15.189:0/3545949729 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f069013ade0 con 0x7f0690062600 2020-11-20T12:20:00.828 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.816+0000 7f069647d700 1 --2- 172.21.15.189:0/3545949729 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0690062220 0x7f069013a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:00.829 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.816+0000 7f0695c7c700 1 --2- 172.21.15.189:0/3545949729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0690062600 0x7f0690066fd0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f06900ab390 tx=0x7f0684006eb0).ready entity=mon.1 client_cookie=89967170418b730c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:00.829 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f069547b700 1 -- 172.21.15.189:0/3545949729 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0684009960 con 0x7f0690062600 2020-11-20T12:20:00.829 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f069547b700 1 -- 172.21.15.189:0/3545949729 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0684009ac0 con 0x7f0690062600 2020-11-20T12:20:00.829 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f069547b700 1 -- 172.21.15.189:0/3545949729 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0684009e40 con 0x7f0690062600 2020-11-20T12:20:00.829 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f069747f700 1 -- 172.21.15.189:0/3545949729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0690062600 msgr2=0x7f0690066fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:00.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f069747f700 1 --2- 172.21.15.189:0/3545949729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0690062600 0x7f0690066fd0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f06900ab390 tx=0x7f0684006eb0).stop 2020-11-20T12:20:00.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f069747f700 1 -- 172.21.15.189:0/3545949729 shutdown_connections 2020-11-20T12:20:00.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f069747f700 1 --2- 172.21.15.189:0/3545949729 >> v2:172.21.15.47:3300/0 conn(0x7f0690137750 0x7f0690137ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:00.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f069747f700 1 --2- 172.21.15.189:0/3545949729 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0690062220 0x7f069013a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:00.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f069747f700 1 --2- 172.21.15.189:0/3545949729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0690062600 0x7f0690066fd0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:00.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f069747f700 1 -- 172.21.15.189:0/3545949729 >> 172.21.15.189:0/3545949729 conn(0x7f0690065fe0 msgr2=0x7f0690061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:00.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f069747f700 1 -- 172.21.15.189:0/3545949729 shutdown_connections 2020-11-20T12:20:00.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f069747f700 1 -- 172.21.15.189:0/3545949729 wait complete. 2020-11-20T12:20:00.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f069747f700 1 Processor -- start 2020-11-20T12:20:00.832 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f069747f700 1 -- start start 2020-11-20T12:20:00.832 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f069747f700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0690062220 0x7f069013dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:00.832 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f069747f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0690062600 0x7f0690134db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:00.832 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f069747f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0690137750 0x7f06901352a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:00.832 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f069747f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06900a2d50 con 0x7f0690137750 2020-11-20T12:20:00.833 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f069747f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06900a2ff0 con 0x7f0690062600 2020-11-20T12:20:00.833 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f069747f700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06900a3290 con 0x7f0690062220 2020-11-20T12:20:00.833 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f0695c7c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0690062600 0x7f0690134db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:00.833 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f0695c7c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0690062600 0x7f0690134db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51604/0 (socket says 172.21.15.189:51604) 2020-11-20T12:20:00.833 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f0695c7c700 1 -- 172.21.15.189:0/384825866 learned_addr learned my addr 172.21.15.189:0/384825866 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:20:00.834 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f069647d700 1 --2- 172.21.15.189:0/384825866 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0690062220 0x7f069013dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:00.834 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f0696c7e700 1 --2- 172.21.15.189:0/384825866 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0690137750 0x7f06901352a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:00.834 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f0695c7c700 1 -- 172.21.15.189:0/384825866 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0690062220 msgr2=0x7f069013dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:00.834 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f0695c7c700 1 --2- 172.21.15.189:0/384825866 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0690062220 0x7f069013dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:00.834 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f0695c7c700 1 -- 172.21.15.189:0/384825866 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0690137750 msgr2=0x7f06901352a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:00.835 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f0695c7c700 1 --2- 172.21.15.189:0/384825866 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0690137750 0x7f06901352a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:00.835 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.820+0000 7f0695c7c700 1 -- 172.21.15.189:0/384825866 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06900a3530 con 0x7f0690062600 2020-11-20T12:20:00.835 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.824+0000 7f0695c7c700 1 --2- 172.21.15.189:0/384825866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0690062600 0x7f0690134db0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7f0684006ef0 tx=0x7f0684015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:00.835 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.824+0000 7f06827fc700 1 -- 172.21.15.189:0/384825866 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0684011e50 con 0x7f0690062600 2020-11-20T12:20:00.835 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.824+0000 7f06827fc700 1 -- 172.21.15.189:0/384825866 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f068401e460 con 0x7f0690062600 2020-11-20T12:20:00.836 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.824+0000 7f069747f700 1 -- 172.21.15.189:0/384825866 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f069013e7d0 con 0x7f0690062600 2020-11-20T12:20:00.836 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.824+0000 7f06827fc700 1 -- 172.21.15.189:0/384825866 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f068401e7e0 con 0x7f0690062600 2020-11-20T12:20:00.836 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.824+0000 7f069747f700 1 -- 172.21.15.189:0/384825866 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06900a7cb0 con 0x7f0690062600 2020-11-20T12:20:00.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.824+0000 7f069747f700 1 -- 172.21.15.189:0/384825866 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06900663c0 con 0x7f0690062600 2020-11-20T12:20:00.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.824+0000 7f06827fc700 1 -- 172.21.15.189:0/384825866 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f0684025030 con 0x7f0690062600 2020-11-20T12:20:00.837 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.824+0000 7f06827fc700 1 --2- 172.21.15.189:0/384825866 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f06680557c0 0x7f0668057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:00.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.824+0000 7f06827fc700 1 -- 172.21.15.189:0/384825866 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6933+0+0 (secure 0 0 0) 0x7f06840702d0 con 0x7f0690062600 2020-11-20T12:20:00.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.824+0000 7f069647d700 1 --2- 172.21.15.189:0/384825866 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f06680557c0 0x7f0668057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:00.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.828+0000 7f069647d700 1 --2- 172.21.15.189:0/384825866 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f06680557c0 0x7f0668057c80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f068c006fd0 tx=0x7f068c008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:00.838 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:00.832+0000 7f06827fc700 1 -- 172.21.15.189:0/384825866 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0684045e10 con 0x7f0690062600 2020-11-20T12:20:01.038 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:01.032+0000 7f069747f700 1 -- 172.21.15.189:0/384825866 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0690065ca0 con 0x7f0690062600 2020-11-20T12:20:01.039 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:01.032+0000 7f06827fc700 1 -- 172.21.15.189:0/384825866 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2134 (secure 0 0 0) 0x7f0684023020 con 0x7f0690062600 2020-11-20T12:20:01.039 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:20:01.039 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5520,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.47:6833/201662157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":201662157},{"type":"v1","addr":"172.21.15.47:6833","nonce":201662157}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:17:55.763199+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5572},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5572":{"gid":5572,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":22,"addr":"172.21.15.189:6835/344330560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":344330560},{"type":"v1","addr":"172.21.15.189:6835","nonce":344330560}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:20:01.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:01.036+0000 7f069747f700 1 -- 172.21.15.189:0/384825866 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f06680557c0 msgr2=0x7f0668057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:01.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:01.036+0000 7f069747f700 1 --2- 172.21.15.189:0/384825866 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f06680557c0 0x7f0668057c80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f068c006fd0 tx=0x7f068c008040).stop 2020-11-20T12:20:01.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:01.036+0000 7f069747f700 1 -- 172.21.15.189:0/384825866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0690062600 msgr2=0x7f0690134db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:01.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:01.036+0000 7f069747f700 1 --2- 172.21.15.189:0/384825866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0690062600 0x7f0690134db0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7f0684006ef0 tx=0x7f0684015000).stop 2020-11-20T12:20:01.042 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:01.036+0000 7f069747f700 1 -- 172.21.15.189:0/384825866 shutdown_connections 2020-11-20T12:20:01.042 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:01.036+0000 7f069747f700 1 --2- 172.21.15.189:0/384825866 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f06680557c0 0x7f0668057c80 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:01.042 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:01.036+0000 7f069747f700 1 --2- 172.21.15.189:0/384825866 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0690137750 0x7f06901352a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:01.042 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:01.036+0000 7f069747f700 1 --2- 172.21.15.189:0/384825866 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0690062220 0x7f069013dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:01.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:01.036+0000 7f069747f700 1 --2- 172.21.15.189:0/384825866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0690062600 0x7f0690134db0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:01.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:01.036+0000 7f069747f700 1 -- 172.21.15.189:0/384825866 >> 172.21.15.189:0/384825866 conn(0x7f0690065fe0 msgr2=0x7f0690060d60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:01.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:01.036+0000 7f069747f700 1 -- 172.21.15.189:0/384825866 shutdown_connections 2020-11-20T12:20:01.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:01.036+0000 7f069747f700 1 -- 172.21.15.189:0/384825866 wait complete. 2020-11-20T12:20:01.044 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 56 2020-11-20T12:20:01.053 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:20:03.055 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:03.181 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.176+0000 7f350a3cd700 1 Processor -- start 2020-11-20T12:20:03.182 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.176+0000 7f350a3cd700 1 -- start start 2020-11-20T12:20:03.184 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.176+0000 7f350a3cd700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f35040aefb0 0x7f35040a9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:03.184 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.176+0000 7f350a3cd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f35040a9fd0 0x7f350413be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:03.184 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.176+0000 7f350a3cd700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35040a9680 con 0x7f35040aebd0 2020-11-20T12:20:03.185 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.176+0000 7f350a3cd700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35040aae50 con 0x7f35040aefb0 2020-11-20T12:20:03.185 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.176+0000 7f350a3cd700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f350413ba10 con 0x7f35040a9fd0 2020-11-20T12:20:03.185 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.176+0000 7f3509bcc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f35040a9fd0 0x7f350413be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:03.185 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.176+0000 7f3508bca700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f35040aefb0 0x7f35040a9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:03.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.176+0000 7f3509bcc700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f35040a9fd0 0x7f350413be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51614/0 (socket says 172.21.15.189:51614) 2020-11-20T12:20:03.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.176+0000 7f3509bcc700 1 -- 172.21.15.189:0/4064300123 learned_addr learned my addr 172.21.15.189:0/4064300123 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:20:03.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.176+0000 7f3509bcc700 1 -- 172.21.15.189:0/4064300123 >> v2:172.21.15.47:3300/0 conn(0x7f35040aefb0 msgr2=0x7f35040a9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:03.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.176+0000 7f3509bcc700 1 --2- 172.21.15.189:0/4064300123 >> v2:172.21.15.47:3300/0 conn(0x7f35040aefb0 0x7f35040a9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.176+0000 7f3509bcc700 1 -- 172.21.15.189:0/4064300123 >> v1:172.21.15.47:6789/0 conn(0x7f35040aebd0 legacy=0x7f35040aa620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:03.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.176+0000 7f3509bcc700 1 -- 172.21.15.189:0/4064300123 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35040ab0f0 con 0x7f35040a9fd0 2020-11-20T12:20:03.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.176+0000 7f3509bcc700 1 --2- 172.21.15.189:0/4064300123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35040a9fd0 0x7f350413be70 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f35040ac5e0 tx=0x7f34fc006eb0).ready entity=mon.1 client_cookie=a4b9e61d3e80118a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:03.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.180+0000 7f3503fff700 1 -- 172.21.15.189:0/4064300123 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34fc009810 con 0x7f35040a9fd0 2020-11-20T12:20:03.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.180+0000 7f3503fff700 1 -- 172.21.15.189:0/4064300123 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34fc009970 con 0x7f35040a9fd0 2020-11-20T12:20:03.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.180+0000 7f3503fff700 1 -- 172.21.15.189:0/4064300123 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34fc009cf0 con 0x7f35040a9fd0 2020-11-20T12:20:03.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.180+0000 7f350a3cd700 1 -- 172.21.15.189:0/4064300123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35040a9fd0 msgr2=0x7f350413be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:03.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.180+0000 7f350a3cd700 1 --2- 172.21.15.189:0/4064300123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35040a9fd0 0x7f350413be70 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f35040ac5e0 tx=0x7f34fc006eb0).stop 2020-11-20T12:20:03.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.180+0000 7f350a3cd700 1 -- 172.21.15.189:0/4064300123 shutdown_connections 2020-11-20T12:20:03.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.180+0000 7f350a3cd700 1 --2- 172.21.15.189:0/4064300123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35040a9fd0 0x7f350413be70 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.180+0000 7f350a3cd700 1 --2- 172.21.15.189:0/4064300123 >> v2:172.21.15.47:3300/0 conn(0x7f35040aefb0 0x7f35040a9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.180+0000 7f350a3cd700 1 -- 172.21.15.189:0/4064300123 >> 172.21.15.189:0/4064300123 conn(0x7f3504062080 msgr2=0x7f3504062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:03.194 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.180+0000 7f350a3cd700 1 -- 172.21.15.189:0/4064300123 shutdown_connections 2020-11-20T12:20:03.194 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.180+0000 7f350a3cd700 1 -- 172.21.15.189:0/4064300123 wait complete. 2020-11-20T12:20:03.194 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.184+0000 7f350a3cd700 1 Processor -- start 2020-11-20T12:20:03.194 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.184+0000 7f350a3cd700 1 -- start start 2020-11-20T12:20:03.194 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.184+0000 7f350a3cd700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f35040a9fd0 0x7f350409c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:03.195 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.184+0000 7f350a3cd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35040aebd0 0x7f35040a2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:03.195 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.184+0000 7f350a3cd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35040aefb0 0x7f3504134f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:03.195 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.184+0000 7f350a3cd700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3504138d30 con 0x7f35040aefb0 2020-11-20T12:20:03.195 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.184+0000 7f350a3cd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3504138fd0 con 0x7f35040aebd0 2020-11-20T12:20:03.195 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.184+0000 7f350a3cd700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f350413dde0 con 0x7f35040a9fd0 2020-11-20T12:20:03.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.184+0000 7f3508bca700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35040aebd0 0x7f35040a2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:03.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.184+0000 7f3508bca700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35040aebd0 0x7f35040a2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51616/0 (socket says 172.21.15.189:51616) 2020-11-20T12:20:03.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.184+0000 7f3508bca700 1 -- 172.21.15.189:0/3489810465 learned_addr learned my addr 172.21.15.189:0/3489810465 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:20:03.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.184+0000 7f35093cb700 1 --2- 172.21.15.189:0/3489810465 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f35040a9fd0 0x7f350409c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:03.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.184+0000 7f3509bcc700 1 --2- 172.21.15.189:0/3489810465 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35040aefb0 0x7f3504134f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:03.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.184+0000 7f3508bca700 1 -- 172.21.15.189:0/3489810465 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f35040a9fd0 msgr2=0x7f350409c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:03.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.184+0000 7f3508bca700 1 --2- 172.21.15.189:0/3489810465 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f35040a9fd0 0x7f350409c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.184+0000 7f3508bca700 1 -- 172.21.15.189:0/3489810465 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35040aefb0 msgr2=0x7f3504134f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:03.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.184+0000 7f3508bca700 1 --2- 172.21.15.189:0/3489810465 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35040aefb0 0x7f3504134f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.184+0000 7f3508bca700 1 -- 172.21.15.189:0/3489810465 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350413bd50 con 0x7f35040aebd0 2020-11-20T12:20:03.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.184+0000 7f3508bca700 1 --2- 172.21.15.189:0/3489810465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35040aebd0 0x7f35040a2360 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f34f400ecf0 tx=0x7f34f40091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:03.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.184+0000 7f35017fa700 1 -- 172.21.15.189:0/3489810465 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f400ca10 con 0x7f35040aebd0 2020-11-20T12:20:03.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.184+0000 7f35017fa700 1 -- 172.21.15.189:0/3489810465 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f400cb70 con 0x7f35040aebd0 2020-11-20T12:20:03.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.184+0000 7f350a3cd700 1 -- 172.21.15.189:0/3489810465 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3504135420 con 0x7f35040aebd0 2020-11-20T12:20:03.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.184+0000 7f35017fa700 1 -- 172.21.15.189:0/3489810465 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f40123f0 con 0x7f35040aebd0 2020-11-20T12:20:03.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.184+0000 7f350a3cd700 1 -- 172.21.15.189:0/3489810465 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f350414ab50 con 0x7f35040aebd0 2020-11-20T12:20:03.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.188+0000 7f350a3cd700 1 -- 172.21.15.189:0/3489810465 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3504139190 con 0x7f35040aebd0 2020-11-20T12:20:03.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.188+0000 7f35017fa700 1 -- 172.21.15.189:0/3489810465 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f34f4022070 con 0x7f35040aebd0 2020-11-20T12:20:03.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.188+0000 7f35017fa700 1 --2- 172.21.15.189:0/3489810465 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f34dc0557c0 0x7f34dc057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:03.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.188+0000 7f35017fa700 1 -- 172.21.15.189:0/3489810465 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7019+0+0 (secure 0 0 0) 0x7f34f406f0f0 con 0x7f35040aebd0 2020-11-20T12:20:03.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.188+0000 7f35093cb700 1 --2- 172.21.15.189:0/3489810465 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f34dc0557c0 0x7f34dc057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:03.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.188+0000 7f35093cb700 1 --2- 172.21.15.189:0/3489810465 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f34dc0557c0 0x7f34dc057c80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f34f8003610 tx=0x7f34f8000b80).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:03.202 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.196+0000 7f35017fa700 1 -- 172.21.15.189:0/3489810465 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f34f4044ad0 con 0x7f35040aebd0 2020-11-20T12:20:03.401 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.396+0000 7f350a3cd700 1 -- 172.21.15.189:0/3489810465 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f350413bee0 con 0x7f35040aebd0 2020-11-20T12:20:03.402 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.396+0000 7f35017fa700 1 -- 172.21.15.189:0/3489810465 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+1781 (secure 0 0 0) 0x7f34f4016020 con 0x7f35040aebd0 2020-11-20T12:20:03.402 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:20:03.403 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:20:02.783923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5520":{"gid":5520,"name":"b","rank":0,"incarnation":57,"state":"up:replay","state_seq":1,"addr":"172.21.15.47:6833/201662157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":201662157},{"type":"v1","addr":"172.21.15.47:6833","nonce":201662157}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:20:03.405 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.400+0000 7f350a3cd700 1 -- 172.21.15.189:0/3489810465 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f34dc0557c0 msgr2=0x7f34dc057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:03.405 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.400+0000 7f350a3cd700 1 --2- 172.21.15.189:0/3489810465 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f34dc0557c0 0x7f34dc057c80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f34f8003610 tx=0x7f34f8000b80).stop 2020-11-20T12:20:03.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.400+0000 7f350a3cd700 1 -- 172.21.15.189:0/3489810465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35040aebd0 msgr2=0x7f35040a2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:03.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.400+0000 7f350a3cd700 1 --2- 172.21.15.189:0/3489810465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35040aebd0 0x7f35040a2360 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f34f400ecf0 tx=0x7f34f40091d0).stop 2020-11-20T12:20:03.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.400+0000 7f350a3cd700 1 -- 172.21.15.189:0/3489810465 shutdown_connections 2020-11-20T12:20:03.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.400+0000 7f350a3cd700 1 --2- 172.21.15.189:0/3489810465 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f34dc0557c0 0x7f34dc057c80 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.400+0000 7f350a3cd700 1 --2- 172.21.15.189:0/3489810465 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f35040aefb0 0x7f3504134f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.400+0000 7f350a3cd700 1 --2- 172.21.15.189:0/3489810465 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f35040a9fd0 0x7f350409c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.400+0000 7f350a3cd700 1 --2- 172.21.15.189:0/3489810465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35040aebd0 0x7f35040a2360 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.400+0000 7f350a3cd700 1 -- 172.21.15.189:0/3489810465 >> 172.21.15.189:0/3489810465 conn(0x7f3504062080 msgr2=0x7f350413b020 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:03.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.400+0000 7f350a3cd700 1 -- 172.21.15.189:0/3489810465 shutdown_connections 2020-11-20T12:20:03.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.400+0000 7f350a3cd700 1 -- 172.21.15.189:0/3489810465 wait complete. 2020-11-20T12:20:03.409 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 57 2020-11-20T12:20:03.422 INFO:tasks.mds_thrash.fs.[None]:mds.a down, removed from mdsmap 2020-11-20T12:20:03.423 INFO:tasks.mds_thrash.fs.[None]:waiting for mds cluster to stabilize... 2020-11-20T12:20:03.423 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:03.543 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff189e87700 1 Processor -- start 2020-11-20T12:20:03.544 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff189e87700 1 -- start start 2020-11-20T12:20:03.547 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff189e87700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ff18413ab80 0x7ff18413e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:03.548 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff189e87700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff18413b300 con 0x7ff18413e6c0 2020-11-20T12:20:03.548 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff189e87700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff18413dd30 con 0x7ff184066de0 2020-11-20T12:20:03.548 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff189e87700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff18413aff0 con 0x7ff18413ab80 2020-11-20T12:20:03.549 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1837fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff184066de0 0x7ff1840671c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53246/0 (socket says 172.21.15.189:53246) 2020-11-20T12:20:03.549 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1837fe700 1 -- 172.21.15.189:0/3234667225 learned_addr learned my addr 172.21.15.189:0/3234667225 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:20:03.549 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff182ffd700 1 --2- 172.21.15.189:0/3234667225 >> v2:172.21.15.47:3300/0 conn(0x7ff18413ab80 0x7ff18413e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:03.549 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1827fc700 1 -- 172.21.15.189:0/3234667225 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763330661 0 0) 0x7ff18413dd30 con 0x7ff184066de0 2020-11-20T12:20:03.550 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1827fc700 1 -- 172.21.15.189:0/3234667225 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff170003240 con 0x7ff184066de0 2020-11-20T12:20:03.550 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1827fc700 1 -- 172.21.15.189:0/3234667225 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2602771640 0 0) 0x7ff170003240 con 0x7ff184066de0 2020-11-20T12:20:03.550 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1827fc700 1 -- 172.21.15.189:0/3234667225 >> v2:172.21.15.47:3300/0 conn(0x7ff18413ab80 msgr2=0x7ff18413e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:03.550 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1827fc700 1 --2- 172.21.15.189:0/3234667225 >> v2:172.21.15.47:3300/0 conn(0x7ff18413ab80 0x7ff18413e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.551 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1827fc700 1 -- 172.21.15.189:0/3234667225 >> v1:172.21.15.47:6789/0 conn(0x7ff18413e6c0 legacy=0x7ff18413cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:03.551 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1827fc700 1 -- 172.21.15.189:0/3234667225 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff18413d5b0 con 0x7ff184066de0 2020-11-20T12:20:03.551 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1827fc700 1 -- 172.21.15.189:0/3234667225 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7ff174001860 con 0x7ff184066de0 2020-11-20T12:20:03.552 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1827fc700 1 -- 172.21.15.189:0/3234667225 >> v1:172.21.15.189:6789/0 conn(0x7ff184066de0 legacy=0x7ff1840671c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:03.552 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1827fc700 1 --2- 172.21.15.189:0/3234667225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff170003730 0x7ff170003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:03.552 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1827fc700 1 --2- 172.21.15.189:0/3234667225 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff1700044e0 0x7ff170004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:03.552 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1827fc700 1 --2- 172.21.15.189:0/3234667225 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff170004e50 0x7ff17000b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:03.553 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1827fc700 1 -- 172.21.15.189:0/3234667225 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18413aff0 con 0x7ff170004e50 2020-11-20T12:20:03.553 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1827fc700 1 -- 172.21.15.189:0/3234667225 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff170004260 con 0x7ff170003730 2020-11-20T12:20:03.553 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1827fc700 1 -- 172.21.15.189:0/3234667225 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff17000bc20 con 0x7ff1700044e0 2020-11-20T12:20:03.553 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1837fe700 1 --2- 172.21.15.189:0/3234667225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff170003730 0x7ff170003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:03.554 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff182ffd700 1 --2- 172.21.15.189:0/3234667225 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff1700044e0 0x7ff170004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:03.554 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1837fe700 1 -- 172.21.15.189:0/3234667225 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff1700044e0 msgr2=0x7ff170004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:03.554 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1837fe700 1 --2- 172.21.15.189:0/3234667225 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff1700044e0 0x7ff170004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.554 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1837fe700 1 -- 172.21.15.189:0/3234667225 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff170004e50 msgr2=0x7ff17000b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:03.555 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1837fe700 1 --2- 172.21.15.189:0/3234667225 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff170004e50 0x7ff17000b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T12:20:03.555 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1837fe700 1 -- 172.21.15.189:0/3234667225 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff170003240 con 0x7ff170003730 2020-11-20T12:20:03.555 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff182ffd700 1 --2- 172.21.15.189:0/3234667225 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff1700044e0 0x7ff170004960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:03.556 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff183fff700 1 -- 172.21.15.189:0/3234667225 reap_dead start 2020-11-20T12:20:03.556 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1837fe700 1 --2- 172.21.15.189:0/3234667225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff170003730 0x7ff170003b50 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7ff17400c620 tx=0x7ff174007b50).ready entity=mon.1 client_cookie=c031666655b1211d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:03.556 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1827fc700 1 -- 172.21.15.189:0/3234667225 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff174005b80 con 0x7ff170003730 2020-11-20T12:20:03.557 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1827fc700 1 -- 172.21.15.189:0/3234667225 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff174005ce0 con 0x7ff170003730 2020-11-20T12:20:03.557 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.540+0000 7ff1827fc700 1 -- 172.21.15.189:0/3234667225 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1740072e0 con 0x7ff170003730 2020-11-20T12:20:03.557 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.544+0000 7ff189e87700 1 -- 172.21.15.189:0/3234667225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff170003730 msgr2=0x7ff170003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:03.557 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.544+0000 7ff189e87700 1 --2- 172.21.15.189:0/3234667225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff170003730 0x7ff170003b50 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7ff17400c620 tx=0x7ff174007b50).stop 2020-11-20T12:20:03.558 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.544+0000 7ff189e87700 1 -- 172.21.15.189:0/3234667225 shutdown_connections 2020-11-20T12:20:03.558 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.544+0000 7ff189e87700 1 --2- 172.21.15.189:0/3234667225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff170003730 0x7ff170003b50 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.558 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.544+0000 7ff189e87700 1 -- 172.21.15.189:0/3234667225 >> 172.21.15.189:0/3234667225 conn(0x7ff184061f60 msgr2=0x7ff184062340 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:03.559 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.544+0000 7ff189e87700 1 -- 172.21.15.189:0/3234667225 shutdown_connections 2020-11-20T12:20:03.559 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.544+0000 7ff189e87700 1 -- 172.21.15.189:0/3234667225 wait complete. 2020-11-20T12:20:03.562 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.544+0000 7ff189e87700 1 Processor -- start 2020-11-20T12:20:03.562 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.544+0000 7ff189e87700 1 -- start start 2020-11-20T12:20:03.562 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.544+0000 7ff189e87700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff170003730 0x7ff1840b0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:03.563 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.544+0000 7ff189e87700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff184149920 0x7ff1840a3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:03.563 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.544+0000 7ff189e87700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1840a08f0 0x7ff1840a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:03.563 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.544+0000 7ff189e87700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1840a1ec0 con 0x7ff170003730 2020-11-20T12:20:03.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.544+0000 7ff189e87700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1840a2160 con 0x7ff1840a08f0 2020-11-20T12:20:03.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.544+0000 7ff189e87700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1840a2400 con 0x7ff184149920 2020-11-20T12:20:03.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.544+0000 7ff183fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1840a08f0 0x7ff1840a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:03.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.544+0000 7ff183fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1840a08f0 0x7ff1840a0d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51640/0 (socket says 172.21.15.189:51640) 2020-11-20T12:20:03.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.544+0000 7ff183fff700 1 -- 172.21.15.189:0/3115143616 learned_addr learned my addr 172.21.15.189:0/3115143616 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:20:03.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.544+0000 7ff1837fe700 1 --2- 172.21.15.189:0/3115143616 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff170003730 0x7ff1840b0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:03.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.544+0000 7ff182ffd700 1 --2- 172.21.15.189:0/3115143616 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff184149920 0x7ff1840a3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:03.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.548+0000 7ff183fff700 1 -- 172.21.15.189:0/3115143616 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff184149920 msgr2=0x7ff1840a3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:03.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.548+0000 7ff183fff700 1 --2- 172.21.15.189:0/3115143616 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff184149920 0x7ff1840a3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.548+0000 7ff183fff700 1 -- 172.21.15.189:0/3115143616 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff170003730 msgr2=0x7ff1840b0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:03.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.548+0000 7ff183fff700 1 --2- 172.21.15.189:0/3115143616 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff170003730 0x7ff1840b0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.548+0000 7ff183fff700 1 -- 172.21.15.189:0/3115143616 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1840b1710 con 0x7ff1840a08f0 2020-11-20T12:20:03.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.548+0000 7ff183fff700 1 --2- 172.21.15.189:0/3115143616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1840a08f0 0x7ff1840a0d70 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7ff1841381a0 tx=0x7ff17c007880).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:03.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.548+0000 7ff1637fe700 1 -- 172.21.15.189:0/3115143616 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c00ce80 con 0x7ff1840a08f0 2020-11-20T12:20:03.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.548+0000 7ff1637fe700 1 -- 172.21.15.189:0/3115143616 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff17c007070 con 0x7ff1840a08f0 2020-11-20T12:20:03.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.548+0000 7ff189e87700 1 -- 172.21.15.189:0/3115143616 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff184149da0 con 0x7ff1840a08f0 2020-11-20T12:20:03.567 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.548+0000 7ff1637fe700 1 -- 172.21.15.189:0/3115143616 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c0054e0 con 0x7ff1840a08f0 2020-11-20T12:20:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.548+0000 7ff189e87700 1 -- 172.21.15.189:0/3115143616 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1840a2a40 con 0x7ff1840a08f0 2020-11-20T12:20:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.548+0000 7ff189e87700 1 -- 172.21.15.189:0/3115143616 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1840a25c0 con 0x7ff1840a08f0 2020-11-20T12:20:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.548+0000 7ff1637fe700 1 -- 172.21.15.189:0/3115143616 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7ff17c01b070 con 0x7ff1840a08f0 2020-11-20T12:20:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.548+0000 7ff1637fe700 1 --2- 172.21.15.189:0/3115143616 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff15c0557f0 0x7ff15c057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:03.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.548+0000 7ff1637fe700 1 -- 172.21.15.189:0/3115143616 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7019+0+0 (secure 0 0 0) 0x7ff17c06dab0 con 0x7ff1840a08f0 2020-11-20T12:20:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.548+0000 7ff1837fe700 1 --2- 172.21.15.189:0/3115143616 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff15c0557f0 0x7ff15c057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:03.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.548+0000 7ff1837fe700 1 --2- 172.21.15.189:0/3115143616 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff15c0557f0 0x7ff15c057cb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7ff17400e260 tx=0x7ff174015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:03.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.560+0000 7ff1637fe700 1 -- 172.21.15.189:0/3115143616 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff17c0435a0 con 0x7ff1840a08f0 2020-11-20T12:20:03.770 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.764+0000 7ff189e87700 1 -- 172.21.15.189:0/3115143616 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff184149f90 con 0x7ff1840a08f0 2020-11-20T12:20:03.771 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.764+0000 7ff1637fe700 1 -- 172.21.15.189:0/3115143616 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+1781 (secure 0 0 0) 0x7ff17c046bc0 con 0x7ff1840a08f0 2020-11-20T12:20:03.771 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:20:03.771 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:20:02.783923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5520":{"gid":5520,"name":"b","rank":0,"incarnation":57,"state":"up:replay","state_seq":1,"addr":"172.21.15.47:6833/201662157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":201662157},{"type":"v1","addr":"172.21.15.47:6833","nonce":201662157}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:20:03.773 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.768+0000 7ff189e87700 1 -- 172.21.15.189:0/3115143616 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff15c0557f0 msgr2=0x7ff15c057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:03.773 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.768+0000 7ff189e87700 1 --2- 172.21.15.189:0/3115143616 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff15c0557f0 0x7ff15c057cb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7ff17400e260 tx=0x7ff174015000).stop 2020-11-20T12:20:03.773 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.768+0000 7ff189e87700 1 -- 172.21.15.189:0/3115143616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1840a08f0 msgr2=0x7ff1840a0d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:03.773 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.768+0000 7ff189e87700 1 --2- 172.21.15.189:0/3115143616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1840a08f0 0x7ff1840a0d70 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7ff1841381a0 tx=0x7ff17c007880).stop 2020-11-20T12:20:03.774 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.768+0000 7ff189e87700 1 -- 172.21.15.189:0/3115143616 shutdown_connections 2020-11-20T12:20:03.774 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.768+0000 7ff189e87700 1 --2- 172.21.15.189:0/3115143616 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff15c0557f0 0x7ff15c057cb0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.774 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.768+0000 7ff189e87700 1 --2- 172.21.15.189:0/3115143616 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff170003730 0x7ff1840b0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.774 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.768+0000 7ff189e87700 1 --2- 172.21.15.189:0/3115143616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1840a08f0 0x7ff1840a0d70 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.774 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.768+0000 7ff189e87700 1 --2- 172.21.15.189:0/3115143616 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff184149920 0x7ff1840a3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:03.775 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.768+0000 7ff189e87700 1 -- 172.21.15.189:0/3115143616 >> 172.21.15.189:0/3115143616 conn(0x7ff184061f60 msgr2=0x7ff184140e00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:03.775 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.768+0000 7ff189e87700 1 -- 172.21.15.189:0/3115143616 shutdown_connections 2020-11-20T12:20:03.775 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:03.768+0000 7ff189e87700 1 -- 172.21.15.189:0/3115143616 wait complete. 2020-11-20T12:20:03.775 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 57 2020-11-20T12:20:03.786 INFO:tasks.mds_thrash.fs.[None]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 57, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2020-11-20T12:04:54.012110+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 57, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5520": { "addr": "172.21.15.47:6833/201662157", "addrs": { "addrvec": [ { "addr": "172.21.15.47:6832", "nonce": 201662157, "type": "v2" }, { "addr": "172.21.15.47:6833", "nonce": 201662157, "type": "v1" } ] }, "export_targets": [], "features": 4540138297136906239, "flags": 0, "gid": 5520, "incarnation": 57, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 33, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2020-11-20T12:20:02.783923+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5520 } } } ], "standbys": [] } 2020-11-20T12:20:05.787 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:05.932 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa0a2312700 1 Processor -- start 2020-11-20T12:20:05.932 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa0a2312700 1 -- start start 2020-11-20T12:20:05.933 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa0a2312700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa09c061040 0x7fa09c1354f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:05.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa0a2312700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa09c065ca0 con 0x7fa09c060bc0 2020-11-20T12:20:05.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa0a2312700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa09c138ba0 con 0x7fa09c0623c0 2020-11-20T12:20:05.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa0a2312700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c0687e0 con 0x7fa09c061040 2020-11-20T12:20:05.935 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa0a1310700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa09c0623c0 0x7fa09c138290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53268/0 (socket says 172.21.15.189:53268) 2020-11-20T12:20:05.935 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa0a1310700 1 -- 172.21.15.189:0/322613551 learned_addr learned my addr 172.21.15.189:0/322613551 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:20:05.935 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa0a1b11700 1 --2- 172.21.15.189:0/322613551 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa09c061040 0x7fa09c1354f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:05.935 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa08bfff700 1 -- 172.21.15.189:0/322613551 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015950101 0 0) 0x7fa09c138ba0 con 0x7fa09c0623c0 2020-11-20T12:20:05.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa08bfff700 1 -- 172.21.15.189:0/322613551 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa084003240 con 0x7fa09c0623c0 2020-11-20T12:20:05.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa08bfff700 1 -- 172.21.15.189:0/322613551 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193212920 0 0) 0x7fa09c065ca0 con 0x7fa09c060bc0 2020-11-20T12:20:05.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa08bfff700 1 -- 172.21.15.189:0/322613551 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa09c138ba0 con 0x7fa09c060bc0 2020-11-20T12:20:05.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa08bfff700 1 -- 172.21.15.189:0/322613551 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1044432776 0 0) 0x7fa084003240 con 0x7fa09c0623c0 2020-11-20T12:20:05.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa08bfff700 1 -- 172.21.15.189:0/322613551 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa09c061040 msgr2=0x7fa09c1354f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:05.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa08bfff700 1 --2- 172.21.15.189:0/322613551 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa09c061040 0x7fa09c1354f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:05.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa08bfff700 1 -- 172.21.15.189:0/322613551 >> v1:172.21.15.47:6789/0 conn(0x7fa09c060bc0 legacy=0x7fa09c134de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:05.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa08bfff700 1 -- 172.21.15.189:0/322613551 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09c138e40 con 0x7fa09c0623c0 2020-11-20T12:20:05.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa08bfff700 1 -- 172.21.15.189:0/322613551 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7fa0980014e0 con 0x7fa09c0623c0 2020-11-20T12:20:05.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa08bfff700 1 -- 172.21.15.189:0/322613551 >> v1:172.21.15.189:6789/0 conn(0x7fa09c0623c0 legacy=0x7fa09c138290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:05.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa08bfff700 1 --2- 172.21.15.189:0/322613551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0840038c0 0x7fa084003cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:05.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa08bfff700 1 --2- 172.21.15.189:0/322613551 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0840044e0 0x7fa084004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:05.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa08bfff700 1 --2- 172.21.15.189:0/322613551 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa09c061040 0x7fa084004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:05.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa0a1310700 1 --2- 172.21.15.189:0/322613551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0840038c0 0x7fa084003cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:05.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa08bfff700 1 -- 172.21.15.189:0/322613551 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa084005340 con 0x7fa0840044e0 2020-11-20T12:20:05.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa08bfff700 1 -- 172.21.15.189:0/322613551 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08400b730 con 0x7fa0840038c0 2020-11-20T12:20:05.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa08bfff700 1 -- 172.21.15.189:0/322613551 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa08400b8c0 con 0x7fa09c061040 2020-11-20T12:20:05.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa0a0b0f700 1 --2- 172.21.15.189:0/322613551 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0840044e0 0x7fa084004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:05.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa0a1b11700 1 --2- 172.21.15.189:0/322613551 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa09c061040 0x7fa084004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:05.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa0a1310700 1 -- 172.21.15.189:0/322613551 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa09c061040 msgr2=0x7fa084004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:05.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa0a1310700 1 --2- 172.21.15.189:0/322613551 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa09c061040 0x7fa084004e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:05.954 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa0a1310700 1 -- 172.21.15.189:0/322613551 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0840044e0 msgr2=0x7fa084004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:05.954 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa0a1310700 1 --2- 172.21.15.189:0/322613551 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa0840044e0 0x7fa084004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:05.954 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.928+0000 7fa0a1310700 1 -- 172.21.15.189:0/322613551 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa084003240 con 0x7fa0840038c0 2020-11-20T12:20:05.955 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.932+0000 7fa0a1310700 1 --2- 172.21.15.189:0/322613551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0840038c0 0x7fa084003cc0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7fa09800e5a0 tx=0x7fa09c065e30).ready entity=mon.1 client_cookie=de1d1544964de945 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:05.955 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.932+0000 7fa08bfff700 1 -- 172.21.15.189:0/322613551 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa098013070 con 0x7fa0840038c0 2020-11-20T12:20:05.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.932+0000 7fa08bfff700 1 -- 172.21.15.189:0/322613551 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0980070d0 con 0x7fa0840038c0 2020-11-20T12:20:05.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.932+0000 7fa08bfff700 1 -- 172.21.15.189:0/322613551 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa098007260 con 0x7fa0840038c0 2020-11-20T12:20:05.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.932+0000 7fa0a2312700 1 -- 172.21.15.189:0/322613551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0840038c0 msgr2=0x7fa084003cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:05.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.932+0000 7fa0a2312700 1 --2- 172.21.15.189:0/322613551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0840038c0 0x7fa084003cc0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7fa09800e5a0 tx=0x7fa09c065e30).stop 2020-11-20T12:20:05.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.932+0000 7fa0a1b11700 1 -- 172.21.15.189:0/322613551 reap_dead start 2020-11-20T12:20:05.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.932+0000 7fa0a2312700 1 -- 172.21.15.189:0/322613551 shutdown_connections 2020-11-20T12:20:05.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.932+0000 7fa0a2312700 1 -- 172.21.15.189:0/322613551 >> 172.21.15.189:0/322613551 conn(0x7fa09c065fe0 msgr2=0x7fa09c061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:05.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.932+0000 7fa0a2312700 1 -- 172.21.15.189:0/322613551 shutdown_connections 2020-11-20T12:20:05.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.932+0000 7fa0a2312700 1 -- 172.21.15.189:0/322613551 wait complete. 2020-11-20T12:20:05.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.932+0000 7fa0a2312700 1 Processor -- start 2020-11-20T12:20:05.960 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.932+0000 7fa0a2312700 1 -- start start 2020-11-20T12:20:05.960 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.932+0000 7fa0a2312700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa09c0a9910 0x7fa09c131e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:05.960 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.932+0000 7fa0a2312700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa09c0a9cf0 0x7fa09c14df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:05.961 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.932+0000 7fa0a2312700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa09c132370 0x7fa09c14e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:05.961 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.932+0000 7fa0a2312700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c0a8390 con 0x7fa09c132370 2020-11-20T12:20:05.961 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.932+0000 7fa0a2312700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c0a8630 con 0x7fa09c0a9cf0 2020-11-20T12:20:05.962 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.932+0000 7fa0a2312700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c0a88d0 con 0x7fa09c0a9910 2020-11-20T12:20:05.962 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.932+0000 7fa0a0b0f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa09c0a9cf0 0x7fa09c14df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:05.962 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.932+0000 7fa0a0b0f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa09c0a9cf0 0x7fa09c14df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51660/0 (socket says 172.21.15.189:51660) 2020-11-20T12:20:05.962 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.932+0000 7fa0a0b0f700 1 -- 172.21.15.189:0/842314956 learned_addr learned my addr 172.21.15.189:0/842314956 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:20:05.962 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.932+0000 7fa0a1310700 1 --2- 172.21.15.189:0/842314956 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa09c0a9910 0x7fa09c131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:05.963 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.932+0000 7fa0a1b11700 1 --2- 172.21.15.189:0/842314956 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa09c132370 0x7fa09c14e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:05.963 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.936+0000 7fa0a0b0f700 1 -- 172.21.15.189:0/842314956 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa09c0a9910 msgr2=0x7fa09c131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:05.963 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.936+0000 7fa0a0b0f700 1 --2- 172.21.15.189:0/842314956 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa09c0a9910 0x7fa09c131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:05.963 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.936+0000 7fa0a0b0f700 1 -- 172.21.15.189:0/842314956 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa09c132370 msgr2=0x7fa09c14e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:05.963 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.936+0000 7fa0a0b0f700 1 --2- 172.21.15.189:0/842314956 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa09c132370 0x7fa09c14e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:05.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.936+0000 7fa0a0b0f700 1 -- 172.21.15.189:0/842314956 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09c0a4900 con 0x7fa09c0a9cf0 2020-11-20T12:20:05.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.936+0000 7fa0a0b0f700 1 --2- 172.21.15.189:0/842314956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa09c0a9cf0 0x7fa09c14df60 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7fa08c00eae0 tx=0x7fa08c009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:05.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.936+0000 7fa0897fa700 1 -- 172.21.15.189:0/842314956 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08c00c7b0 con 0x7fa09c0a9cf0 2020-11-20T12:20:05.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.936+0000 7fa0a2312700 1 -- 172.21.15.189:0/842314956 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa09c137a10 con 0x7fa09c0a9cf0 2020-11-20T12:20:05.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.936+0000 7fa0897fa700 1 -- 172.21.15.189:0/842314956 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa08c00c910 con 0x7fa09c0a9cf0 2020-11-20T12:20:05.965 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.936+0000 7fa0897fa700 1 -- 172.21.15.189:0/842314956 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08c00cc90 con 0x7fa09c0a9cf0 2020-11-20T12:20:05.965 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.936+0000 7fa0a2312700 1 -- 172.21.15.189:0/842314956 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa09c0a8120 con 0x7fa09c0a9cf0 2020-11-20T12:20:05.965 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.936+0000 7fa0897fa700 1 -- 172.21.15.189:0/842314956 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fa08c01b030 con 0x7fa09c0a9cf0 2020-11-20T12:20:05.965 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.936+0000 7fa0a2312700 1 -- 172.21.15.189:0/842314956 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa09c0a8a90 con 0x7fa09c0a9cf0 2020-11-20T12:20:05.965 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.936+0000 7fa0897fa700 1 --2- 172.21.15.189:0/842314956 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa0740557f0 0x7fa074057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:05.966 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.936+0000 7fa0897fa700 1 -- 172.21.15.189:0/842314956 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7019+0+0 (secure 0 0 0) 0x7fa08c06e870 con 0x7fa09c0a9cf0 2020-11-20T12:20:05.966 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.936+0000 7fa0a1310700 1 --2- 172.21.15.189:0/842314956 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa0740557f0 0x7fa074057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:05.966 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.936+0000 7fa0a1310700 1 --2- 172.21.15.189:0/842314956 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa0740557f0 0x7fa074057cb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fa098010000 tx=0x7fa098017000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:05.967 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:05.944+0000 7fa0897fa700 1 -- 172.21.15.189:0/842314956 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa08c044410 con 0x7fa09c0a9cf0 2020-11-20T12:20:06.172 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:06.160+0000 7fa0a2312700 1 -- 172.21.15.189:0/842314956 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa09c0a8c90 con 0x7fa09c0a9cf0 2020-11-20T12:20:06.173 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:06.160+0000 7fa0897fa700 1 -- 172.21.15.189:0/842314956 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+1782 (secure 0 0 0) 0x7fa08c047a30 con 0x7fa09c0a9cf0 2020-11-20T12:20:06.173 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:06.160+0000 7fa0a2312700 1 -- 172.21.15.189:0/842314956 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa0740557f0 msgr2=0x7fa074057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:06.174 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:06.160+0000 7fa0a2312700 1 --2- 172.21.15.189:0/842314956 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa0740557f0 0x7fa074057cb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fa098010000 tx=0x7fa098017000).stop 2020-11-20T12:20:06.174 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:06.160+0000 7fa0a2312700 1 -- 172.21.15.189:0/842314956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa09c0a9cf0 msgr2=0x7fa09c14df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:06.174 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:06.160+0000 7fa0a2312700 1 --2- 172.21.15.189:0/842314956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa09c0a9cf0 0x7fa09c14df60 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7fa08c00eae0 tx=0x7fa08c009040).stop 2020-11-20T12:20:06.175 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:06.160+0000 7fa0a2312700 1 -- 172.21.15.189:0/842314956 shutdown_connections 2020-11-20T12:20:06.175 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:06.160+0000 7fa0a2312700 1 --2- 172.21.15.189:0/842314956 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa0740557f0 0x7fa074057cb0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:06.175 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:06.160+0000 7fa0a2312700 1 --2- 172.21.15.189:0/842314956 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa09c132370 0x7fa09c14e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:06.175 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:06.160+0000 7fa0a2312700 1 --2- 172.21.15.189:0/842314956 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa09c0a9910 0x7fa09c131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:06.176 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:06.160+0000 7fa0a2312700 1 --2- 172.21.15.189:0/842314956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa09c0a9cf0 0x7fa09c14df60 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:06.176 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:06.160+0000 7fa0a2312700 1 -- 172.21.15.189:0/842314956 >> 172.21.15.189:0/842314956 conn(0x7fa09c065fe0 msgr2=0x7fa09c132c30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:06.176 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:06.160+0000 7fa0a2312700 1 -- 172.21.15.189:0/842314956 shutdown_connections 2020-11-20T12:20:06.177 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:06.160+0000 7fa0a2312700 1 -- 172.21.15.189:0/842314956 wait complete. 2020-11-20T12:20:06.177 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 60 2020-11-20T12:20:06.178 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:20:06.178 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:20:05.807690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5520":{"gid":5520,"name":"b","rank":0,"incarnation":57,"state":"up:active","state_seq":32,"addr":"172.21.15.47:6833/201662157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":201662157},{"type":"v1","addr":"172.21.15.47:6833","nonce":201662157}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:20:06.180 INFO:tasks.mds_thrash.fs.[None]:mds.b has gained rank=0, replacing gid=5572 2020-11-20T12:20:06.181 INFO:tasks.mds_thrash.fs.[None]:waiting for 7 secs before reviving mds.a 2020-11-20T12:20:07.644 INFO:tasks.ceph.mds.b.smithi047.stderr:2020-11-20T12:20:07.635+0000 7f0e5d196700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:20:13.181 INFO:tasks.mds_thrash.fs.[None]:reviving mds.a 2020-11-20T12:20:13.183 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:20:13.184 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:20:13.194 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:20:13.196 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:13.269 INFO:tasks.ceph.mds.a.smithi189.stdout:starting mds.a at 2020-11-20T12:20:13.340 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8fffff700 1 Processor -- start 2020-11-20T12:20:13.341 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8fffff700 1 -- start start 2020-11-20T12:20:13.352 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8fffff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff9000623c0 0x7ff900138290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:13.353 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8fffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff900061040 0x7ff90013bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:13.353 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8fffff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff900065ca0 con 0x7ff900060bc0 2020-11-20T12:20:13.354 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8fffff700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9000687e0 con 0x7ff900061040 2020-11-20T12:20:13.355 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8fffff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff900138ba0 con 0x7ff9000623c0 2020-11-20T12:20:13.355 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8fe7fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff900060bc0 0x7ff90013b860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53302/0 (socket says 172.21.15.189:53302) 2020-11-20T12:20:13.356 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8fe7fc700 1 -- 172.21.15.189:0/326361304 learned_addr learned my addr 172.21.15.189:0/326361304 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:20:13.357 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8ff7fe700 1 --2- 172.21.15.189:0/326361304 >> v2:172.21.15.189:3300/0 conn(0x7ff900061040 0x7ff90013bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:13.358 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8feffd700 1 --2- 172.21.15.189:0/326361304 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff9000623c0 0x7ff900138290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:13.359 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8fdffb700 1 -- 172.21.15.189:0/326361304 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3998368342 0 0) 0x7ff900065ca0 con 0x7ff900060bc0 2020-11-20T12:20:13.360 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8fdffb700 1 -- 172.21.15.189:0/326361304 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8ec003240 con 0x7ff900060bc0 2020-11-20T12:20:13.360 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8ff7fe700 1 -- 172.21.15.189:0/326361304 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff9000623c0 msgr2=0x7ff900138290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:13.361 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8ff7fe700 1 --2- 172.21.15.189:0/326361304 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff9000623c0 0x7ff900138290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:13.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8ff7fe700 1 -- 172.21.15.189:0/326361304 >> v1:172.21.15.189:6789/0 conn(0x7ff900060bc0 legacy=0x7ff90013b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:13.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8ff7fe700 1 -- 172.21.15.189:0/326361304 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff900138e40 con 0x7ff900061040 2020-11-20T12:20:13.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8ff7fe700 1 --2- 172.21.15.189:0/326361304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff900061040 0x7ff90013bf70 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7ff8f8009c90 tx=0x7ff8f80091d0).ready entity=mon.1 client_cookie=4b7061a35073f618 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:13.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8fdffb700 1 -- 172.21.15.189:0/326361304 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f800ba50 con 0x7ff900061040 2020-11-20T12:20:13.363 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8fdffb700 1 -- 172.21.15.189:0/326361304 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f800bbb0 con 0x7ff900061040 2020-11-20T12:20:13.364 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8fdffb700 1 -- 172.21.15.189:0/326361304 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f8011400 con 0x7ff900061040 2020-11-20T12:20:13.364 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8fffff700 1 -- 172.21.15.189:0/326361304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff900061040 msgr2=0x7ff90013bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:13.364 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8fffff700 1 --2- 172.21.15.189:0/326361304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff900061040 0x7ff90013bf70 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7ff8f8009c90 tx=0x7ff8f80091d0).stop 2020-11-20T12:20:13.364 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8fffff700 1 -- 172.21.15.189:0/326361304 shutdown_connections 2020-11-20T12:20:13.364 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8fffff700 1 --2- 172.21.15.189:0/326361304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff900061040 0x7ff90013bf70 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:13.365 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8fffff700 1 --2- 172.21.15.189:0/326361304 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff9000623c0 0x7ff900138290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:13.365 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8fffff700 1 -- 172.21.15.189:0/326361304 >> 172.21.15.189:0/326361304 conn(0x7ff900065fe0 msgr2=0x7ff900061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:13.365 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8fffff700 1 -- 172.21.15.189:0/326361304 shutdown_connections 2020-11-20T12:20:13.365 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.336+0000 7ff8fffff700 1 -- 172.21.15.189:0/326361304 wait complete. 2020-11-20T12:20:13.365 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.340+0000 7ff8fffff700 1 Processor -- start 2020-11-20T12:20:13.366 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.340+0000 7ff8fffff700 1 -- start start 2020-11-20T12:20:13.366 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.340+0000 7ff8fffff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff900060bc0 0x7ff9000aac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:13.366 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.340+0000 7ff8fffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff900061040 0x7ff900139ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:13.366 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.340+0000 7ff8fffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff9000623c0 0x7ff90014ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:13.366 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.340+0000 7ff8fffff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90014c9d0 con 0x7ff9000623c0 2020-11-20T12:20:13.367 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.340+0000 7ff8fffff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff900139690 con 0x7ff900061040 2020-11-20T12:20:13.367 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.340+0000 7ff8fffff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff90013aea0 con 0x7ff900060bc0 2020-11-20T12:20:13.367 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.340+0000 7ff8fe7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff900061040 0x7ff900139ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:13.367 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.340+0000 7ff8fe7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff900061040 0x7ff900139ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51686/0 (socket says 172.21.15.189:51686) 2020-11-20T12:20:13.367 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.340+0000 7ff8fe7fc700 1 -- 172.21.15.189:0/1869465459 learned_addr learned my addr 172.21.15.189:0/1869465459 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:20:13.368 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.340+0000 7ff8feffd700 1 --2- 172.21.15.189:0/1869465459 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff900060bc0 0x7ff9000aac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:13.368 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.340+0000 7ff8ff7fe700 1 --2- 172.21.15.189:0/1869465459 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff9000623c0 0x7ff90014ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:13.368 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.340+0000 7ff8fe7fc700 1 -- 172.21.15.189:0/1869465459 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff900060bc0 msgr2=0x7ff9000aac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:13.368 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.340+0000 7ff8fe7fc700 1 --2- 172.21.15.189:0/1869465459 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff900060bc0 0x7ff9000aac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:13.368 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.340+0000 7ff8fe7fc700 1 -- 172.21.15.189:0/1869465459 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff9000623c0 msgr2=0x7ff90014ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:13.369 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.340+0000 7ff8fe7fc700 1 --2- 172.21.15.189:0/1869465459 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff9000623c0 0x7ff90014ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:13.369 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.340+0000 7ff8fe7fc700 1 -- 172.21.15.189:0/1869465459 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff90013b140 con 0x7ff900061040 2020-11-20T12:20:13.369 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.340+0000 7ff8fe7fc700 1 --2- 172.21.15.189:0/1869465459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff900061040 0x7ff900139ac0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7ff8f400c740 tx=0x7ff8f4007a60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:13.369 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.340+0000 7ff8e6ffd700 1 -- 172.21.15.189:0/1869465459 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f40072e0 con 0x7ff900061040 2020-11-20T12:20:13.370 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.340+0000 7ff8e6ffd700 1 -- 172.21.15.189:0/1869465459 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f4007440 con 0x7ff900061040 2020-11-20T12:20:13.370 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.340+0000 7ff8fffff700 1 -- 172.21.15.189:0/1869465459 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9000a6340 con 0x7ff900061040 2020-11-20T12:20:13.370 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.340+0000 7ff8e6ffd700 1 -- 172.21.15.189:0/1869465459 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f4005820 con 0x7ff900061040 2020-11-20T12:20:13.370 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.340+0000 7ff8fffff700 1 -- 172.21.15.189:0/1869465459 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9000a0a90 con 0x7ff900061040 2020-11-20T12:20:13.370 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.344+0000 7ff8fffff700 1 -- 172.21.15.189:0/1869465459 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff900139800 con 0x7ff900061040 2020-11-20T12:20:13.370 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.344+0000 7ff8e6ffd700 1 -- 172.21.15.189:0/1869465459 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7ff8f4010070 con 0x7ff900061040 2020-11-20T12:20:13.371 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.344+0000 7ff8e6ffd700 1 --2- 172.21.15.189:0/1869465459 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff8dc0557f0 0x7ff8dc057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:13.371 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.344+0000 7ff8e6ffd700 1 -- 172.21.15.189:0/1869465459 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7019+0+0 (secure 0 0 0) 0x7ff8f406e110 con 0x7ff900061040 2020-11-20T12:20:13.371 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.344+0000 7ff8feffd700 1 --2- 172.21.15.189:0/1869465459 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff8dc0557f0 0x7ff8dc057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:13.371 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.344+0000 7ff8feffd700 1 --2- 172.21.15.189:0/1869465459 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff8dc0557f0 0x7ff8dc057cb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7ff8f0006fd0 tx=0x7ff8f0008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:13.372 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.348+0000 7ff8e6ffd700 1 -- 172.21.15.189:0/1869465459 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff8f4043c00 con 0x7ff900061040 2020-11-20T12:20:13.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.572+0000 7ff8fffff700 1 -- 172.21.15.189:0/1869465459 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9000663c0 con 0x7ff900061040 2020-11-20T12:20:13.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.576+0000 7ff8e6ffd700 1 -- 172.21.15.189:0/1869465459 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+1782 (secure 0 0 0) 0x7ff8f4047220 con 0x7ff900061040 2020-11-20T12:20:13.583 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:20:13.584 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:20:05.807690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5520":{"gid":5520,"name":"b","rank":0,"incarnation":57,"state":"up:active","state_seq":32,"addr":"172.21.15.47:6833/201662157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":201662157},{"type":"v1","addr":"172.21.15.47:6833","nonce":201662157}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:20:13.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.576+0000 7ff8fffff700 1 -- 172.21.15.189:0/1869465459 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff8dc0557f0 msgr2=0x7ff8dc057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:13.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.576+0000 7ff8fffff700 1 --2- 172.21.15.189:0/1869465459 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff8dc0557f0 0x7ff8dc057cb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7ff8f0006fd0 tx=0x7ff8f0008040).stop 2020-11-20T12:20:13.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.576+0000 7ff8fffff700 1 -- 172.21.15.189:0/1869465459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff900061040 msgr2=0x7ff900139ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:13.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.576+0000 7ff8fffff700 1 --2- 172.21.15.189:0/1869465459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff900061040 0x7ff900139ac0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7ff8f400c740 tx=0x7ff8f4007a60).stop 2020-11-20T12:20:13.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.576+0000 7ff8fffff700 1 -- 172.21.15.189:0/1869465459 shutdown_connections 2020-11-20T12:20:13.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.576+0000 7ff8fffff700 1 --2- 172.21.15.189:0/1869465459 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ff8dc0557f0 0x7ff8dc057cb0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:13.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.576+0000 7ff8fffff700 1 --2- 172.21.15.189:0/1869465459 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ff9000623c0 0x7ff90014ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:13.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.576+0000 7ff8fffff700 1 --2- 172.21.15.189:0/1869465459 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ff900060bc0 0x7ff9000aac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:13.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.576+0000 7ff8fffff700 1 --2- 172.21.15.189:0/1869465459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff900061040 0x7ff900139ac0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:13.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.576+0000 7ff8fffff700 1 -- 172.21.15.189:0/1869465459 >> 172.21.15.189:0/1869465459 conn(0x7ff900065fe0 msgr2=0x7ff900134d50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:13.588 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.576+0000 7ff8fffff700 1 -- 172.21.15.189:0/1869465459 shutdown_connections 2020-11-20T12:20:13.588 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:13.576+0000 7ff8fffff700 1 -- 172.21.15.189:0/1869465459 wait complete. 2020-11-20T12:20:13.588 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 60 2020-11-20T12:20:13.595 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2020-11-20T12:20:15.597 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:15.736 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a3508700 1 Processor -- start 2020-11-20T12:20:15.736 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a3508700 1 -- start start 2020-11-20T12:20:15.738 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a3508700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd49c0ad000 0x7fd49c13e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:15.738 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a3508700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49c067730 con 0x7fd49c13a6e0 2020-11-20T12:20:15.738 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a3508700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49c13dd40 con 0x7fd49c0ad3e0 2020-11-20T12:20:15.738 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a3508700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd49c138190 con 0x7fd49c0ad000 2020-11-20T12:20:15.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a1d05700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd49c0ad3e0 0x7fd49c066d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53316/0 (socket says 172.21.15.189:53316) 2020-11-20T12:20:15.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a1d05700 1 -- 172.21.15.189:0/685240060 learned_addr learned my addr 172.21.15.189:0/685240060 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:20:15.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a2506700 1 --2- 172.21.15.189:0/685240060 >> v2:172.21.15.47:3300/0 conn(0x7fd49c0ad000 0x7fd49c13e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:15.739 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a1504700 1 -- 172.21.15.189:0/685240060 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1645112601 0 0) 0x7fd49c13dd40 con 0x7fd49c0ad3e0 2020-11-20T12:20:15.740 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a1504700 1 -- 172.21.15.189:0/685240060 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd484003240 con 0x7fd49c0ad3e0 2020-11-20T12:20:15.745 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a1504700 1 -- 172.21.15.189:0/685240060 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1459815688 0 0) 0x7fd484003240 con 0x7fd49c0ad3e0 2020-11-20T12:20:15.745 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a1504700 1 -- 172.21.15.189:0/685240060 >> v2:172.21.15.47:3300/0 conn(0x7fd49c0ad000 msgr2=0x7fd49c13e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:15.745 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a1504700 1 --2- 172.21.15.189:0/685240060 >> v2:172.21.15.47:3300/0 conn(0x7fd49c0ad000 0x7fd49c13e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:15.746 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a1504700 1 -- 172.21.15.189:0/685240060 >> v1:172.21.15.47:6789/0 conn(0x7fd49c13a6e0 legacy=0x7fd49c13ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:15.746 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a1504700 1 -- 172.21.15.189:0/685240060 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd49c13dfe0 con 0x7fd49c0ad3e0 2020-11-20T12:20:15.746 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a1504700 1 -- 172.21.15.189:0/685240060 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7fd48c0014e0 con 0x7fd49c0ad3e0 2020-11-20T12:20:15.746 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a1504700 1 -- 172.21.15.189:0/685240060 >> v1:172.21.15.189:6789/0 conn(0x7fd49c0ad3e0 legacy=0x7fd49c066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:15.747 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a1504700 1 --2- 172.21.15.189:0/685240060 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd484003730 0x7fd484003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:15.747 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a1504700 1 --2- 172.21.15.189:0/685240060 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd4840044e0 0x7fd484004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:15.747 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a1504700 1 --2- 172.21.15.189:0/685240060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd484004e50 0x7fd48400b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:15.747 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a1504700 1 -- 172.21.15.189:0/685240060 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c138190 con 0x7fd484003730 2020-11-20T12:20:15.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a1504700 1 -- 172.21.15.189:0/685240060 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd484004260 con 0x7fd484004e50 2020-11-20T12:20:15.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a1504700 1 -- 172.21.15.189:0/685240060 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd48400bc20 con 0x7fd4840044e0 2020-11-20T12:20:15.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a2d07700 1 --2- 172.21.15.189:0/685240060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd484004e50 0x7fd48400b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:15.748 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a1d05700 1 --2- 172.21.15.189:0/685240060 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd4840044e0 0x7fd484004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:15.749 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.732+0000 7fd4a2506700 1 --2- 172.21.15.189:0/685240060 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd484003730 0x7fd484003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:15.749 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a2d07700 1 -- 172.21.15.189:0/685240060 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd4840044e0 msgr2=0x7fd484004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:15.749 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a2d07700 1 --2- 172.21.15.189:0/685240060 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd4840044e0 0x7fd484004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:15.749 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a2d07700 1 -- 172.21.15.189:0/685240060 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd484003730 msgr2=0x7fd484003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:15.750 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a2d07700 1 --2- 172.21.15.189:0/685240060 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd484003730 0x7fd484003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:15.750 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a2d07700 1 -- 172.21.15.189:0/685240060 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd484003240 con 0x7fd484004e50 2020-11-20T12:20:15.750 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a2d07700 1 -- 172.21.15.189:0/685240060 reap_dead start 2020-11-20T12:20:15.750 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a2d07700 1 --2- 172.21.15.189:0/685240060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd484004e50 0x7fd48400b730 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7fd494001e40 tx=0x7fd494007f10).ready entity=mon.1 client_cookie=a0f474cd9e719e31 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:15.751 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a1504700 1 -- 172.21.15.189:0/685240060 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd494001540 con 0x7fd484004e50 2020-11-20T12:20:15.751 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a1504700 1 -- 172.21.15.189:0/685240060 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd49400cad0 con 0x7fd484004e50 2020-11-20T12:20:15.751 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a1504700 1 -- 172.21.15.189:0/685240060 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49400ce50 con 0x7fd484004e50 2020-11-20T12:20:15.751 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a3508700 1 -- 172.21.15.189:0/685240060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd484004e50 msgr2=0x7fd48400b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:15.751 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a3508700 1 --2- 172.21.15.189:0/685240060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd484004e50 0x7fd48400b730 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7fd494001e40 tx=0x7fd494007f10).stop 2020-11-20T12:20:15.752 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a3508700 1 -- 172.21.15.189:0/685240060 shutdown_connections 2020-11-20T12:20:15.752 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a3508700 1 --2- 172.21.15.189:0/685240060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd484004e50 0x7fd48400b730 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:15.752 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a3508700 1 -- 172.21.15.189:0/685240060 >> 172.21.15.189:0/685240060 conn(0x7fd49c062080 msgr2=0x7fd49c062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:15.752 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a3508700 1 -- 172.21.15.189:0/685240060 shutdown_connections 2020-11-20T12:20:15.753 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a3508700 1 -- 172.21.15.189:0/685240060 wait complete. 2020-11-20T12:20:15.753 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a3508700 1 Processor -- start 2020-11-20T12:20:15.753 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a3508700 1 -- start start 2020-11-20T12:20:15.753 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a3508700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd484004e50 0x7fd49c0a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:15.753 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a3508700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd49c0a3450 0x7fd49c134e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:15.754 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a3508700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49c135360 0x7fd49c1357e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:15.754 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a3508700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c135cd0 con 0x7fd49c0a3450 2020-11-20T12:20:15.755 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a3508700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c0afde0 con 0x7fd49c135360 2020-11-20T12:20:15.755 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a3508700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c13fa90 con 0x7fd484004e50 2020-11-20T12:20:15.755 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a2d07700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49c135360 0x7fd49c1357e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:15.755 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a2d07700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49c135360 0x7fd49c1357e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51708/0 (socket says 172.21.15.189:51708) 2020-11-20T12:20:15.756 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a2d07700 1 -- 172.21.15.189:0/3257294609 learned_addr learned my addr 172.21.15.189:0/3257294609 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:20:15.756 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a2506700 1 --2- 172.21.15.189:0/3257294609 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd484004e50 0x7fd49c0a0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:15.756 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a1d05700 1 --2- 172.21.15.189:0/3257294609 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd49c0a3450 0x7fd49c134e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:15.756 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a2d07700 1 -- 172.21.15.189:0/3257294609 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd484004e50 msgr2=0x7fd49c0a0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:15.756 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a2d07700 1 --2- 172.21.15.189:0/3257294609 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd484004e50 0x7fd49c0a0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:15.757 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a2d07700 1 -- 172.21.15.189:0/3257294609 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd49c0a3450 msgr2=0x7fd49c134e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:15.757 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a2d07700 1 --2- 172.21.15.189:0/3257294609 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd49c0a3450 0x7fd49c134e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:15.757 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a2d07700 1 -- 172.21.15.189:0/3257294609 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd49c13fd30 con 0x7fd49c135360 2020-11-20T12:20:15.757 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd4a2d07700 1 --2- 172.21.15.189:0/3257294609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49c135360 0x7fd49c1357e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7fd494001e40 tx=0x7fd494015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:15.758 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.736+0000 7fd48a7fc700 1 -- 172.21.15.189:0/3257294609 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd494013b20 con 0x7fd49c135360 2020-11-20T12:20:15.758 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.740+0000 7fd48a7fc700 1 -- 172.21.15.189:0/3257294609 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd494013ca0 con 0x7fd49c135360 2020-11-20T12:20:15.758 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.740+0000 7fd48a7fc700 1 -- 172.21.15.189:0/3257294609 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49401d650 con 0x7fd49c135360 2020-11-20T12:20:15.758 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.740+0000 7fd4a3508700 1 -- 172.21.15.189:0/3257294609 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd49c13ffd0 con 0x7fd49c135360 2020-11-20T12:20:15.758 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.740+0000 7fd4a3508700 1 -- 172.21.15.189:0/3257294609 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd49c0a41c0 con 0x7fd49c135360 2020-11-20T12:20:15.759 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.740+0000 7fd4a3508700 1 -- 172.21.15.189:0/3257294609 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd49c0a3250 con 0x7fd49c135360 2020-11-20T12:20:15.760 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.740+0000 7fd48a7fc700 1 -- 172.21.15.189:0/3257294609 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fd494010070 con 0x7fd49c135360 2020-11-20T12:20:15.760 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.740+0000 7fd48a7fc700 1 --2- 172.21.15.189:0/3257294609 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd4740557c0 0x7fd474057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:15.760 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.740+0000 7fd48a7fc700 1 -- 172.21.15.189:0/3257294609 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7019+0+0 (secure 0 0 0) 0x7fd494070320 con 0x7fd49c135360 2020-11-20T12:20:15.761 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.744+0000 7fd4a2506700 1 --2- 172.21.15.189:0/3257294609 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd4740557c0 0x7fd474057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:15.761 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.744+0000 7fd4a2506700 1 --2- 172.21.15.189:0/3257294609 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd4740557c0 0x7fd474057c80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fd498006a90 tx=0x7fd498008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:15.761 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.748+0000 7fd48a7fc700 1 -- 172.21.15.189:0/3257294609 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd494045e10 con 0x7fd49c135360 2020-11-20T12:20:15.978 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.972+0000 7fd4a3508700 1 -- 172.21.15.189:0/3257294609 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd49c0b13d0 con 0x7fd49c135360 2020-11-20T12:20:15.980 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.972+0000 7fd48a7fc700 1 -- 172.21.15.189:0/3257294609 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+1782 (secure 0 0 0) 0x7fd494049430 con 0x7fd49c135360 2020-11-20T12:20:15.980 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:20:15.981 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:20:05.807690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5520":{"gid":5520,"name":"b","rank":0,"incarnation":57,"state":"up:active","state_seq":32,"addr":"172.21.15.47:6833/201662157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":201662157},{"type":"v1","addr":"172.21.15.47:6833","nonce":201662157}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:20:15.983 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.976+0000 7fd4a3508700 1 -- 172.21.15.189:0/3257294609 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd4740557c0 msgr2=0x7fd474057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:15.983 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.976+0000 7fd4a3508700 1 --2- 172.21.15.189:0/3257294609 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd4740557c0 0x7fd474057c80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fd498006a90 tx=0x7fd498008040).stop 2020-11-20T12:20:15.984 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.976+0000 7fd4a3508700 1 -- 172.21.15.189:0/3257294609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49c135360 msgr2=0x7fd49c1357e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:15.984 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.976+0000 7fd4a3508700 1 --2- 172.21.15.189:0/3257294609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49c135360 0x7fd49c1357e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7fd494001e40 tx=0x7fd494015000).stop 2020-11-20T12:20:15.984 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.976+0000 7fd4a3508700 1 -- 172.21.15.189:0/3257294609 shutdown_connections 2020-11-20T12:20:15.984 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.976+0000 7fd4a3508700 1 --2- 172.21.15.189:0/3257294609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd49c135360 0x7fd49c1357e0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:15.985 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.976+0000 7fd4a3508700 1 --2- 172.21.15.189:0/3257294609 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd484004e50 0x7fd49c0a0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:15.985 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.976+0000 7fd4a3508700 1 --2- 172.21.15.189:0/3257294609 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd4740557c0 0x7fd474057c80 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:15.985 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.976+0000 7fd4a3508700 1 --2- 172.21.15.189:0/3257294609 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd49c0a3450 0x7fd49c134e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:15.985 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.976+0000 7fd4a3508700 1 -- 172.21.15.189:0/3257294609 >> 172.21.15.189:0/3257294609 conn(0x7fd49c062080 msgr2=0x7fd49c066c90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:15.986 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.976+0000 7fd4a3508700 1 -- 172.21.15.189:0/3257294609 shutdown_connections 2020-11-20T12:20:15.986 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:15.976+0000 7fd4a3508700 1 -- 172.21.15.189:0/3257294609 wait complete. 2020-11-20T12:20:15.987 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 60 2020-11-20T12:20:16.002 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2020-11-20T12:20:18.003 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:18.163 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.156+0000 7f807bfff700 1 Processor -- start 2020-11-20T12:20:18.486 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.156+0000 7f807bfff700 1 -- start start 2020-11-20T12:20:18.495 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.160+0000 7f807bfff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f807c1347f0 0x7f807c0677f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:18.495 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.160+0000 7f807bfff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f807c0a9c60 con 0x7f807c13b1d0 2020-11-20T12:20:18.495 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.160+0000 7f807bfff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f807c067e90 con 0x7f807c0623c0 2020-11-20T12:20:18.495 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.160+0000 7f807bfff700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f807c0a8e60 con 0x7f807c1347f0 2020-11-20T12:20:18.496 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.160+0000 7f807affd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f807c0623c0 0x7f807c13aac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53336/0 (socket says 172.21.15.189:53336) 2020-11-20T12:20:18.496 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.160+0000 7f807affd700 1 -- 172.21.15.189:0/2043158160 learned_addr learned my addr 172.21.15.189:0/2043158160 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:20:18.496 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.160+0000 7f807b7fe700 1 --2- 172.21.15.189:0/2043158160 >> v2:172.21.15.189:3300/0 conn(0x7f807c1347f0 0x7f807c0677f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:18.496 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.160+0000 7f807b7fe700 1 -- 172.21.15.189:0/2043158160 >> v1:172.21.15.189:6789/0 conn(0x7f807c0623c0 legacy=0x7f807c13aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:18.496 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.160+0000 7f807b7fe700 1 -- 172.21.15.189:0/2043158160 >> v1:172.21.15.47:6789/0 conn(0x7f807c13b1d0 legacy=0x7f807c1340e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:18.497 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.160+0000 7f8079ffb700 1 -- 172.21.15.189:0/2043158160 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2132881236 0 0) 0x7f807c067e90 con 0x7f807c0623c0 2020-11-20T12:20:18.497 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.160+0000 7f807b7fe700 1 -- 172.21.15.189:0/2043158160 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807c068130 con 0x7f807c1347f0 2020-11-20T12:20:18.498 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.160+0000 7f807b7fe700 1 --2- 172.21.15.189:0/2043158160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f807c1347f0 0x7f807c0677f0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f8074009c90 tx=0x7f80740091d0).ready entity=mon.1 client_cookie=990068a96faf65fb server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:18.498 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.160+0000 7f8079ffb700 1 -- 172.21.15.189:0/2043158160 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807400bbc0 con 0x7f807c1347f0 2020-11-20T12:20:18.498 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.160+0000 7f8079ffb700 1 -- 172.21.15.189:0/2043158160 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f807400bd20 con 0x7f807c1347f0 2020-11-20T12:20:18.498 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.160+0000 7f8079ffb700 1 -- 172.21.15.189:0/2043158160 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8074011610 con 0x7f807c1347f0 2020-11-20T12:20:18.499 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.160+0000 7f807bfff700 1 -- 172.21.15.189:0/2043158160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f807c1347f0 msgr2=0x7f807c0677f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:18.499 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.160+0000 7f807bfff700 1 --2- 172.21.15.189:0/2043158160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f807c1347f0 0x7f807c0677f0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f8074009c90 tx=0x7f80740091d0).stop 2020-11-20T12:20:18.499 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.160+0000 7f807bfff700 1 -- 172.21.15.189:0/2043158160 shutdown_connections 2020-11-20T12:20:18.499 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.160+0000 7f807bfff700 1 --2- 172.21.15.189:0/2043158160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f807c1347f0 0x7f807c0677f0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:18.499 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.160+0000 7f807bfff700 1 -- 172.21.15.189:0/2043158160 >> 172.21.15.189:0/2043158160 conn(0x7f807c065fe0 msgr2=0x7f807c061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:18.500 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.160+0000 7f807bfff700 1 -- 172.21.15.189:0/2043158160 shutdown_connections 2020-11-20T12:20:18.500 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.160+0000 7f807bfff700 1 -- 172.21.15.189:0/2043158160 wait complete. 2020-11-20T12:20:18.500 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.160+0000 7f807bfff700 1 Processor -- start 2020-11-20T12:20:18.500 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.160+0000 7f807bfff700 1 -- start start 2020-11-20T12:20:18.500 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.164+0000 7f807bfff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f807c0623c0 0x7f807c0a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:18.501 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.164+0000 7f807bfff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f807c1347f0 0x7f807c0a2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:18.501 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.164+0000 7f807bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f807c13b1d0 0x7f807c149d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:18.501 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.164+0000 7f807bfff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c133530 con 0x7f807c1347f0 2020-11-20T12:20:18.501 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.164+0000 7f807bfff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f807c1337d0 con 0x7f807c13b1d0 2020-11-20T12:20:18.501 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.164+0000 7f807bfff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f807c0a11b0 con 0x7f807c0623c0 2020-11-20T12:20:18.502 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.164+0000 7f807b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f807c13b1d0 0x7f807c149d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:18.502 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.164+0000 7f807b7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f807c13b1d0 0x7f807c149d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51724/0 (socket says 172.21.15.189:51724) 2020-11-20T12:20:18.502 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.164+0000 7f807b7fe700 1 -- 172.21.15.189:0/1058653598 learned_addr learned my addr 172.21.15.189:0/1058653598 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:20:18.502 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.164+0000 7f807a7fc700 1 --2- 172.21.15.189:0/1058653598 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f807c1347f0 0x7f807c0a2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:18.502 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.164+0000 7f807affd700 1 --2- 172.21.15.189:0/1058653598 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f807c0623c0 0x7f807c0a0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:18.503 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.164+0000 7f807b7fe700 1 -- 172.21.15.189:0/1058653598 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f807c0623c0 msgr2=0x7f807c0a0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:18.503 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.164+0000 7f807b7fe700 1 --2- 172.21.15.189:0/1058653598 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f807c0623c0 0x7f807c0a0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:18.503 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.164+0000 7f807b7fe700 1 -- 172.21.15.189:0/1058653598 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f807c1347f0 msgr2=0x7f807c0a2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:18.503 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.164+0000 7f807b7fe700 1 --2- 172.21.15.189:0/1058653598 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f807c1347f0 0x7f807c0a2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:18.503 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.164+0000 7f807b7fe700 1 -- 172.21.15.189:0/1058653598 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807c0a34b0 con 0x7f807c13b1d0 2020-11-20T12:20:18.504 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.164+0000 7f807a7fc700 1 --2- 172.21.15.189:0/1058653598 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f807c1347f0 0x7f807c0a2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:20:18.504 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.164+0000 7f807affd700 1 --2- 172.21.15.189:0/1058653598 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f807c0623c0 0x7f807c0a0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:20:18.504 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.164+0000 7f807b7fe700 1 --2- 172.21.15.189:0/1058653598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f807c13b1d0 0x7f807c149d20 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7f8074009440 tx=0x7f8074016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:18.504 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.164+0000 7f8072ffd700 1 -- 172.21.15.189:0/1058653598 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8074011e30 con 0x7f807c13b1d0 2020-11-20T12:20:18.504 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.164+0000 7f8072ffd700 1 -- 172.21.15.189:0/1058653598 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8074010c40 con 0x7f807c13b1d0 2020-11-20T12:20:18.505 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.164+0000 7f807bfff700 1 -- 172.21.15.189:0/1058653598 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f807c149660 con 0x7f807c13b1d0 2020-11-20T12:20:18.505 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.164+0000 7f8072ffd700 1 -- 172.21.15.189:0/1058653598 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807401d480 con 0x7f807c13b1d0 2020-11-20T12:20:18.505 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.164+0000 7f807bfff700 1 -- 172.21.15.189:0/1058653598 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f807c139e20 con 0x7f807c13b1d0 2020-11-20T12:20:18.506 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.164+0000 7f807bfff700 1 -- 172.21.15.189:0/1058653598 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f807c0663c0 con 0x7f807c13b1d0 2020-11-20T12:20:18.506 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.164+0000 7f8072ffd700 1 -- 172.21.15.189:0/1058653598 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f8074017070 con 0x7f807c13b1d0 2020-11-20T12:20:18.507 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.168+0000 7f8072ffd700 1 --2- 172.21.15.189:0/1058653598 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f80580557c0 0x7f8058057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:18.507 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.168+0000 7f8072ffd700 1 -- 172.21.15.189:0/1058653598 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7019+0+0 (secure 0 0 0) 0x7f8074071200 con 0x7f807c13b1d0 2020-11-20T12:20:18.507 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.168+0000 7f807affd700 1 --2- 172.21.15.189:0/1058653598 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f80580557c0 0x7f8058057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:18.507 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.168+0000 7f807affd700 1 --2- 172.21.15.189:0/1058653598 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f80580557c0 0x7f8058057c80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f807c0a15e0 tx=0x7f8064009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:18.508 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.172+0000 7f8072ffd700 1 -- 172.21.15.189:0/1058653598 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8074046cf0 con 0x7f807c13b1d0 2020-11-20T12:20:18.508 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.396+0000 7f807bfff700 1 -- 172.21.15.189:0/1058653598 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f807c0446d0 con 0x7f807c13b1d0 2020-11-20T12:20:18.508 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.400+0000 7f8072ffd700 1 -- 172.21.15.189:0/1058653598 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f8074015070 con 0x7f807c13b1d0 2020-11-20T12:20:18.508 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.400+0000 7f807bfff700 1 -- 172.21.15.189:0/1058653598 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f80580557c0 msgr2=0x7f8058057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:18.508 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.400+0000 7f807bfff700 1 --2- 172.21.15.189:0/1058653598 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f80580557c0 0x7f8058057c80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f807c0a15e0 tx=0x7f8064009040).stop 2020-11-20T12:20:18.509 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.400+0000 7f807bfff700 1 -- 172.21.15.189:0/1058653598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f807c13b1d0 msgr2=0x7f807c149d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:18.509 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.400+0000 7f807bfff700 1 --2- 172.21.15.189:0/1058653598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f807c13b1d0 0x7f807c149d20 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7f8074009440 tx=0x7f8074016000).stop 2020-11-20T12:20:18.509 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.400+0000 7f807bfff700 1 -- 172.21.15.189:0/1058653598 shutdown_connections 2020-11-20T12:20:18.509 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.400+0000 7f807bfff700 1 --2- 172.21.15.189:0/1058653598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f807c13b1d0 0x7f807c149d20 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:18.509 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.400+0000 7f807bfff700 1 --2- 172.21.15.189:0/1058653598 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f807c0623c0 0x7f807c0a0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:18.510 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.400+0000 7f807bfff700 1 --2- 172.21.15.189:0/1058653598 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f80580557c0 0x7f8058057c80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:18.510 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.400+0000 7f807bfff700 1 --2- 172.21.15.189:0/1058653598 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f807c1347f0 0x7f807c0a2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:18.510 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.400+0000 7f807bfff700 1 -- 172.21.15.189:0/1058653598 >> 172.21.15.189:0/1058653598 conn(0x7f807c065fe0 msgr2=0x7f807c05ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:18.510 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.400+0000 7f807bfff700 1 -- 172.21.15.189:0/1058653598 shutdown_connections 2020-11-20T12:20:18.510 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:18.400+0000 7f807bfff700 1 -- 172.21.15.189:0/1058653598 wait complete. 2020-11-20T12:20:18.511 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 61 2020-11-20T12:20:18.511 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:20:18.511 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5746,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.189:6835/3538437301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3538437301},{"type":"v1","addr":"172.21.15.189:6835","nonce":3538437301}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:20:05.807690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5520":{"gid":5520,"name":"b","rank":0,"incarnation":57,"state":"up:active","state_seq":32,"addr":"172.21.15.47:6833/201662157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":201662157},{"type":"v1","addr":"172.21.15.47:6833","nonce":201662157}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:20:18.511 INFO:tasks.mds_thrash.fs.[None]:mds.a reported in up:standby state 2020-11-20T12:20:18.512 INFO:tasks.mds_thrash.fs.[None]:waiting for 25 secs before thrashing 2020-11-20T12:20:26.065 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:20:26.071 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:20:43.513 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:43.643 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.635+0000 7efcbd8b0700 1 Processor -- start 2020-11-20T12:20:43.643 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.635+0000 7efcbd8b0700 1 -- start start 2020-11-20T12:20:43.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcbd8b0700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efcb80a81b0 0x7efcb80aa3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:43.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcbd8b0700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb80a8fa0 con 0x7efcb8060310 2020-11-20T12:20:43.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcbd8b0700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb80a9240 con 0x7efcb80a7d40 2020-11-20T12:20:43.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcbd8b0700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb80618b0 con 0x7efcb80a81b0 2020-11-20T12:20:43.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcb7fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7efcb80a7d40 0x7efcb80a9cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53352/0 (socket says 172.21.15.189:53352) 2020-11-20T12:20:43.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcb7fff700 1 -- 172.21.15.189:0/4247914830 learned_addr learned my addr 172.21.15.189:0/4247914830 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:20:43.646 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcbd0af700 1 --2- 172.21.15.189:0/4247914830 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efcb80a81b0 0x7efcb80aa3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:43.646 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcb77fe700 1 -- 172.21.15.189:0/4247914830 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134496215 0 0) 0x7efcb80a9240 con 0x7efcb80a7d40 2020-11-20T12:20:43.646 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcb77fe700 1 -- 172.21.15.189:0/4247914830 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc9c003240 con 0x7efcb80a7d40 2020-11-20T12:20:43.649 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcb77fe700 1 -- 172.21.15.189:0/4247914830 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3435503180 0 0) 0x7efc9c003240 con 0x7efcb80a7d40 2020-11-20T12:20:43.649 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcb77fe700 1 -- 172.21.15.189:0/4247914830 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efcb80a81b0 msgr2=0x7efcb80aa3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:43.649 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcb77fe700 1 --2- 172.21.15.189:0/4247914830 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efcb80a81b0 0x7efcb80aa3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:43.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcb77fe700 1 -- 172.21.15.189:0/4247914830 >> v1:172.21.15.47:6789/0 conn(0x7efcb8060310 legacy=0x7efcb80a1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:43.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcb77fe700 1 -- 172.21.15.189:0/4247914830 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb80a94e0 con 0x7efcb80a7d40 2020-11-20T12:20:43.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcb77fe700 1 -- 172.21.15.189:0/4247914830 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7efca40014e0 con 0x7efcb80a7d40 2020-11-20T12:20:43.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcb77fe700 1 -- 172.21.15.189:0/4247914830 >> v1:172.21.15.189:6789/0 conn(0x7efcb80a7d40 legacy=0x7efcb80a9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:43.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcb77fe700 1 --2- 172.21.15.189:0/4247914830 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efcb80a81b0 0x7efc9c003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:43.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcb77fe700 1 --2- 172.21.15.189:0/4247914830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc9c003c40 0x7efc9c0044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:43.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcb77fe700 1 --2- 172.21.15.189:0/4247914830 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc9c0049d0 0x7efc9c004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:43.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcb77fe700 1 -- 172.21.15.189:0/4247914830 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb80618b0 con 0x7efc9c0049d0 2020-11-20T12:20:43.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcb77fe700 1 -- 172.21.15.189:0/4247914830 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c0041d0 con 0x7efc9c003c40 2020-11-20T12:20:43.651 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcb77fe700 1 -- 172.21.15.189:0/4247914830 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c00b730 con 0x7efcb80a81b0 2020-11-20T12:20:43.652 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcbd0af700 1 --2- 172.21.15.189:0/4247914830 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc9c0049d0 0x7efc9c004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:43.652 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcbc8ae700 1 --2- 172.21.15.189:0/4247914830 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efcb80a81b0 0x7efc9c003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:43.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcb7fff700 1 --2- 172.21.15.189:0/4247914830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc9c003c40 0x7efc9c0044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:43.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcbc8ae700 1 -- 172.21.15.189:0/4247914830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc9c003c40 msgr2=0x7efc9c0044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:43.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcbc8ae700 1 --2- 172.21.15.189:0/4247914830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc9c003c40 0x7efc9c0044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:43.653 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcbc8ae700 1 -- 172.21.15.189:0/4247914830 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc9c0049d0 msgr2=0x7efc9c004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:43.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcbc8ae700 1 --2- 172.21.15.189:0/4247914830 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efc9c0049d0 0x7efc9c004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:43.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcbc8ae700 1 -- 172.21.15.189:0/4247914830 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc9c003240 con 0x7efcb80a81b0 2020-11-20T12:20:43.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcb7fff700 1 --2- 172.21.15.189:0/4247914830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc9c003c40 0x7efc9c0044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:20:43.654 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcbc8ae700 1 --2- 172.21.15.189:0/4247914830 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efcb80a81b0 0x7efc9c003730 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7efcac001920 tx=0x7efcac007b90).ready entity=mon.2 client_cookie=64956c9d09c76bcd server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:43.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcb77fe700 1 -- 172.21.15.189:0/4247914830 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac007020 con 0x7efcb80a81b0 2020-11-20T12:20:43.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcb77fe700 1 -- 172.21.15.189:0/4247914830 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcac007180 con 0x7efcb80a81b0 2020-11-20T12:20:43.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcb77fe700 1 -- 172.21.15.189:0/4247914830 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac00c830 con 0x7efcb80a81b0 2020-11-20T12:20:43.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcbd8b0700 1 -- 172.21.15.189:0/4247914830 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efcb80a81b0 msgr2=0x7efc9c003730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:43.655 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcbd8b0700 1 --2- 172.21.15.189:0/4247914830 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efcb80a81b0 0x7efc9c003730 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7efcac001920 tx=0x7efcac007b90).stop 2020-11-20T12:20:43.656 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcbd0af700 1 -- 172.21.15.189:0/4247914830 reap_dead start 2020-11-20T12:20:43.656 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcbd8b0700 1 -- 172.21.15.189:0/4247914830 shutdown_connections 2020-11-20T12:20:43.656 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcbd8b0700 1 -- 172.21.15.189:0/4247914830 >> 172.21.15.189:0/4247914830 conn(0x7efcb8063210 msgr2=0x7efcb8060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:43.656 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcbd8b0700 1 -- 172.21.15.189:0/4247914830 shutdown_connections 2020-11-20T12:20:43.656 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.639+0000 7efcbd8b0700 1 -- 172.21.15.189:0/4247914830 wait complete. 2020-11-20T12:20:43.657 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.643+0000 7efcbd8b0700 1 Processor -- start 2020-11-20T12:20:43.657 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.643+0000 7efcbd8b0700 1 -- start start 2020-11-20T12:20:43.659 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.643+0000 7efcbd8b0700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcb809c880 0x7efcb80a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:43.660 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.643+0000 7efcbd8b0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcb809cc60 0x7efcb813a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:43.660 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.643+0000 7efcbd8b0700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efcb814b080 0x7efcb814b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:43.660 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.643+0000 7efcbd8b0700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb813b770 con 0x7efcb809c880 2020-11-20T12:20:43.660 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.643+0000 7efcbd8b0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb8139d10 con 0x7efcb809cc60 2020-11-20T12:20:43.661 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.643+0000 7efcbd8b0700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb8139fb0 con 0x7efcb814b080 2020-11-20T12:20:43.661 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.643+0000 7efcb7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcb809cc60 0x7efcb813a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:43.661 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.643+0000 7efcb7fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcb809cc60 0x7efcb813a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51742/0 (socket says 172.21.15.189:51742) 2020-11-20T12:20:43.661 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.643+0000 7efcb7fff700 1 -- 172.21.15.189:0/3151542330 learned_addr learned my addr 172.21.15.189:0/3151542330 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:20:43.662 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.643+0000 7efcbc8ae700 1 --2- 172.21.15.189:0/3151542330 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcb809c880 0x7efcb80a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:43.662 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.643+0000 7efcbd0af700 1 --2- 172.21.15.189:0/3151542330 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efcb814b080 0x7efcb814b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:43.662 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.643+0000 7efcb7fff700 1 -- 172.21.15.189:0/3151542330 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efcb814b080 msgr2=0x7efcb814b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:43.662 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.643+0000 7efcb7fff700 1 --2- 172.21.15.189:0/3151542330 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efcb814b080 0x7efcb814b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:43.663 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.643+0000 7efcb7fff700 1 -- 172.21.15.189:0/3151542330 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcb809c880 msgr2=0x7efcb80a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:43.663 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.643+0000 7efcb7fff700 1 --2- 172.21.15.189:0/3151542330 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcb809c880 0x7efcb80a5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:43.663 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.643+0000 7efcb7fff700 1 -- 172.21.15.189:0/3151542330 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb8139620 con 0x7efcb809cc60 2020-11-20T12:20:43.663 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.643+0000 7efcbc8ae700 1 --2- 172.21.15.189:0/3151542330 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcb809c880 0x7efcb80a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:20:43.663 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.643+0000 7efcb7fff700 1 --2- 172.21.15.189:0/3151542330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcb809cc60 0x7efcb813a450 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7efca4004560 tx=0x7efcb8138490).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:43.664 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.647+0000 7efcb4ff9700 1 -- 172.21.15.189:0/3151542330 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca400fcc0 con 0x7efcb809cc60 2020-11-20T12:20:43.664 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.647+0000 7efcb4ff9700 1 -- 172.21.15.189:0/3151542330 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca400fe20 con 0x7efcb809cc60 2020-11-20T12:20:43.664 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.647+0000 7efcbd8b0700 1 -- 172.21.15.189:0/3151542330 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb80a6f30 con 0x7efcb809cc60 2020-11-20T12:20:43.665 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.647+0000 7efcb4ff9700 1 -- 172.21.15.189:0/3151542330 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca4012560 con 0x7efcb809cc60 2020-11-20T12:20:43.665 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.647+0000 7efcbd8b0700 1 -- 172.21.15.189:0/3151542330 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb813b4e0 con 0x7efcb809cc60 2020-11-20T12:20:43.665 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.647+0000 7efcbd8b0700 1 -- 172.21.15.189:0/3151542330 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb80a0c70 con 0x7efcb809cc60 2020-11-20T12:20:43.665 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.647+0000 7efcb4ff9700 1 -- 172.21.15.189:0/3151542330 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7efca401b030 con 0x7efcb809cc60 2020-11-20T12:20:43.665 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.647+0000 7efcb4ff9700 1 --2- 172.21.15.189:0/3151542330 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7efc900557c0 0x7efc90057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:43.666 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.647+0000 7efcb4ff9700 1 -- 172.21.15.189:0/3151542330 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7019+0+0 (secure 0 0 0) 0x7efca406e5c0 con 0x7efcb809cc60 2020-11-20T12:20:43.666 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.647+0000 7efcbc8ae700 1 --2- 172.21.15.189:0/3151542330 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7efc900557c0 0x7efc90057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:43.666 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.647+0000 7efcbc8ae700 1 --2- 172.21.15.189:0/3151542330 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7efc900557c0 0x7efc90057c80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7efcac000fd0 tx=0x7efcac015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:43.666 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.655+0000 7efcb4ff9700 1 -- 172.21.15.189:0/3151542330 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7efcb80a0c70 con 0x7efcb809cc60 2020-11-20T12:20:43.888 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.883+0000 7efcbd8b0700 1 -- 172.21.15.189:0/3151542330 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcb80a0c70 con 0x7efcb809cc60 2020-11-20T12:20:43.889 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.883+0000 7efcb4ff9700 1 -- 172.21.15.189:0/3151542330 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2137 (secure 0 0 0) 0x7efcb80a0c70 con 0x7efcb809cc60 2020-11-20T12:20:43.890 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:20:43.890 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5746,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.189:6835/3538437301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3538437301},{"type":"v1","addr":"172.21.15.189:6835","nonce":3538437301}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:20:05.807690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5520":{"gid":5520,"name":"b","rank":0,"incarnation":57,"state":"up:active","state_seq":32,"addr":"172.21.15.47:6833/201662157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":201662157},{"type":"v1","addr":"172.21.15.47:6833","nonce":201662157}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:20:43.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.883+0000 7efcbd8b0700 1 -- 172.21.15.189:0/3151542330 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7efc900557c0 msgr2=0x7efc90057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:43.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.883+0000 7efcbd8b0700 1 --2- 172.21.15.189:0/3151542330 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7efc900557c0 0x7efc90057c80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7efcac000fd0 tx=0x7efcac015000).stop 2020-11-20T12:20:43.891 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.883+0000 7efcbd8b0700 1 -- 172.21.15.189:0/3151542330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcb809cc60 msgr2=0x7efcb813a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:43.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.883+0000 7efcbd8b0700 1 --2- 172.21.15.189:0/3151542330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcb809cc60 0x7efcb813a450 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7efca4004560 tx=0x7efcb8138490).stop 2020-11-20T12:20:43.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.883+0000 7efcbd8b0700 1 -- 172.21.15.189:0/3151542330 shutdown_connections 2020-11-20T12:20:43.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.883+0000 7efcbd8b0700 1 --2- 172.21.15.189:0/3151542330 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7efc900557c0 0x7efc90057c80 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:43.892 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.883+0000 7efcbd8b0700 1 --2- 172.21.15.189:0/3151542330 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efcb809c880 0x7efcb80a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:43.893 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.883+0000 7efcbd8b0700 1 --2- 172.21.15.189:0/3151542330 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efcb814b080 0x7efcb814b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:43.893 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.883+0000 7efcbd8b0700 1 --2- 172.21.15.189:0/3151542330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcb809cc60 0x7efcb813a450 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:43.893 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.883+0000 7efcbd8b0700 1 -- 172.21.15.189:0/3151542330 >> 172.21.15.189:0/3151542330 conn(0x7efcb8063210 msgr2=0x7efcb8060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:43.893 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.887+0000 7efcbd8b0700 1 -- 172.21.15.189:0/3151542330 shutdown_connections 2020-11-20T12:20:43.893 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:43.887+0000 7efcbd8b0700 1 -- 172.21.15.189:0/3151542330 wait complete. 2020-11-20T12:20:43.894 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 61 2020-11-20T12:20:43.904 INFO:tasks.mds_thrash.fs.[None]:kill mds.b (rank=0) 2020-11-20T12:20:43.904 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:20:43.904 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:20:43.923 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:20:43.924 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:44.057 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.051+0000 7fca83c7f700 1 Processor -- start 2020-11-20T12:20:44.057 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.051+0000 7fca83c7f700 1 -- start start 2020-11-20T12:20:44.061 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.051+0000 7fca83c7f700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fca7c0a8bc0 0x7fca7c13dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:44.062 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.051+0000 7fca83c7f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fca7c066ab0 0x7fca7c0a9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:44.062 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.051+0000 7fca83c7f700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7c066fc0 con 0x7fca7c0a8340 2020-11-20T12:20:44.063 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.051+0000 7fca83c7f700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca7c0a7ea0 con 0x7fca7c0a8bc0 2020-11-20T12:20:44.063 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.051+0000 7fca83c7f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca7c0a8fa0 con 0x7fca7c066ab0 2020-11-20T12:20:44.063 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.051+0000 7fca8121a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fca7c066ab0 0x7fca7c0a9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:44.064 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.051+0000 7fca8121a700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fca7c066ab0 0x7fca7c0a9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51752/0 (socket says 172.21.15.189:51752) 2020-11-20T12:20:44.064 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.051+0000 7fca8121a700 1 -- 172.21.15.189:0/1058076382 learned_addr learned my addr 172.21.15.189:0/1058076382 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:20:44.064 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.051+0000 7fca81a1b700 1 --2- 172.21.15.189:0/1058076382 >> v2:172.21.15.47:3300/0 conn(0x7fca7c0a8bc0 0x7fca7c13dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:44.065 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.051+0000 7fca8121a700 1 -- 172.21.15.189:0/1058076382 >> v2:172.21.15.47:3300/0 conn(0x7fca7c0a8bc0 msgr2=0x7fca7c13dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:44.065 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.051+0000 7fca8121a700 1 --2- 172.21.15.189:0/1058076382 >> v2:172.21.15.47:3300/0 conn(0x7fca7c0a8bc0 0x7fca7c13dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:44.066 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.051+0000 7fca8121a700 1 -- 172.21.15.189:0/1058076382 >> v1:172.21.15.47:6789/0 conn(0x7fca7c0a8340 legacy=0x7fca7c060170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:44.066 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.051+0000 7fca8121a700 1 -- 172.21.15.189:0/1058076382 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7c0aa1d0 con 0x7fca7c066ab0 2020-11-20T12:20:44.066 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.051+0000 7fca81a1b700 1 --2- 172.21.15.189:0/1058076382 >> v2:172.21.15.47:3300/0 conn(0x7fca7c0a8bc0 0x7fca7c13dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:20:44.067 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.051+0000 7fca8121a700 1 --2- 172.21.15.189:0/1058076382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca7c066ab0 0x7fca7c0a9ce0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7fca7c0aa890 tx=0x7fca6c006e50).ready entity=mon.1 client_cookie=e9fcf93516a9ea78 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:44.067 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.055+0000 7fca80a19700 1 -- 172.21.15.189:0/1058076382 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6c006030 con 0x7fca7c066ab0 2020-11-20T12:20:44.068 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.055+0000 7fca80a19700 1 -- 172.21.15.189:0/1058076382 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca6c006190 con 0x7fca7c066ab0 2020-11-20T12:20:44.068 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.055+0000 7fca80a19700 1 -- 172.21.15.189:0/1058076382 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6c006510 con 0x7fca7c066ab0 2020-11-20T12:20:44.069 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.055+0000 7fca83c7f700 1 -- 172.21.15.189:0/1058076382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca7c066ab0 msgr2=0x7fca7c0a9ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:44.069 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.055+0000 7fca83c7f700 1 --2- 172.21.15.189:0/1058076382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca7c066ab0 0x7fca7c0a9ce0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7fca7c0aa890 tx=0x7fca6c006e50).stop 2020-11-20T12:20:44.070 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.055+0000 7fca83c7f700 1 -- 172.21.15.189:0/1058076382 shutdown_connections 2020-11-20T12:20:44.070 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.055+0000 7fca83c7f700 1 --2- 172.21.15.189:0/1058076382 >> v2:172.21.15.47:3300/0 conn(0x7fca7c0a8bc0 0x7fca7c13dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:44.071 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.055+0000 7fca83c7f700 1 --2- 172.21.15.189:0/1058076382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca7c066ab0 0x7fca7c0a9ce0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:44.071 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.055+0000 7fca83c7f700 1 -- 172.21.15.189:0/1058076382 >> 172.21.15.189:0/1058076382 conn(0x7fca7c0627f0 msgr2=0x7fca7c065870 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:44.071 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.055+0000 7fca83c7f700 1 -- 172.21.15.189:0/1058076382 shutdown_connections 2020-11-20T12:20:44.072 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.055+0000 7fca83c7f700 1 -- 172.21.15.189:0/1058076382 wait complete. 2020-11-20T12:20:44.077 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.055+0000 7fca83c7f700 1 Processor -- start 2020-11-20T12:20:44.077 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.055+0000 7fca83c7f700 1 -- start start 2020-11-20T12:20:44.078 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.055+0000 7fca83c7f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca7c066ab0 0x7fca7c13cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:44.078 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.055+0000 7fca83c7f700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fca7c0a8340 0x7fca7c0a2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:44.079 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.055+0000 7fca83c7f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca7c0a8bc0 0x7fca7c0a3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:44.079 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.055+0000 7fca83c7f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c0a3f10 con 0x7fca7c0a8bc0 2020-11-20T12:20:44.079 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.055+0000 7fca83c7f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c0a25d0 con 0x7fca7c066ab0 2020-11-20T12:20:44.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.055+0000 7fca83c7f700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c0a5e90 con 0x7fca7c0a8340 2020-11-20T12:20:44.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.055+0000 7fca81a1b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca7c066ab0 0x7fca7c13cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:44.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.055+0000 7fca81a1b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca7c066ab0 0x7fca7c13cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51756/0 (socket says 172.21.15.189:51756) 2020-11-20T12:20:44.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.055+0000 7fca81a1b700 1 -- 172.21.15.189:0/416580099 learned_addr learned my addr 172.21.15.189:0/416580099 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:20:44.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.055+0000 7fca8221c700 1 --2- 172.21.15.189:0/416580099 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca7c0a8bc0 0x7fca7c0a3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:44.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.055+0000 7fca8121a700 1 --2- 172.21.15.189:0/416580099 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fca7c0a8340 0x7fca7c0a2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:44.082 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.059+0000 7fca81a1b700 1 -- 172.21.15.189:0/416580099 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fca7c0a8340 msgr2=0x7fca7c0a2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:44.082 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.059+0000 7fca81a1b700 1 --2- 172.21.15.189:0/416580099 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fca7c0a8340 0x7fca7c0a2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:44.082 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.059+0000 7fca81a1b700 1 -- 172.21.15.189:0/416580099 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca7c0a8bc0 msgr2=0x7fca7c0a3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:44.082 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.059+0000 7fca81a1b700 1 --2- 172.21.15.189:0/416580099 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca7c0a8bc0 0x7fca7c0a3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:44.082 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.059+0000 7fca81a1b700 1 -- 172.21.15.189:0/416580099 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7c149a00 con 0x7fca7c066ab0 2020-11-20T12:20:44.083 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.059+0000 7fca81a1b700 1 --2- 172.21.15.189:0/416580099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca7c066ab0 0x7fca7c13cc20 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0x7fca7800dcb0 tx=0x7fca78008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:44.083 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.059+0000 7fca69ffb700 1 -- 172.21.15.189:0/416580099 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7800b9f0 con 0x7fca7c066ab0 2020-11-20T12:20:44.083 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.059+0000 7fca69ffb700 1 -- 172.21.15.189:0/416580099 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7800bb50 con 0x7fca7c066ab0 2020-11-20T12:20:44.083 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.059+0000 7fca69ffb700 1 -- 172.21.15.189:0/416580099 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca78012420 con 0x7fca7c066ab0 2020-11-20T12:20:44.083 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.059+0000 7fca83c7f700 1 -- 172.21.15.189:0/416580099 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca7c0a2f60 con 0x7fca7c066ab0 2020-11-20T12:20:44.084 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.059+0000 7fca83c7f700 1 -- 172.21.15.189:0/416580099 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca7c0a3c20 con 0x7fca7c066ab0 2020-11-20T12:20:44.084 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.059+0000 7fca83c7f700 1 -- 172.21.15.189:0/416580099 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca7c0a3150 con 0x7fca7c066ab0 2020-11-20T12:20:44.084 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.059+0000 7fca69ffb700 1 -- 172.21.15.189:0/416580099 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fca78012790 con 0x7fca7c066ab0 2020-11-20T12:20:44.084 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.059+0000 7fca69ffb700 1 --2- 172.21.15.189:0/416580099 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fca540557c0 0x7fca54057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:44.084 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.059+0000 7fca69ffb700 1 -- 172.21.15.189:0/416580099 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7019+0+0 (secure 0 0 0) 0x7fca7806ea80 con 0x7fca7c066ab0 2020-11-20T12:20:44.085 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.059+0000 7fca8121a700 1 --2- 172.21.15.189:0/416580099 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fca540557c0 0x7fca54057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:44.085 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.059+0000 7fca8121a700 1 --2- 172.21.15.189:0/416580099 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fca540557c0 0x7fca54057c80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7fca6c006e90 tx=0x7fca6c015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:44.085 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.067+0000 7fca69ffb700 1 -- 172.21.15.189:0/416580099 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fca78044570 con 0x7fca7c066ab0 2020-11-20T12:20:44.279 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.271+0000 7fca83c7f700 1 -- 172.21.15.189:0/416580099 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca7c0a4080 con 0x7fca7c066ab0 2020-11-20T12:20:44.279 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.275+0000 7fca69ffb700 1 -- 172.21.15.189:0/416580099 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2137 (secure 0 0 0) 0x7fca78047b90 con 0x7fca7c066ab0 2020-11-20T12:20:44.280 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:20:44.280 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5746,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.189:6835/3538437301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3538437301},{"type":"v1","addr":"172.21.15.189:6835","nonce":3538437301}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:20:05.807690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5520":{"gid":5520,"name":"b","rank":0,"incarnation":57,"state":"up:active","state_seq":32,"addr":"172.21.15.47:6833/201662157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":201662157},{"type":"v1","addr":"172.21.15.47:6833","nonce":201662157}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:20:44.282 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.275+0000 7fca83c7f700 1 -- 172.21.15.189:0/416580099 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fca540557c0 msgr2=0x7fca54057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:44.282 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.275+0000 7fca83c7f700 1 --2- 172.21.15.189:0/416580099 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fca540557c0 0x7fca54057c80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7fca6c006e90 tx=0x7fca6c015000).stop 2020-11-20T12:20:44.282 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.275+0000 7fca83c7f700 1 -- 172.21.15.189:0/416580099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca7c066ab0 msgr2=0x7fca7c13cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:44.283 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.275+0000 7fca83c7f700 1 --2- 172.21.15.189:0/416580099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca7c066ab0 0x7fca7c13cc20 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0x7fca7800dcb0 tx=0x7fca78008040).stop 2020-11-20T12:20:44.283 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.275+0000 7fca83c7f700 1 -- 172.21.15.189:0/416580099 shutdown_connections 2020-11-20T12:20:44.283 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.275+0000 7fca83c7f700 1 --2- 172.21.15.189:0/416580099 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fca540557c0 0x7fca54057c80 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:44.283 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.275+0000 7fca83c7f700 1 --2- 172.21.15.189:0/416580099 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fca7c0a8bc0 0x7fca7c0a3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:44.283 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.275+0000 7fca83c7f700 1 --2- 172.21.15.189:0/416580099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca7c066ab0 0x7fca7c13cc20 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:44.284 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.275+0000 7fca83c7f700 1 --2- 172.21.15.189:0/416580099 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fca7c0a8340 0x7fca7c0a2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:44.284 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.275+0000 7fca83c7f700 1 -- 172.21.15.189:0/416580099 >> 172.21.15.189:0/416580099 conn(0x7fca7c0627f0 msgr2=0x7fca7c05ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:44.284 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.275+0000 7fca83c7f700 1 -- 172.21.15.189:0/416580099 shutdown_connections 2020-11-20T12:20:44.284 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:44.275+0000 7fca83c7f700 1 -- 172.21.15.189:0/416580099 wait complete. 2020-11-20T12:20:44.285 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 61 2020-11-20T12:20:44.299 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:20:46.300 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:46.433 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.427+0000 7f72b2128700 1 Processor -- start 2020-11-20T12:20:46.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.427+0000 7f72b2128700 1 -- start start 2020-11-20T12:20:46.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.427+0000 7f72b2128700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f72ac060170 0x7f72ac065fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:46.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.427+0000 7f72b2128700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f72ac0a8870 0x7f72ac136b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:46.436 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.427+0000 7f72b2128700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ac065b40 con 0x7f72ac0a9660 2020-11-20T12:20:46.436 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.427+0000 7f72b2128700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72ac060990 con 0x7f72ac0a8870 2020-11-20T12:20:46.436 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.427+0000 7f72b2128700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72ac0a8cd0 con 0x7f72ac060170 2020-11-20T12:20:46.436 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.427+0000 7f72abfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f72ac0a9660 0x7f72ac0a62e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53394/0 (socket says 172.21.15.189:53394) 2020-11-20T12:20:46.437 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.427+0000 7f72abfff700 1 -- 172.21.15.189:0/1809265821 learned_addr learned my addr 172.21.15.189:0/1809265821 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:20:46.437 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.427+0000 7f72ab7fe700 1 --2- 172.21.15.189:0/1809265821 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f72ac060170 0x7f72ac065fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:46.437 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.427+0000 7f72aaffd700 1 --2- 172.21.15.189:0/1809265821 >> v2:172.21.15.47:3300/0 conn(0x7f72ac0a8870 0x7f72ac136b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:46.437 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.427+0000 7f72aa7fc700 1 -- 172.21.15.189:0/1809265821 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203965450 0 0) 0x7f72ac065b40 con 0x7f72ac0a9660 2020-11-20T12:20:46.437 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.427+0000 7f72aa7fc700 1 -- 172.21.15.189:0/1809265821 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f728c003240 con 0x7f72ac0a9660 2020-11-20T12:20:46.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.431+0000 7f72ab7fe700 1 -- 172.21.15.189:0/1809265821 >> v2:172.21.15.47:3300/0 conn(0x7f72ac0a8870 msgr2=0x7f72ac136b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:46.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.431+0000 7f72ab7fe700 1 --2- 172.21.15.189:0/1809265821 >> v2:172.21.15.47:3300/0 conn(0x7f72ac0a8870 0x7f72ac136b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:46.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.431+0000 7f72ab7fe700 1 -- 172.21.15.189:0/1809265821 >> v1:172.21.15.189:6789/0 conn(0x7f72ac0a9660 legacy=0x7f72ac0a62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:46.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.431+0000 7f72ab7fe700 1 -- 172.21.15.189:0/1809265821 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ac0606f0 con 0x7f72ac060170 2020-11-20T12:20:46.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.431+0000 7f72aaffd700 1 --2- 172.21.15.189:0/1809265821 >> v2:172.21.15.47:3300/0 conn(0x7f72ac0a8870 0x7f72ac136b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:20:46.439 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.431+0000 7f72ab7fe700 1 --2- 172.21.15.189:0/1809265821 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f72ac060170 0x7f72ac065fa0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f729c00b9a0 tx=0x7f729c006df0).ready entity=mon.2 client_cookie=9a77bf546d87f5ea server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:46.441 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.431+0000 7f72aa7fc700 1 -- 172.21.15.189:0/1809265821 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f729c009680 con 0x7f72ac060170 2020-11-20T12:20:46.442 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.431+0000 7f72aa7fc700 1 -- 172.21.15.189:0/1809265821 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f729c0097e0 con 0x7f72ac060170 2020-11-20T12:20:46.442 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.431+0000 7f72aa7fc700 1 -- 172.21.15.189:0/1809265821 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f729c009b60 con 0x7f72ac060170 2020-11-20T12:20:46.442 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.431+0000 7f72b2128700 1 -- 172.21.15.189:0/1809265821 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f72ac060170 msgr2=0x7f72ac065fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:46.442 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.431+0000 7f72b2128700 1 --2- 172.21.15.189:0/1809265821 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f72ac060170 0x7f72ac065fa0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f729c00b9a0 tx=0x7f729c006df0).stop 2020-11-20T12:20:46.443 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.431+0000 7f72b2128700 1 -- 172.21.15.189:0/1809265821 shutdown_connections 2020-11-20T12:20:46.443 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.431+0000 7f72b2128700 1 --2- 172.21.15.189:0/1809265821 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f72ac060170 0x7f72ac065fa0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:46.443 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.431+0000 7f72b2128700 1 --2- 172.21.15.189:0/1809265821 >> v2:172.21.15.47:3300/0 conn(0x7f72ac0a8870 0x7f72ac136b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:46.443 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.431+0000 7f72b2128700 1 -- 172.21.15.189:0/1809265821 >> 172.21.15.189:0/1809265821 conn(0x7f72ac061f60 msgr2=0x7f72ac062370 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:46.443 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.431+0000 7f72b2128700 1 -- 172.21.15.189:0/1809265821 shutdown_connections 2020-11-20T12:20:46.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.431+0000 7f72b2128700 1 -- 172.21.15.189:0/1809265821 wait complete. 2020-11-20T12:20:46.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.431+0000 7f72b2128700 1 Processor -- start 2020-11-20T12:20:46.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.431+0000 7f72b2128700 1 -- start start 2020-11-20T12:20:46.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.435+0000 7f72b2128700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72ac060170 0x7f72ac0a1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:46.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.435+0000 7f72b2128700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72ac0a8870 0x7f72ac09e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:46.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.435+0000 7f72b2128700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f72ac0a9660 0x7f72ac138720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:46.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.435+0000 7f72b2128700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ac0a0af0 con 0x7f72ac060170 2020-11-20T12:20:46.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.435+0000 7f72b2128700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ac0a0d20 con 0x7f72ac0a8870 2020-11-20T12:20:46.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.435+0000 7f72b2128700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72ac0a0fc0 con 0x7f72ac0a9660 2020-11-20T12:20:46.445 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.435+0000 7f72aaffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72ac0a8870 0x7f72ac09e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:46.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.435+0000 7f72aaffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72ac0a8870 0x7f72ac09e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51776/0 (socket says 172.21.15.189:51776) 2020-11-20T12:20:46.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.435+0000 7f72aaffd700 1 -- 172.21.15.189:0/1148032040 learned_addr learned my addr 172.21.15.189:0/1148032040 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:20:46.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.435+0000 7f72ab7fe700 1 --2- 172.21.15.189:0/1148032040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72ac060170 0x7f72ac0a1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:46.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.435+0000 7f72abfff700 1 --2- 172.21.15.189:0/1148032040 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f72ac0a9660 0x7f72ac138720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:46.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.435+0000 7f72aaffd700 1 -- 172.21.15.189:0/1148032040 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f72ac0a9660 msgr2=0x7f72ac138720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:46.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.435+0000 7f72aaffd700 1 --2- 172.21.15.189:0/1148032040 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f72ac0a9660 0x7f72ac138720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:46.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.435+0000 7f72aaffd700 1 -- 172.21.15.189:0/1148032040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72ac060170 msgr2=0x7f72ac0a1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:46.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.435+0000 7f72aaffd700 1 --2- 172.21.15.189:0/1148032040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72ac060170 0x7f72ac0a1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:46.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.435+0000 7f72aaffd700 1 -- 172.21.15.189:0/1148032040 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ac137c80 con 0x7f72ac0a8870 2020-11-20T12:20:46.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.435+0000 7f72aaffd700 1 --2- 172.21.15.189:0/1148032040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72ac0a8870 0x7f72ac09e630 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7f729400ed20 tx=0x7f7294006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:46.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.435+0000 7f72a37fe700 1 -- 172.21.15.189:0/1148032040 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f729400cb90 con 0x7f72ac0a8870 2020-11-20T12:20:46.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.435+0000 7f72a37fe700 1 -- 172.21.15.189:0/1148032040 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f729400ccf0 con 0x7f72ac0a8870 2020-11-20T12:20:46.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.435+0000 7f72b2128700 1 -- 172.21.15.189:0/1148032040 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72ac0a20e0 con 0x7f72ac0a8870 2020-11-20T12:20:46.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.435+0000 7f72a37fe700 1 -- 172.21.15.189:0/1148032040 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7294012600 con 0x7f72ac0a8870 2020-11-20T12:20:46.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.435+0000 7f72b2128700 1 -- 172.21.15.189:0/1148032040 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72ac131500 con 0x7f72ac0a8870 2020-11-20T12:20:46.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.435+0000 7f72b2128700 1 -- 172.21.15.189:0/1148032040 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72ac0a1180 con 0x7f72ac0a8870 2020-11-20T12:20:46.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.439+0000 7f72a37fe700 1 -- 172.21.15.189:0/1148032040 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f72940129f0 con 0x7f72ac0a8870 2020-11-20T12:20:46.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.439+0000 7f72a37fe700 1 --2- 172.21.15.189:0/1148032040 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f72880557c0 0x7f7288057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:46.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.439+0000 7f72a37fe700 1 -- 172.21.15.189:0/1148032040 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7019+0+0 (secure 0 0 0) 0x7f729406fbd0 con 0x7f72ac0a8870 2020-11-20T12:20:46.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.439+0000 7f72ab7fe700 1 --2- 172.21.15.189:0/1148032040 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f72880557c0 0x7f7288057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:46.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.439+0000 7f72ab7fe700 1 --2- 172.21.15.189:0/1148032040 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f72880557c0 0x7f7288057c80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f729c006e30 tx=0x7f729c015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:46.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.443+0000 7f72a37fe700 1 -- 172.21.15.189:0/1148032040 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7294045630 con 0x7f72ac0a8870 2020-11-20T12:20:46.656 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.651+0000 7f72b2128700 1 -- 172.21.15.189:0/1148032040 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72ac044f60 con 0x7f72ac0a8870 2020-11-20T12:20:46.657 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.651+0000 7f72a37fe700 1 -- 172.21.15.189:0/1148032040 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f729400ce60 con 0x7f72ac0a8870 2020-11-20T12:20:46.657 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:20:46.658 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5746,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.189:6835/3538437301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3538437301},{"type":"v1","addr":"172.21.15.189:6835","nonce":3538437301}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:20:05.807690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5520":{"gid":5520,"name":"b","rank":0,"incarnation":57,"state":"up:active","state_seq":32,"addr":"172.21.15.47:6833/201662157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":201662157},{"type":"v1","addr":"172.21.15.47:6833","nonce":201662157}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:20:46.659 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.651+0000 7f72b2128700 1 -- 172.21.15.189:0/1148032040 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f72880557c0 msgr2=0x7f7288057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:46.659 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.651+0000 7f72b2128700 1 --2- 172.21.15.189:0/1148032040 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f72880557c0 0x7f7288057c80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f729c006e30 tx=0x7f729c015000).stop 2020-11-20T12:20:46.659 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.651+0000 7f72b2128700 1 -- 172.21.15.189:0/1148032040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72ac0a8870 msgr2=0x7f72ac09e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:46.660 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.651+0000 7f72b2128700 1 --2- 172.21.15.189:0/1148032040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72ac0a8870 0x7f72ac09e630 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7f729400ed20 tx=0x7f7294006df0).stop 2020-11-20T12:20:46.660 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.651+0000 7f72b2128700 1 -- 172.21.15.189:0/1148032040 shutdown_connections 2020-11-20T12:20:46.660 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.651+0000 7f72b2128700 1 --2- 172.21.15.189:0/1148032040 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f72880557c0 0x7f7288057c80 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:46.660 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.651+0000 7f72b2128700 1 --2- 172.21.15.189:0/1148032040 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f72ac060170 0x7f72ac0a1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:46.661 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.651+0000 7f72b2128700 1 --2- 172.21.15.189:0/1148032040 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f72ac0a9660 0x7f72ac138720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:46.661 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.651+0000 7f72b2128700 1 --2- 172.21.15.189:0/1148032040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72ac0a8870 0x7f72ac09e630 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:46.661 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.651+0000 7f72b2128700 1 -- 172.21.15.189:0/1148032040 >> 172.21.15.189:0/1148032040 conn(0x7f72ac061f60 msgr2=0x7f72ac134e00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:46.661 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.651+0000 7f72b2128700 1 -- 172.21.15.189:0/1148032040 shutdown_connections 2020-11-20T12:20:46.662 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:46.655+0000 7f72b2128700 1 -- 172.21.15.189:0/1148032040 wait complete. 2020-11-20T12:20:46.662 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 61 2020-11-20T12:20:46.678 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:20:48.679 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:48.812 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef32dd0700 1 Processor -- start 2020-11-20T12:20:48.813 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef32dd0700 1 -- start start 2020-11-20T12:20:48.814 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef32dd0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fef2c066ed0 0x7fef2c13b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:48.815 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef32dd0700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef2c13ba60 0x7fef2c13bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:48.815 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef32dd0700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef2c13c3c0 con 0x7fef2c0a8840 2020-11-20T12:20:48.815 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef32dd0700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef2c066800 con 0x7fef2c066ed0 2020-11-20T12:20:48.815 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef32dd0700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c066aa0 con 0x7fef2c13ba60 2020-11-20T12:20:48.815 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef315cd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fef2c066ed0 0x7fef2c13b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:48.816 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef315cd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fef2c066ed0 0x7fef2c13b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51786/0 (socket says 172.21.15.189:51786) 2020-11-20T12:20:48.816 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef315cd700 1 -- 172.21.15.189:0/2806090386 learned_addr learned my addr 172.21.15.189:0/2806090386 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:20:48.816 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef315cd700 1 -- 172.21.15.189:0/2806090386 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef2c13ba60 msgr2=0x7fef2c13bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:48.816 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef325cf700 1 --2- 172.21.15.189:0/2806090386 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef2c13ba60 0x7fef2c13bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:48.816 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef315cd700 1 --2- 172.21.15.189:0/2806090386 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef2c13ba60 0x7fef2c13bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:48.817 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef315cd700 1 -- 172.21.15.189:0/2806090386 >> v1:172.21.15.47:6789/0 conn(0x7fef2c0a8840 legacy=0x7fef2c13eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:48.817 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef315cd700 1 -- 172.21.15.189:0/2806090386 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2c067450 con 0x7fef2c066ed0 2020-11-20T12:20:48.817 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef325cf700 1 --2- 172.21.15.189:0/2806090386 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef2c13ba60 0x7fef2c13bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:48.817 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef315cd700 1 --2- 172.21.15.189:0/2806090386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef2c066ed0 0x7fef2c13b570 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7fef28009d50 tx=0x7fef28009290).ready entity=mon.1 client_cookie=a2e10ea5bf8fa09a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:48.817 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef30dcc700 1 -- 172.21.15.189:0/2806090386 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2800bcd0 con 0x7fef2c066ed0 2020-11-20T12:20:48.818 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef30dcc700 1 -- 172.21.15.189:0/2806090386 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef2800be30 con 0x7fef2c066ed0 2020-11-20T12:20:48.818 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef30dcc700 1 -- 172.21.15.189:0/2806090386 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef28011610 con 0x7fef2c066ed0 2020-11-20T12:20:48.818 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef32dd0700 1 -- 172.21.15.189:0/2806090386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef2c066ed0 msgr2=0x7fef2c13b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:48.818 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef32dd0700 1 --2- 172.21.15.189:0/2806090386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef2c066ed0 0x7fef2c13b570 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7fef28009d50 tx=0x7fef28009290).stop 2020-11-20T12:20:48.820 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef32dd0700 1 -- 172.21.15.189:0/2806090386 shutdown_connections 2020-11-20T12:20:48.821 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef32dd0700 1 --2- 172.21.15.189:0/2806090386 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef2c13ba60 0x7fef2c13bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:48.821 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef32dd0700 1 --2- 172.21.15.189:0/2806090386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef2c066ed0 0x7fef2c13b570 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:48.821 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef32dd0700 1 -- 172.21.15.189:0/2806090386 >> 172.21.15.189:0/2806090386 conn(0x7fef2c060170 msgr2=0x7fef2c060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:48.821 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef32dd0700 1 -- 172.21.15.189:0/2806090386 shutdown_connections 2020-11-20T12:20:48.822 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.807+0000 7fef32dd0700 1 -- 172.21.15.189:0/2806090386 wait complete. 2020-11-20T12:20:48.822 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.811+0000 7fef32dd0700 1 Processor -- start 2020-11-20T12:20:48.822 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.811+0000 7fef32dd0700 1 -- start start 2020-11-20T12:20:48.822 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.811+0000 7fef32dd0700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef2c066ed0 0x7fef2c0a25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:48.822 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.811+0000 7fef32dd0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef2c0a8840 0x7fef2c140440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:48.823 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.811+0000 7fef32dd0700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef2c13ba60 0x7fef2c0a12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:48.823 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.815+0000 7fef32dd0700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c0a1c70 con 0x7fef2c066ed0 2020-11-20T12:20:48.823 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.815+0000 7fef32dd0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c0a1ea0 con 0x7fef2c0a8840 2020-11-20T12:20:48.823 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.815+0000 7fef32dd0700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c0a2140 con 0x7fef2c13ba60 2020-11-20T12:20:48.823 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.815+0000 7fef315cd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef2c0a8840 0x7fef2c140440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:48.824 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.815+0000 7fef315cd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef2c0a8840 0x7fef2c140440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51794/0 (socket says 172.21.15.189:51794) 2020-11-20T12:20:48.824 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.815+0000 7fef315cd700 1 -- 172.21.15.189:0/1381778274 learned_addr learned my addr 172.21.15.189:0/1381778274 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:20:48.824 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.815+0000 7fef31dce700 1 --2- 172.21.15.189:0/1381778274 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef2c066ed0 0x7fef2c0a25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:48.824 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.815+0000 7fef325cf700 1 --2- 172.21.15.189:0/1381778274 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef2c13ba60 0x7fef2c0a12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:48.824 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.815+0000 7fef315cd700 1 -- 172.21.15.189:0/1381778274 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef2c13ba60 msgr2=0x7fef2c0a12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:48.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.815+0000 7fef315cd700 1 --2- 172.21.15.189:0/1381778274 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef2c13ba60 0x7fef2c0a12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:48.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.815+0000 7fef315cd700 1 -- 172.21.15.189:0/1381778274 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef2c066ed0 msgr2=0x7fef2c0a25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:48.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.815+0000 7fef315cd700 1 --2- 172.21.15.189:0/1381778274 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef2c066ed0 0x7fef2c0a25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:48.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.815+0000 7fef315cd700 1 -- 172.21.15.189:0/1381778274 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2c0a17d0 con 0x7fef2c0a8840 2020-11-20T12:20:48.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.815+0000 7fef315cd700 1 --2- 172.21.15.189:0/1381778274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef2c0a8840 0x7fef2c140440 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7fef2c064a70 tx=0x7fef28016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:48.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.815+0000 7fef19ffb700 1 -- 172.21.15.189:0/1381778274 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef28010700 con 0x7fef2c0a8840 2020-11-20T12:20:48.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.815+0000 7fef19ffb700 1 -- 172.21.15.189:0/1381778274 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef28010860 con 0x7fef2c0a8840 2020-11-20T12:20:48.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.815+0000 7fef19ffb700 1 -- 172.21.15.189:0/1381778274 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef28010be0 con 0x7fef2c0a8840 2020-11-20T12:20:48.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.815+0000 7fef32dd0700 1 -- 172.21.15.189:0/1381778274 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef2c0a68f0 con 0x7fef2c0a8840 2020-11-20T12:20:48.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.815+0000 7fef32dd0700 1 -- 172.21.15.189:0/1381778274 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef2c1515b0 con 0x7fef2c0a8840 2020-11-20T12:20:48.827 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.815+0000 7fef32dd0700 1 -- 172.21.15.189:0/1381778274 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef2c0a6ab0 con 0x7fef2c0a8840 2020-11-20T12:20:48.828 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.815+0000 7fef19ffb700 1 -- 172.21.15.189:0/1381778274 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fef28017070 con 0x7fef2c0a8840 2020-11-20T12:20:48.828 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.819+0000 7fef19ffb700 1 --2- 172.21.15.189:0/1381778274 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fef040557f0 0x7fef04057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:48.828 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.819+0000 7fef31dce700 1 --2- 172.21.15.189:0/1381778274 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fef040557f0 0x7fef04057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:48.828 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.819+0000 7fef19ffb700 1 -- 172.21.15.189:0/1381778274 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7019+0+0 (secure 0 0 0) 0x7fef28070cb0 con 0x7fef2c0a8840 2020-11-20T12:20:48.828 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.819+0000 7fef31dce700 1 --2- 172.21.15.189:0/1381778274 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fef040557f0 0x7fef04057cb0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fef1c003500 tx=0x7fef1c009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:48.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:48.823+0000 7fef19ffb700 1 -- 172.21.15.189:0/1381778274 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fef28046710 con 0x7fef2c0a8840 2020-11-20T12:20:49.020 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:49.015+0000 7fef32dd0700 1 -- 172.21.15.189:0/1381778274 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef2c0a2300 con 0x7fef2c0a8840 2020-11-20T12:20:49.021 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:49.015+0000 7fef19ffb700 1 -- 172.21.15.189:0/1381778274 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2137 (secure 0 0 0) 0x7fef28015070 con 0x7fef2c0a8840 2020-11-20T12:20:49.021 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:20:49.022 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5746,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.189:6835/3538437301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3538437301},{"type":"v1","addr":"172.21.15.189:6835","nonce":3538437301}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:20:05.807690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5520":{"gid":5520,"name":"b","rank":0,"incarnation":57,"state":"up:active","state_seq":32,"addr":"172.21.15.47:6833/201662157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":201662157},{"type":"v1","addr":"172.21.15.47:6833","nonce":201662157}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:20:49.024 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:49.019+0000 7fef32dd0700 1 -- 172.21.15.189:0/1381778274 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fef040557f0 msgr2=0x7fef04057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:49.024 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:49.019+0000 7fef32dd0700 1 --2- 172.21.15.189:0/1381778274 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fef040557f0 0x7fef04057cb0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fef1c003500 tx=0x7fef1c009040).stop 2020-11-20T12:20:49.024 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:49.019+0000 7fef32dd0700 1 -- 172.21.15.189:0/1381778274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef2c0a8840 msgr2=0x7fef2c140440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:49.025 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:49.019+0000 7fef32dd0700 1 --2- 172.21.15.189:0/1381778274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef2c0a8840 0x7fef2c140440 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7fef2c064a70 tx=0x7fef28016000).stop 2020-11-20T12:20:49.025 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:49.019+0000 7fef32dd0700 1 -- 172.21.15.189:0/1381778274 shutdown_connections 2020-11-20T12:20:49.025 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:49.019+0000 7fef32dd0700 1 --2- 172.21.15.189:0/1381778274 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fef040557f0 0x7fef04057cb0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:49.025 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:49.019+0000 7fef32dd0700 1 --2- 172.21.15.189:0/1381778274 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fef2c066ed0 0x7fef2c0a25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:49.025 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:49.019+0000 7fef32dd0700 1 --2- 172.21.15.189:0/1381778274 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fef2c13ba60 0x7fef2c0a12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:49.026 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:49.019+0000 7fef32dd0700 1 --2- 172.21.15.189:0/1381778274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef2c0a8840 0x7fef2c140440 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:49.026 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:49.019+0000 7fef32dd0700 1 -- 172.21.15.189:0/1381778274 >> 172.21.15.189:0/1381778274 conn(0x7fef2c060170 msgr2=0x7fef2c0a7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:49.026 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:49.019+0000 7fef32dd0700 1 -- 172.21.15.189:0/1381778274 shutdown_connections 2020-11-20T12:20:49.026 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:49.019+0000 7fef32dd0700 1 -- 172.21.15.189:0/1381778274 wait complete. 2020-11-20T12:20:49.027 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 61 2020-11-20T12:20:49.043 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:20:51.044 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:51.172 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.167+0000 7fa86ffff700 1 Processor -- start 2020-11-20T12:20:51.173 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.167+0000 7fa86ffff700 1 -- start start 2020-11-20T12:20:51.175 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.167+0000 7fa86ffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa8700ad000 0x7fa87013e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:51.175 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.167+0000 7fa86ffff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa870067730 con 0x7fa87013a6e0 2020-11-20T12:20:51.175 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.167+0000 7fa86ffff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa87013dd40 con 0x7fa8700ad3e0 2020-11-20T12:20:51.175 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.167+0000 7fa86ffff700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa870138190 con 0x7fa8700ad000 2020-11-20T12:20:51.176 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.167+0000 7fa86effd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa8700ad000 0x7fa87013e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:51.176 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.167+0000 7fa86effd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa8700ad000 0x7fa87013e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51802/0 (socket says 172.21.15.189:51802) 2020-11-20T12:20:51.176 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.167+0000 7fa86effd700 1 -- 172.21.15.189:0/2175173278 learned_addr learned my addr 172.21.15.189:0/2175173278 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:20:51.176 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.167+0000 7fa86e7fc700 1 --1- 172.21.15.189:0/2175173278 >> v1:172.21.15.189:6789/0 conn(0x7fa8700ad3e0 0x7fa870066d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53428/0 (socket says 172.21.15.189:53428) 2020-11-20T12:20:51.177 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.167+0000 7fa86effd700 1 -- 172.21.15.189:0/2175173278 >> v1:172.21.15.189:6789/0 conn(0x7fa8700ad3e0 legacy=0x7fa870066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:51.177 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.167+0000 7fa86effd700 1 -- 172.21.15.189:0/2175173278 >> v1:172.21.15.47:6789/0 conn(0x7fa87013a6e0 legacy=0x7fa87013ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:51.178 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.167+0000 7fa86effd700 1 -- 172.21.15.189:0/2175173278 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87013dfe0 con 0x7fa8700ad000 2020-11-20T12:20:51.178 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.167+0000 7fa86effd700 1 --2- 172.21.15.189:0/2175173278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8700ad000 0x7fa87013e480 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7fa860006fd0 tx=0x7fa860006eb0).ready entity=mon.1 client_cookie=4e01b2fceb168abf server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:51.178 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.167+0000 7fa86dffb700 1 -- 172.21.15.189:0/2175173278 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8600098d0 con 0x7fa8700ad000 2020-11-20T12:20:51.179 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.167+0000 7fa86dffb700 1 -- 172.21.15.189:0/2175173278 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa860009a30 con 0x7fa8700ad000 2020-11-20T12:20:51.179 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.167+0000 7fa86dffb700 1 -- 172.21.15.189:0/2175173278 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa860009db0 con 0x7fa8700ad000 2020-11-20T12:20:51.182 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86ffff700 1 -- 172.21.15.189:0/2175173278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8700ad000 msgr2=0x7fa87013e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:51.183 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86ffff700 1 --2- 172.21.15.189:0/2175173278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8700ad000 0x7fa87013e480 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7fa860006fd0 tx=0x7fa860006eb0).stop 2020-11-20T12:20:51.183 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86ffff700 1 -- 172.21.15.189:0/2175173278 shutdown_connections 2020-11-20T12:20:51.185 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86ffff700 1 --2- 172.21.15.189:0/2175173278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8700ad000 0x7fa87013e480 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:51.185 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86ffff700 1 -- 172.21.15.189:0/2175173278 >> 172.21.15.189:0/2175173278 conn(0x7fa870062080 msgr2=0x7fa870062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:51.185 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86ffff700 1 -- 172.21.15.189:0/2175173278 shutdown_connections 2020-11-20T12:20:51.185 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86ffff700 1 -- 172.21.15.189:0/2175173278 wait complete. 2020-11-20T12:20:51.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86ffff700 1 Processor -- start 2020-11-20T12:20:51.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86ffff700 1 -- start start 2020-11-20T12:20:51.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8700ad000 0x7fa8700a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:51.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8700ad3e0 0x7fa870149f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:51.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86ffff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa8700af280 0x7fa8700af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:51.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86ffff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87014a7c0 con 0x7fa8700ad000 2020-11-20T12:20:51.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86ffff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8700aede0 con 0x7fa8700ad3e0 2020-11-20T12:20:51.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86ffff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8700afbb0 con 0x7fa8700af280 2020-11-20T12:20:51.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8700ad3e0 0x7fa870149f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:51.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8700ad3e0 0x7fa870149f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51808/0 (socket says 172.21.15.189:51808) 2020-11-20T12:20:51.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86e7fc700 1 -- 172.21.15.189:0/518741942 learned_addr learned my addr 172.21.15.189:0/518741942 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:20:51.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86effd700 1 --2- 172.21.15.189:0/518741942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8700ad000 0x7fa8700a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:51.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86f7fe700 1 --2- 172.21.15.189:0/518741942 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa8700af280 0x7fa8700af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:51.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86e7fc700 1 -- 172.21.15.189:0/518741942 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa8700af280 msgr2=0x7fa8700af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:51.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86e7fc700 1 --2- 172.21.15.189:0/518741942 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa8700af280 0x7fa8700af6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:51.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86e7fc700 1 -- 172.21.15.189:0/518741942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8700ad000 msgr2=0x7fa8700a53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:51.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86e7fc700 1 --2- 172.21.15.189:0/518741942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8700ad000 0x7fa8700a53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:51.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86e7fc700 1 -- 172.21.15.189:0/518741942 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87014a450 con 0x7fa8700ad3e0 2020-11-20T12:20:51.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86effd700 1 --2- 172.21.15.189:0/518741942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8700ad000 0x7fa8700a53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:51.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86f7fe700 1 --2- 172.21.15.189:0/518741942 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa8700af280 0x7fa8700af6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:51.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.171+0000 7fa86e7fc700 1 --2- 172.21.15.189:0/518741942 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8700ad3e0 0x7fa870149f60 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7fa86400c710 tx=0x7fa864007c40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:51.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.175+0000 7fa856ffd700 1 -- 172.21.15.189:0/518741942 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa864007320 con 0x7fa8700ad3e0 2020-11-20T12:20:51.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.175+0000 7fa856ffd700 1 -- 172.21.15.189:0/518741942 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa864005530 con 0x7fa8700ad3e0 2020-11-20T12:20:51.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.175+0000 7fa86ffff700 1 -- 172.21.15.189:0/518741942 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8700b0a80 con 0x7fa8700ad3e0 2020-11-20T12:20:51.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.175+0000 7fa856ffd700 1 -- 172.21.15.189:0/518741942 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8640058b0 con 0x7fa8700ad3e0 2020-11-20T12:20:51.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.175+0000 7fa86ffff700 1 -- 172.21.15.189:0/518741942 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8700a9c50 con 0x7fa8700ad3e0 2020-11-20T12:20:51.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.175+0000 7fa86ffff700 1 -- 172.21.15.189:0/518741942 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8700aefa0 con 0x7fa8700ad3e0 2020-11-20T12:20:51.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.175+0000 7fa856ffd700 1 -- 172.21.15.189:0/518741942 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fa864010030 con 0x7fa8700ad3e0 2020-11-20T12:20:51.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.175+0000 7fa856ffd700 1 --2- 172.21.15.189:0/518741942 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa84c0557c0 0x7fa84c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:51.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.175+0000 7fa856ffd700 1 -- 172.21.15.189:0/518741942 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7019+0+0 (secure 0 0 0) 0x7fa86406e710 con 0x7fa8700ad3e0 2020-11-20T12:20:51.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.175+0000 7fa86effd700 1 --2- 172.21.15.189:0/518741942 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa84c0557c0 0x7fa84c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:51.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.175+0000 7fa86effd700 1 --2- 172.21.15.189:0/518741942 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa84c0557c0 0x7fa84c057c80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fa86000b1b0 tx=0x7fa860015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:51.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.183+0000 7fa856ffd700 1 -- 172.21.15.189:0/518741942 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa864044200 con 0x7fa8700ad3e0 2020-11-20T12:20:51.388 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.383+0000 7fa86ffff700 1 -- 172.21.15.189:0/518741942 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa87014a930 con 0x7fa8700ad3e0 2020-11-20T12:20:51.389 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.383+0000 7fa856ffd700 1 -- 172.21.15.189:0/518741942 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2137 (secure 0 0 0) 0x7fa864047820 con 0x7fa8700ad3e0 2020-11-20T12:20:51.389 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:20:51.389 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5746,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.189:6835/3538437301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3538437301},{"type":"v1","addr":"172.21.15.189:6835","nonce":3538437301}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:20:05.807690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5520":{"gid":5520,"name":"b","rank":0,"incarnation":57,"state":"up:active","state_seq":32,"addr":"172.21.15.47:6833/201662157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":201662157},{"type":"v1","addr":"172.21.15.47:6833","nonce":201662157}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:20:51.390 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.383+0000 7fa86ffff700 1 -- 172.21.15.189:0/518741942 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa84c0557c0 msgr2=0x7fa84c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:51.391 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.383+0000 7fa86ffff700 1 --2- 172.21.15.189:0/518741942 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa84c0557c0 0x7fa84c057c80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fa86000b1b0 tx=0x7fa860015000).stop 2020-11-20T12:20:51.391 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.383+0000 7fa86ffff700 1 -- 172.21.15.189:0/518741942 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8700ad3e0 msgr2=0x7fa870149f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:51.391 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.383+0000 7fa86ffff700 1 --2- 172.21.15.189:0/518741942 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8700ad3e0 0x7fa870149f60 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7fa86400c710 tx=0x7fa864007c40).stop 2020-11-20T12:20:51.391 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.383+0000 7fa86ffff700 1 -- 172.21.15.189:0/518741942 shutdown_connections 2020-11-20T12:20:51.392 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.383+0000 7fa86ffff700 1 --2- 172.21.15.189:0/518741942 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa84c0557c0 0x7fa84c057c80 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:51.392 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.383+0000 7fa86ffff700 1 --2- 172.21.15.189:0/518741942 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8700ad000 0x7fa8700a53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:51.392 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.383+0000 7fa86ffff700 1 --2- 172.21.15.189:0/518741942 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa8700af280 0x7fa8700af6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:51.392 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.383+0000 7fa86ffff700 1 --2- 172.21.15.189:0/518741942 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8700ad3e0 0x7fa870149f60 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:51.392 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.383+0000 7fa86ffff700 1 -- 172.21.15.189:0/518741942 >> 172.21.15.189:0/518741942 conn(0x7fa870062080 msgr2=0x7fa87005ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:51.393 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.383+0000 7fa86ffff700 1 -- 172.21.15.189:0/518741942 shutdown_connections 2020-11-20T12:20:51.393 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:51.383+0000 7fa86ffff700 1 -- 172.21.15.189:0/518741942 wait complete. 2020-11-20T12:20:51.393 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 61 2020-11-20T12:20:51.409 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:20:53.409 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:53.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd4cec2700 1 Processor -- start 2020-11-20T12:20:53.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd4cec2700 1 -- start start 2020-11-20T12:20:53.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd4cec2700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fcd481347f0 0x7fcd480677f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:53.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd4cec2700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd480a9c60 con 0x7fcd4813b1d0 2020-11-20T12:20:53.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd4cec2700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd48067e90 con 0x7fcd480623c0 2020-11-20T12:20:53.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd4cec2700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd480a8e60 con 0x7fcd481347f0 2020-11-20T12:20:53.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd477fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fcd480623c0 0x7fcd4813aac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53438/0 (socket says 172.21.15.189:53438) 2020-11-20T12:20:53.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd477fe700 1 -- 172.21.15.189:0/4083800868 learned_addr learned my addr 172.21.15.189:0/4083800868 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:20:53.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd47fff700 1 --2- 172.21.15.189:0/4083800868 >> v2:172.21.15.47:3300/0 conn(0x7fcd481347f0 0x7fcd480677f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:53.579 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd467fc700 1 -- 172.21.15.189:0/4083800868 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575708789 0 0) 0x7fcd48067e90 con 0x7fcd480623c0 2020-11-20T12:20:53.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd467fc700 1 -- 172.21.15.189:0/4083800868 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd28003240 con 0x7fcd480623c0 2020-11-20T12:20:53.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd467fc700 1 -- 172.21.15.189:0/4083800868 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3729500637 0 0) 0x7fcd28003240 con 0x7fcd480623c0 2020-11-20T12:20:53.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd467fc700 1 -- 172.21.15.189:0/4083800868 >> v2:172.21.15.47:3300/0 conn(0x7fcd481347f0 msgr2=0x7fcd480677f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:53.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd467fc700 1 --2- 172.21.15.189:0/4083800868 >> v2:172.21.15.47:3300/0 conn(0x7fcd481347f0 0x7fcd480677f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:53.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd467fc700 1 -- 172.21.15.189:0/4083800868 >> v1:172.21.15.47:6789/0 conn(0x7fcd4813b1d0 legacy=0x7fcd481340e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:53.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd467fc700 1 -- 172.21.15.189:0/4083800868 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd48068130 con 0x7fcd480623c0 2020-11-20T12:20:53.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd467fc700 1 -- 172.21.15.189:0/4083800868 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7fcd380014e0 con 0x7fcd480623c0 2020-11-20T12:20:53.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd467fc700 1 -- 172.21.15.189:0/4083800868 >> v1:172.21.15.189:6789/0 conn(0x7fcd480623c0 legacy=0x7fcd4813aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:53.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd467fc700 1 --2- 172.21.15.189:0/4083800868 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcd28003730 0x7fcd28003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:53.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd467fc700 1 --2- 172.21.15.189:0/4083800868 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcd280044e0 0x7fcd28004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:53.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd467fc700 1 --2- 172.21.15.189:0/4083800868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd28004e50 0x7fcd2800d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:53.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd467fc700 1 -- 172.21.15.189:0/4083800868 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd28004260 con 0x7fcd280044e0 2020-11-20T12:20:53.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd467fc700 1 -- 172.21.15.189:0/4083800868 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2800de30 con 0x7fcd28004e50 2020-11-20T12:20:53.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd467fc700 1 -- 172.21.15.189:0/4083800868 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2800df70 con 0x7fcd28003730 2020-11-20T12:20:53.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd47fff700 1 --2- 172.21.15.189:0/4083800868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd28004e50 0x7fcd2800d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:53.588 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd477fe700 1 --2- 172.21.15.189:0/4083800868 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcd28003730 0x7fcd28003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:53.588 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd46ffd700 1 --2- 172.21.15.189:0/4083800868 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcd280044e0 0x7fcd28004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:53.588 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd47fff700 1 -- 172.21.15.189:0/4083800868 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcd28003730 msgr2=0x7fcd28003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:53.588 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd47fff700 1 --2- 172.21.15.189:0/4083800868 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcd28003730 0x7fcd28003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:53.588 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd47fff700 1 -- 172.21.15.189:0/4083800868 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcd280044e0 msgr2=0x7fcd28004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:53.589 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd47fff700 1 --2- 172.21.15.189:0/4083800868 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcd280044e0 0x7fcd28004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:53.589 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd47fff700 1 -- 172.21.15.189:0/4083800868 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd28003240 con 0x7fcd28004e50 2020-11-20T12:20:53.589 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd47fff700 1 -- 172.21.15.189:0/4083800868 reap_dead start 2020-11-20T12:20:53.589 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.571+0000 7fcd47fff700 1 --2- 172.21.15.189:0/4083800868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd28004e50 0x7fcd2800d940 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0x7fcd40009ea0 tx=0x7fcd400091d0).ready entity=mon.1 client_cookie=1153b811d964bea1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:53.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd467fc700 1 -- 172.21.15.189:0/4083800868 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd400095c0 con 0x7fcd28004e50 2020-11-20T12:20:53.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd467fc700 1 -- 172.21.15.189:0/4083800868 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd4000ec40 con 0x7fcd28004e50 2020-11-20T12:20:53.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd467fc700 1 -- 172.21.15.189:0/4083800868 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd400123f0 con 0x7fcd28004e50 2020-11-20T12:20:53.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd4cec2700 1 -- 172.21.15.189:0/4083800868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd28004e50 msgr2=0x7fcd2800d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:53.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd4cec2700 1 --2- 172.21.15.189:0/4083800868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd28004e50 0x7fcd2800d940 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0x7fcd40009ea0 tx=0x7fcd400091d0).stop 2020-11-20T12:20:53.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd4cec2700 1 -- 172.21.15.189:0/4083800868 shutdown_connections 2020-11-20T12:20:53.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd4cec2700 1 --2- 172.21.15.189:0/4083800868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd28004e50 0x7fcd2800d940 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:53.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd4cec2700 1 -- 172.21.15.189:0/4083800868 >> 172.21.15.189:0/4083800868 conn(0x7fcd48065fe0 msgr2=0x7fcd48061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:53.592 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd4cec2700 1 -- 172.21.15.189:0/4083800868 shutdown_connections 2020-11-20T12:20:53.592 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd4cec2700 1 -- 172.21.15.189:0/4083800868 wait complete. 2020-11-20T12:20:53.592 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd4cec2700 1 Processor -- start 2020-11-20T12:20:53.592 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd4cec2700 1 -- start start 2020-11-20T12:20:53.592 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd4cec2700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcd28004e50 0x7fcd480a5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:53.593 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd4cec2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd48132dd0 0x7fcd48145c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:53.593 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd4cec2700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcd481460f0 0x7fcd4812fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:53.593 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd4cec2700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd48146570 con 0x7fcd28004e50 2020-11-20T12:20:53.593 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd4cec2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd480a66b0 con 0x7fcd48132dd0 2020-11-20T12:20:53.594 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd4cec2700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd48133250 con 0x7fcd481460f0 2020-11-20T12:20:53.594 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd46ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd48132dd0 0x7fcd48145c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:53.594 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd46ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd48132dd0 0x7fcd48145c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51832/0 (socket says 172.21.15.189:51832) 2020-11-20T12:20:53.595 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd46ffd700 1 -- 172.21.15.189:0/3437377827 learned_addr learned my addr 172.21.15.189:0/3437377827 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:20:53.595 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd47fff700 1 --2- 172.21.15.189:0/3437377827 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcd481460f0 0x7fcd4812fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:53.595 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd477fe700 1 --2- 172.21.15.189:0/3437377827 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcd28004e50 0x7fcd480a5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:53.596 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd46ffd700 1 -- 172.21.15.189:0/3437377827 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcd481460f0 msgr2=0x7fcd4812fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:53.596 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd46ffd700 1 --2- 172.21.15.189:0/3437377827 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcd481460f0 0x7fcd4812fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:53.596 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd46ffd700 1 -- 172.21.15.189:0/3437377827 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcd28004e50 msgr2=0x7fcd480a5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:53.596 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd46ffd700 1 --2- 172.21.15.189:0/3437377827 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcd28004e50 0x7fcd480a5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:53.597 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd46ffd700 1 -- 172.21.15.189:0/3437377827 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd480a6410 con 0x7fcd48132dd0 2020-11-20T12:20:53.597 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.575+0000 7fcd46ffd700 1 --2- 172.21.15.189:0/3437377827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd48132dd0 0x7fcd48145c00 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7fcd3000f8a0 tx=0x7fcd30007b90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:53.597 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.579+0000 7fcd3f7fe700 1 -- 172.21.15.189:0/3437377827 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd30011650 con 0x7fcd48132dd0 2020-11-20T12:20:53.598 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.579+0000 7fcd3f7fe700 1 -- 172.21.15.189:0/3437377827 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd300117b0 con 0x7fcd48132dd0 2020-11-20T12:20:53.598 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.579+0000 7fcd4cec2700 1 -- 172.21.15.189:0/3437377827 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd48145760 con 0x7fcd48132dd0 2020-11-20T12:20:53.598 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.579+0000 7fcd3f7fe700 1 -- 172.21.15.189:0/3437377827 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd30011b30 con 0x7fcd48132dd0 2020-11-20T12:20:53.598 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.579+0000 7fcd4cec2700 1 -- 172.21.15.189:0/3437377827 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd481334c0 con 0x7fcd48132dd0 2020-11-20T12:20:53.600 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.579+0000 7fcd4cec2700 1 -- 172.21.15.189:0/3437377827 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd480663c0 con 0x7fcd48132dd0 2020-11-20T12:20:53.600 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.579+0000 7fcd3f7fe700 1 -- 172.21.15.189:0/3437377827 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fcd3001b030 con 0x7fcd48132dd0 2020-11-20T12:20:53.601 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.579+0000 7fcd3f7fe700 1 --2- 172.21.15.189:0/3437377827 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcd240557c0 0x7fcd24057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:53.601 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.579+0000 7fcd3f7fe700 1 -- 172.21.15.189:0/3437377827 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7019+0+0 (secure 0 0 0) 0x7fcd3006e470 con 0x7fcd48132dd0 2020-11-20T12:20:53.602 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.579+0000 7fcd477fe700 1 --2- 172.21.15.189:0/3437377827 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcd240557c0 0x7fcd24057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:53.602 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.579+0000 7fcd477fe700 1 --2- 172.21.15.189:0/3437377827 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcd240557c0 0x7fcd24057c80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fcd380013b0 tx=0x7fcd480a9df0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:53.602 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.587+0000 7fcd3f7fe700 1 -- 172.21.15.189:0/3437377827 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcd30044010 con 0x7fcd48132dd0 2020-11-20T12:20:53.797 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.791+0000 7fcd4cec2700 1 -- 172.21.15.189:0/3437377827 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd480446d0 con 0x7fcd48132dd0 2020-11-20T12:20:53.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.791+0000 7fcd3f7fe700 1 -- 172.21.15.189:0/3437377827 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2137 (secure 0 0 0) 0x7fcd30019070 con 0x7fcd48132dd0 2020-11-20T12:20:53.798 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:20:53.798 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5746,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.189:6835/3538437301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3538437301},{"type":"v1","addr":"172.21.15.189:6835","nonce":3538437301}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:20:05.807690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5520":{"gid":5520,"name":"b","rank":0,"incarnation":57,"state":"up:active","state_seq":32,"addr":"172.21.15.47:6833/201662157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":201662157},{"type":"v1","addr":"172.21.15.47:6833","nonce":201662157}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:20:53.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.795+0000 7fcd4cec2700 1 -- 172.21.15.189:0/3437377827 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcd240557c0 msgr2=0x7fcd24057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:53.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.795+0000 7fcd4cec2700 1 --2- 172.21.15.189:0/3437377827 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcd240557c0 0x7fcd24057c80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fcd380013b0 tx=0x7fcd480a9df0).stop 2020-11-20T12:20:53.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.795+0000 7fcd4cec2700 1 -- 172.21.15.189:0/3437377827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd48132dd0 msgr2=0x7fcd48145c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:53.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.795+0000 7fcd4cec2700 1 --2- 172.21.15.189:0/3437377827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd48132dd0 0x7fcd48145c00 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7fcd3000f8a0 tx=0x7fcd30007b90).stop 2020-11-20T12:20:53.801 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.795+0000 7fcd4cec2700 1 -- 172.21.15.189:0/3437377827 shutdown_connections 2020-11-20T12:20:53.801 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.795+0000 7fcd4cec2700 1 --2- 172.21.15.189:0/3437377827 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcd240557c0 0x7fcd24057c80 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:53.801 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.795+0000 7fcd4cec2700 1 --2- 172.21.15.189:0/3437377827 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcd28004e50 0x7fcd480a5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:53.801 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.795+0000 7fcd4cec2700 1 --2- 172.21.15.189:0/3437377827 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcd481460f0 0x7fcd4812fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:53.801 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.795+0000 7fcd4cec2700 1 --2- 172.21.15.189:0/3437377827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd48132dd0 0x7fcd48145c00 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:53.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.795+0000 7fcd4cec2700 1 -- 172.21.15.189:0/3437377827 >> 172.21.15.189:0/3437377827 conn(0x7fcd48065fe0 msgr2=0x7fcd48133eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:53.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.795+0000 7fcd4cec2700 1 -- 172.21.15.189:0/3437377827 shutdown_connections 2020-11-20T12:20:53.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:53.795+0000 7fcd4cec2700 1 -- 172.21.15.189:0/3437377827 wait complete. 2020-11-20T12:20:53.803 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 61 2020-11-20T12:20:53.813 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:20:55.814 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:55.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.939+0000 7f669acf5700 1 Processor -- start 2020-11-20T12:20:55.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.939+0000 7f669acf5700 1 -- start start 2020-11-20T12:20:55.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f669acf5700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f66940623c0 0x7f66940677f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:55.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f669acf5700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f669413aac0 0x7f669413af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:55.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f669acf5700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f66941340e0 0x7f6694134570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:55.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f669acf5700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66940a8e60 con 0x7f66941340e0 2020-11-20T12:20:55.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f669acf5700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f669413b430 con 0x7f66940623c0 2020-11-20T12:20:55.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f669acf5700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66940a9c60 con 0x7f669413aac0 2020-11-20T12:20:55.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f6699cf3700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f66940623c0 0x7f66940677f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:55.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f6699cf3700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f66940623c0 0x7f66940677f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51836/0 (socket says 172.21.15.189:51836) 2020-11-20T12:20:55.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f6699cf3700 1 -- 172.21.15.189:0/2008199959 learned_addr learned my addr 172.21.15.189:0/2008199959 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:20:55.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f66994f2700 1 --2- 172.21.15.189:0/2008199959 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f669413aac0 0x7f669413af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:55.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f669a4f4700 1 --2- 172.21.15.189:0/2008199959 >> v2:172.21.15.47:3300/0 conn(0x7f66941340e0 0x7f6694134570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:55.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f6699cf3700 1 -- 172.21.15.189:0/2008199959 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f669413aac0 msgr2=0x7f669413af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:55.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f6699cf3700 1 --2- 172.21.15.189:0/2008199959 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f669413aac0 0x7f669413af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:55.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f6699cf3700 1 -- 172.21.15.189:0/2008199959 >> v2:172.21.15.47:3300/0 conn(0x7f66941340e0 msgr2=0x7f6694134570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:55.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f6699cf3700 1 --2- 172.21.15.189:0/2008199959 >> v2:172.21.15.47:3300/0 conn(0x7f66941340e0 0x7f6694134570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:55.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f6699cf3700 1 -- 172.21.15.189:0/2008199959 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6694067e90 con 0x7f66940623c0 2020-11-20T12:20:55.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f669a4f4700 1 --2- 172.21.15.189:0/2008199959 >> v2:172.21.15.47:3300/0 conn(0x7f66941340e0 0x7f6694134570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:20:55.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f6699cf3700 1 --2- 172.21.15.189:0/2008199959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66940623c0 0x7f66940677f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7f66940ae8f0 tx=0x7f6684006eb0).ready entity=mon.1 client_cookie=93b1f826c3701351 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:55.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f6698cf1700 1 -- 172.21.15.189:0/2008199959 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6684009810 con 0x7f66940623c0 2020-11-20T12:20:55.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f6698cf1700 1 -- 172.21.15.189:0/2008199959 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6684009970 con 0x7f66940623c0 2020-11-20T12:20:55.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f6698cf1700 1 -- 172.21.15.189:0/2008199959 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6684009cf0 con 0x7f66940623c0 2020-11-20T12:20:55.955 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f669acf5700 1 -- 172.21.15.189:0/2008199959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66940623c0 msgr2=0x7f66940677f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:55.955 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f669acf5700 1 --2- 172.21.15.189:0/2008199959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66940623c0 0x7f66940677f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7f66940ae8f0 tx=0x7f6684006eb0).stop 2020-11-20T12:20:55.955 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f669acf5700 1 -- 172.21.15.189:0/2008199959 shutdown_connections 2020-11-20T12:20:55.955 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f669acf5700 1 --2- 172.21.15.189:0/2008199959 >> v2:172.21.15.47:3300/0 conn(0x7f66941340e0 0x7f6694134570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:55.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f669acf5700 1 --2- 172.21.15.189:0/2008199959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66940623c0 0x7f66940677f0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:55.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f669acf5700 1 --2- 172.21.15.189:0/2008199959 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f669413aac0 0x7f669413af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:55.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f669acf5700 1 -- 172.21.15.189:0/2008199959 >> 172.21.15.189:0/2008199959 conn(0x7f6694065fe0 msgr2=0x7f6694061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:55.956 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f669acf5700 1 -- 172.21.15.189:0/2008199959 shutdown_connections 2020-11-20T12:20:55.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.943+0000 7f669acf5700 1 -- 172.21.15.189:0/2008199959 wait complete. 2020-11-20T12:20:55.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.947+0000 7f669acf5700 1 Processor -- start 2020-11-20T12:20:55.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.947+0000 7f669acf5700 1 -- start start 2020-11-20T12:20:55.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.947+0000 7f669acf5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66940623c0 0x7f66940ac880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:55.957 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.947+0000 7f669acf5700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66941340e0 0x7f669409c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:55.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.947+0000 7f669acf5700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f669413aac0 0x7f669409ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:55.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.947+0000 7f669acf5700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66941325c0 con 0x7f66941340e0 2020-11-20T12:20:55.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.947+0000 7f669acf5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66940ad220 con 0x7f66940623c0 2020-11-20T12:20:55.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.947+0000 7f669acf5700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6694132820 con 0x7f669413aac0 2020-11-20T12:20:55.958 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.947+0000 7f6699cf3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66940623c0 0x7f66940ac880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:55.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.947+0000 7f6699cf3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66940623c0 0x7f66940ac880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51842/0 (socket says 172.21.15.189:51842) 2020-11-20T12:20:55.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.947+0000 7f6699cf3700 1 -- 172.21.15.189:0/281445070 learned_addr learned my addr 172.21.15.189:0/281445070 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:20:55.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.947+0000 7f66994f2700 1 --2- 172.21.15.189:0/281445070 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66941340e0 0x7f669409c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:55.959 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.947+0000 7f669a4f4700 1 --2- 172.21.15.189:0/281445070 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f669413aac0 0x7f669409ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:55.961 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.951+0000 7f6699cf3700 1 -- 172.21.15.189:0/281445070 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f669413aac0 msgr2=0x7f669409ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:55.961 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.951+0000 7f6699cf3700 1 --2- 172.21.15.189:0/281445070 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f669413aac0 0x7f669409ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:55.962 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.951+0000 7f6699cf3700 1 -- 172.21.15.189:0/281445070 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66941340e0 msgr2=0x7f669409c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:55.962 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.951+0000 7f6699cf3700 1 --2- 172.21.15.189:0/281445070 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66941340e0 0x7f669409c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:55.962 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.951+0000 7f6699cf3700 1 -- 172.21.15.189:0/281445070 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6694132ac0 con 0x7f66940623c0 2020-11-20T12:20:55.962 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.951+0000 7f6699cf3700 1 --2- 172.21.15.189:0/281445070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66940623c0 0x7f66940ac880 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7f6684006fd0 tx=0x7f6684015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:55.962 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.951+0000 7f6681ffb700 1 -- 172.21.15.189:0/281445070 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6684011c80 con 0x7f66940623c0 2020-11-20T12:20:55.963 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.951+0000 7f6681ffb700 1 -- 172.21.15.189:0/281445070 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6684011de0 con 0x7f66940623c0 2020-11-20T12:20:55.963 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.951+0000 7f669acf5700 1 -- 172.21.15.189:0/281445070 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6694130dd0 con 0x7f66940623c0 2020-11-20T12:20:55.963 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.951+0000 7f6681ffb700 1 -- 172.21.15.189:0/281445070 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668401e670 con 0x7f66940623c0 2020-11-20T12:20:55.963 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.951+0000 7f669acf5700 1 -- 172.21.15.189:0/281445070 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66941312a0 con 0x7f66940623c0 2020-11-20T12:20:55.963 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.951+0000 7f669acf5700 1 -- 172.21.15.189:0/281445070 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66940663c0 con 0x7f66940623c0 2020-11-20T12:20:55.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.951+0000 7f6681ffb700 1 -- 172.21.15.189:0/281445070 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f6684025030 con 0x7f66940623c0 2020-11-20T12:20:55.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.951+0000 7f6681ffb700 1 --2- 172.21.15.189:0/281445070 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f666c0557c0 0x7f666c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:55.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.951+0000 7f6681ffb700 1 -- 172.21.15.189:0/281445070 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7019+0+0 (secure 0 0 0) 0x7f668406fed0 con 0x7f66940623c0 2020-11-20T12:20:55.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.951+0000 7f66994f2700 1 --2- 172.21.15.189:0/281445070 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f666c0557c0 0x7f666c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:55.964 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.955+0000 7f66994f2700 1 --2- 172.21.15.189:0/281445070 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f666c0557c0 0x7f666c057c80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f66940a4cf0 tx=0x7f6690006df0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:55.965 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:55.959+0000 7f6681ffb700 1 -- 172.21.15.189:0/281445070 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6684045a70 con 0x7f66940623c0 2020-11-20T12:20:56.171 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:56.163+0000 7f669acf5700 1 -- 172.21.15.189:0/281445070 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66940446d0 con 0x7f66940623c0 2020-11-20T12:20:56.171 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:56.163+0000 7f6681ffb700 1 -- 172.21.15.189:0/281445070 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f6684023030 con 0x7f66940623c0 2020-11-20T12:20:56.171 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:20:56.172 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5746,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.189:6835/3538437301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3538437301},{"type":"v1","addr":"172.21.15.189:6835","nonce":3538437301}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":61}],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:20:05.807690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5520},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5520":{"gid":5520,"name":"b","rank":0,"incarnation":57,"state":"up:active","state_seq":32,"addr":"172.21.15.47:6833/201662157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":201662157},{"type":"v1","addr":"172.21.15.47:6833","nonce":201662157}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:20:56.173 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:20:56.173 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:56.167+0000 7f669acf5700 1 -- 172.21.15.189:0/281445070 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f666c0557c0 msgr2=0x7f666c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:56.174 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:56.167+0000 7f669acf5700 1 --2- 172.21.15.189:0/281445070 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f666c0557c0 0x7f666c057c80 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f66940a4cf0 tx=0x7f6690006df0).stop 2020-11-20T12:20:56.174 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:56.167+0000 7f669acf5700 1 -- 172.21.15.189:0/281445070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66940623c0 msgr2=0x7f66940ac880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:56.174 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:56.167+0000 7f669acf5700 1 --2- 172.21.15.189:0/281445070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66940623c0 0x7f66940ac880 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7f6684006fd0 tx=0x7f6684015000).stop 2020-11-20T12:20:56.174 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:56.167+0000 7f669acf5700 1 -- 172.21.15.189:0/281445070 shutdown_connections 2020-11-20T12:20:56.175 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:56.167+0000 7f669acf5700 1 --2- 172.21.15.189:0/281445070 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f669413aac0 0x7f669409ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:56.175 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:56.167+0000 7f669acf5700 1 --2- 172.21.15.189:0/281445070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66940623c0 0x7f66940ac880 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:56.175 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:56.167+0000 7f669acf5700 1 --2- 172.21.15.189:0/281445070 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f666c0557c0 0x7f666c057c80 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:56.175 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:56.167+0000 7f669acf5700 1 --2- 172.21.15.189:0/281445070 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f66941340e0 0x7f669409c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:56.175 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:56.167+0000 7f669acf5700 1 -- 172.21.15.189:0/281445070 >> 172.21.15.189:0/281445070 conn(0x7f6694065fe0 msgr2=0x7f6694060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:56.176 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:56.167+0000 7f669acf5700 1 -- 172.21.15.189:0/281445070 shutdown_connections 2020-11-20T12:20:56.176 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:56.167+0000 7f669acf5700 1 -- 172.21.15.189:0/281445070 wait complete. 2020-11-20T12:20:56.177 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 61 2020-11-20T12:20:56.178 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:20:56.186 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:20:58.187 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:58.300 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe076d03700 1 Processor -- start 2020-11-20T12:20:58.301 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe076d03700 1 -- start start 2020-11-20T12:20:58.302 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe076d03700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe0700a85a0 0x7fe0700a8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:58.303 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe076d03700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fe0701339a0 0x7fe070133e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:58.303 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe076d03700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0700a0f60 con 0x7fe0700612b0 2020-11-20T12:20:58.303 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe076d03700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe070068440 con 0x7fe0701339a0 2020-11-20T12:20:58.303 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe076d03700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe070067d00 con 0x7fe0700a85a0 2020-11-20T12:20:58.303 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe06ffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe0700a85a0 0x7fe0700a8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:58.304 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe06ffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe0700a85a0 0x7fe0700a8a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51862/0 (socket says 172.21.15.189:51862) 2020-11-20T12:20:58.304 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe06ffff700 1 -- 172.21.15.189:0/3584169369 learned_addr learned my addr 172.21.15.189:0/3584169369 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:20:58.304 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe0752a0700 1 --2- 172.21.15.189:0/3584169369 >> v2:172.21.15.47:3300/0 conn(0x7fe0701339a0 0x7fe070133e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:58.304 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe06ffff700 1 -- 172.21.15.189:0/3584169369 >> v2:172.21.15.47:3300/0 conn(0x7fe0701339a0 msgr2=0x7fe070133e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:58.304 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe06ffff700 1 --2- 172.21.15.189:0/3584169369 >> v2:172.21.15.47:3300/0 conn(0x7fe0701339a0 0x7fe070133e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:58.305 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe06ffff700 1 -- 172.21.15.189:0/3584169369 >> v1:172.21.15.47:6789/0 conn(0x7fe0700612b0 legacy=0x7fe0700a0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:58.305 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe06ffff700 1 -- 172.21.15.189:0/3584169369 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe070067fa0 con 0x7fe0700a85a0 2020-11-20T12:20:58.305 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe06ffff700 1 --2- 172.21.15.189:0/3584169369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0700a85a0 0x7fe0700a8a50 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7fe058006fd0 tx=0x7fe058006eb0).ready entity=mon.1 client_cookie=a9340c4f8b5af83b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:58.305 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe06f7fe700 1 -- 172.21.15.189:0/3584169369 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe058009850 con 0x7fe0700a85a0 2020-11-20T12:20:58.306 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe06f7fe700 1 -- 172.21.15.189:0/3584169369 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0580099b0 con 0x7fe0700a85a0 2020-11-20T12:20:58.306 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe06f7fe700 1 -- 172.21.15.189:0/3584169369 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe058009d30 con 0x7fe0700a85a0 2020-11-20T12:20:58.309 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe076d03700 1 -- 172.21.15.189:0/3584169369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0700a85a0 msgr2=0x7fe0700a8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:58.309 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe076d03700 1 --2- 172.21.15.189:0/3584169369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0700a85a0 0x7fe0700a8a50 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7fe058006fd0 tx=0x7fe058006eb0).stop 2020-11-20T12:20:58.309 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe076d03700 1 -- 172.21.15.189:0/3584169369 shutdown_connections 2020-11-20T12:20:58.309 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe076d03700 1 --2- 172.21.15.189:0/3584169369 >> v2:172.21.15.47:3300/0 conn(0x7fe0701339a0 0x7fe070133e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:58.310 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe076d03700 1 --2- 172.21.15.189:0/3584169369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0700a85a0 0x7fe0700a8a50 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:58.310 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe076d03700 1 -- 172.21.15.189:0/3584169369 >> 172.21.15.189:0/3584169369 conn(0x7fe070065910 msgr2=0x7fe070060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:58.310 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe076d03700 1 -- 172.21.15.189:0/3584169369 shutdown_connections 2020-11-20T12:20:58.310 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.295+0000 7fe076d03700 1 -- 172.21.15.189:0/3584169369 wait complete. 2020-11-20T12:20:58.310 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.299+0000 7fe076d03700 1 Processor -- start 2020-11-20T12:20:58.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.299+0000 7fe076d03700 1 -- start start 2020-11-20T12:20:58.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.299+0000 7fe076d03700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe0700612b0 0x7fe07013a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:58.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.299+0000 7fe076d03700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0700a85a0 0x7fe070136990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:58.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.299+0000 7fe076d03700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0701339a0 0x7fe0700a4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:58.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.299+0000 7fe076d03700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0700a65d0 con 0x7fe0701339a0 2020-11-20T12:20:58.312 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.299+0000 7fe076d03700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0700a6870 con 0x7fe0700a85a0 2020-11-20T12:20:58.312 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.299+0000 7fe076d03700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0701364f0 con 0x7fe0700612b0 2020-11-20T12:20:58.312 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.299+0000 7fe06ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0700a85a0 0x7fe070136990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:58.312 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.299+0000 7fe06ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0700a85a0 0x7fe070136990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51868/0 (socket says 172.21.15.189:51868) 2020-11-20T12:20:58.312 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.299+0000 7fe06ffff700 1 -- 172.21.15.189:0/3860209385 learned_addr learned my addr 172.21.15.189:0/3860209385 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:20:58.313 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.299+0000 7fe0752a0700 1 --2- 172.21.15.189:0/3860209385 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0701339a0 0x7fe0700a4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:58.313 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.299+0000 7fe074a9f700 1 --2- 172.21.15.189:0/3860209385 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe0700612b0 0x7fe07013a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:58.313 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.299+0000 7fe06ffff700 1 -- 172.21.15.189:0/3860209385 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe0700612b0 msgr2=0x7fe07013a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:58.313 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.299+0000 7fe06ffff700 1 --2- 172.21.15.189:0/3860209385 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe0700612b0 0x7fe07013a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:58.313 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.299+0000 7fe06ffff700 1 -- 172.21.15.189:0/3860209385 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0701339a0 msgr2=0x7fe0700a4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:58.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.299+0000 7fe06ffff700 1 --2- 172.21.15.189:0/3860209385 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0701339a0 0x7fe0700a4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:58.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.299+0000 7fe06ffff700 1 -- 172.21.15.189:0/3860209385 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe070136e80 con 0x7fe0700a85a0 2020-11-20T12:20:58.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.299+0000 7fe074a9f700 1 --2- 172.21.15.189:0/3860209385 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe0700612b0 0x7fe07013a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:58.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.299+0000 7fe0752a0700 1 --2- 172.21.15.189:0/3860209385 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0701339a0 0x7fe0700a4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:20:58.315 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.299+0000 7fe06ffff700 1 --2- 172.21.15.189:0/3860209385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0700a85a0 0x7fe070136990 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7fe05800b1b0 tx=0x7fe058015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:58.315 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.299+0000 7fe06cff9700 1 -- 172.21.15.189:0/3860209385 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05801e460 con 0x7fe0700a85a0 2020-11-20T12:20:58.315 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.299+0000 7fe06cff9700 1 -- 172.21.15.189:0/3860209385 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05801e5c0 con 0x7fe0700a85a0 2020-11-20T12:20:58.315 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.299+0000 7fe076d03700 1 -- 172.21.15.189:0/3860209385 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0700a4c20 con 0x7fe0700a85a0 2020-11-20T12:20:58.315 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.299+0000 7fe06cff9700 1 -- 172.21.15.189:0/3860209385 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05801e960 con 0x7fe0700a85a0 2020-11-20T12:20:58.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.299+0000 7fe076d03700 1 -- 172.21.15.189:0/3860209385 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0700a62d0 con 0x7fe0700a85a0 2020-11-20T12:20:58.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.303+0000 7fe06cff9700 1 -- 172.21.15.189:0/3860209385 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fe058025030 con 0x7fe0700a85a0 2020-11-20T12:20:58.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.303+0000 7fe06cff9700 1 --2- 172.21.15.189:0/3860209385 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe04c0557c0 0x7fe04c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:58.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.303+0000 7fe076d03700 1 -- 172.21.15.189:0/3860209385 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe070066c60 con 0x7fe0700a85a0 2020-11-20T12:20:58.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.303+0000 7fe06cff9700 1 -- 172.21.15.189:0/3860209385 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 7105+0+0 (secure 0 0 0) 0x7fe058070940 con 0x7fe0700a85a0 2020-11-20T12:20:58.317 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.303+0000 7fe074a9f700 1 --2- 172.21.15.189:0/3860209385 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe04c0557c0 0x7fe04c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:58.317 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.303+0000 7fe074a9f700 1 --2- 172.21.15.189:0/3860209385 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe04c0557c0 0x7fe04c057c80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fe0680079d0 tx=0x7fe068008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:58.317 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.307+0000 7fe06cff9700 1 -- 172.21.15.189:0/3860209385 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe070066c60 con 0x7fe0700a85a0 2020-11-20T12:20:58.500 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.491+0000 7fe076d03700 1 -- 172.21.15.189:0/3860209385 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe070066c60 con 0x7fe0700a85a0 2020-11-20T12:20:58.501 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.491+0000 7fe06cff9700 1 -- 172.21.15.189:0/3860209385 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+1787 (secure 0 0 0) 0x7fe070066c60 con 0x7fe0700a85a0 2020-11-20T12:20:58.501 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:20:58.501 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:20:57.801034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5746":{"gid":5746,"name":"a","rank":0,"incarnation":62,"state":"up:replay","state_seq":2,"addr":"172.21.15.189:6835/3538437301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3538437301},{"type":"v1","addr":"172.21.15.189:6835","nonce":3538437301}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:20:58.502 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.495+0000 7fe076d03700 1 -- 172.21.15.189:0/3860209385 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe04c0557c0 msgr2=0x7fe04c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:58.502 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.495+0000 7fe076d03700 1 --2- 172.21.15.189:0/3860209385 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe04c0557c0 0x7fe04c057c80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fe0680079d0 tx=0x7fe068008040).stop 2020-11-20T12:20:58.503 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.495+0000 7fe076d03700 1 -- 172.21.15.189:0/3860209385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0700a85a0 msgr2=0x7fe070136990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:58.503 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.495+0000 7fe076d03700 1 --2- 172.21.15.189:0/3860209385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0700a85a0 0x7fe070136990 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7fe05800b1b0 tx=0x7fe058015000).stop 2020-11-20T12:20:58.503 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.495+0000 7fe076d03700 1 -- 172.21.15.189:0/3860209385 shutdown_connections 2020-11-20T12:20:58.503 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.495+0000 7fe076d03700 1 --2- 172.21.15.189:0/3860209385 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fe04c0557c0 0x7fe04c057c80 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:58.503 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.495+0000 7fe076d03700 1 --2- 172.21.15.189:0/3860209385 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fe0701339a0 0x7fe0700a4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:58.504 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.495+0000 7fe076d03700 1 --2- 172.21.15.189:0/3860209385 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fe0700612b0 0x7fe07013a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:58.504 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.495+0000 7fe076d03700 1 --2- 172.21.15.189:0/3860209385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0700a85a0 0x7fe070136990 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:58.504 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.495+0000 7fe076d03700 1 -- 172.21.15.189:0/3860209385 >> 172.21.15.189:0/3860209385 conn(0x7fe070065910 msgr2=0x7fe0700aa030 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:58.504 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.495+0000 7fe076d03700 1 -- 172.21.15.189:0/3860209385 shutdown_connections 2020-11-20T12:20:58.504 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.495+0000 7fe076d03700 1 -- 172.21.15.189:0/3860209385 wait complete. 2020-11-20T12:20:58.505 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 62 2020-11-20T12:20:58.514 INFO:tasks.mds_thrash.fs.[None]:mds.b down, removed from mdsmap 2020-11-20T12:20:58.514 INFO:tasks.mds_thrash.fs.[None]:waiting for mds cluster to stabilize... 2020-11-20T12:20:58.515 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:20:58.652 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.647+0000 7f88a0a4b700 1 Processor -- start 2020-11-20T12:20:58.687 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.647+0000 7f88a0a4b700 1 -- start start 2020-11-20T12:20:58.691 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.647+0000 7f88a0a4b700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f889c13a680 0x7f889c13ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:58.691 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.647+0000 7f88a0a4b700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f889c13b000 0x7f889c1451c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:58.691 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.647+0000 7f88a0a4b700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f889c13e320 con 0x7f889c0a8840 2020-11-20T12:20:58.692 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.647+0000 7f88a0a4b700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f889c066800 con 0x7f889c13a680 2020-11-20T12:20:58.692 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.647+0000 7f88a0a4b700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f889c066aa0 con 0x7f889c13b000 2020-11-20T12:20:58.692 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.647+0000 7f889b59e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f889c13b000 0x7f889c1451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:58.692 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.647+0000 7f889a59c700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f889c13a680 0x7f889c13ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:58.692 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.647+0000 7f889ad9d700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f889c0a8840 0x7f889c13dc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.189:48160/0 (socket says 172.21.15.189:48160) 2020-11-20T12:20:58.693 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.647+0000 7f889a59c700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f889c13a680 0x7f889c13ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.189:58100/0 (socket says 172.21.15.189:58100) 2020-11-20T12:20:58.693 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.647+0000 7f889b59e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f889c13b000 0x7f889c1451c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51878/0 (socket says 172.21.15.189:51878) 2020-11-20T12:20:58.693 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.647+0000 7f889ad9d700 1 -- 172.21.15.189:0/1597750475 learned_addr learned my addr 172.21.15.189:0/1597750475 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:20:58.693 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.647+0000 7f889b59e700 1 -- 172.21.15.189:0/1597750475 >> v2:172.21.15.47:3300/0 conn(0x7f889c13a680 msgr2=0x7f889c13ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:20:58.693 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.647+0000 7f889b59e700 1 --2- 172.21.15.189:0/1597750475 >> v2:172.21.15.47:3300/0 conn(0x7f889c13a680 0x7f889c13ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:58.694 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.647+0000 7f889b59e700 1 -- 172.21.15.189:0/1597750475 >> v1:172.21.15.47:6789/0 conn(0x7f889c0a8840 legacy=0x7f889c13dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:58.694 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.647+0000 7f889b59e700 1 -- 172.21.15.189:0/1597750475 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889c0672f0 con 0x7f889c13b000 2020-11-20T12:20:58.694 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.647+0000 7f889b59e700 1 --2- 172.21.15.189:0/1597750475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f889c13b000 0x7f889c1451c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7f889400b9a0 tx=0x7f8894006df0).ready entity=mon.1 client_cookie=f849e52695a2c172 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:58.694 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.647+0000 7f8899d9b700 1 -- 172.21.15.189:0/1597750475 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8894009770 con 0x7f889c13b000 2020-11-20T12:20:58.694 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.647+0000 7f8899d9b700 1 -- 172.21.15.189:0/1597750475 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88940098d0 con 0x7f889c13b000 2020-11-20T12:20:58.695 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.647+0000 7f8899d9b700 1 -- 172.21.15.189:0/1597750475 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8894009c50 con 0x7f889c13b000 2020-11-20T12:20:58.695 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.651+0000 7f88a0a4b700 1 -- 172.21.15.189:0/1597750475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f889c13b000 msgr2=0x7f889c1451c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:58.695 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.651+0000 7f88a0a4b700 1 --2- 172.21.15.189:0/1597750475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f889c13b000 0x7f889c1451c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7f889400b9a0 tx=0x7f8894006df0).stop 2020-11-20T12:20:58.695 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.651+0000 7f88a0a4b700 1 -- 172.21.15.189:0/1597750475 shutdown_connections 2020-11-20T12:20:58.695 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.651+0000 7f88a0a4b700 1 --2- 172.21.15.189:0/1597750475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f889c13b000 0x7f889c1451c0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:58.696 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.651+0000 7f88a0a4b700 1 --2- 172.21.15.189:0/1597750475 >> v2:172.21.15.47:3300/0 conn(0x7f889c13a680 0x7f889c13ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:58.696 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.651+0000 7f88a0a4b700 1 -- 172.21.15.189:0/1597750475 >> 172.21.15.189:0/1597750475 conn(0x7f889c060170 msgr2=0x7f889c060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:58.696 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.651+0000 7f88a0a4b700 1 -- 172.21.15.189:0/1597750475 shutdown_connections 2020-11-20T12:20:58.696 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.651+0000 7f88a0a4b700 1 -- 172.21.15.189:0/1597750475 wait complete. 2020-11-20T12:20:58.696 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.651+0000 7f88a0a4b700 1 Processor -- start 2020-11-20T12:20:58.697 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.651+0000 7f88a0a4b700 1 -- start start 2020-11-20T12:20:58.697 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.655+0000 7f88a0a4b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f889c0a8840 0x7f889c135cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:58.697 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.655+0000 7f88a0a4b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f889c13a680 0x7f889c0aac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:58.697 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.655+0000 7f88a0a4b700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f889c13b000 0x7f889c14d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:58.697 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.655+0000 7f88a0a4b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889c138290 con 0x7f889c0a8840 2020-11-20T12:20:58.698 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.655+0000 7f88a0a4b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889c13db50 con 0x7f889c13a680 2020-11-20T12:20:58.698 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.655+0000 7f88a0a4b700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f889c0a3af0 con 0x7f889c13b000 2020-11-20T12:20:58.698 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.655+0000 7f889a59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f889c13a680 0x7f889c0aac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:58.698 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.655+0000 7f889a59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f889c13a680 0x7f889c0aac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51882/0 (socket says 172.21.15.189:51882) 2020-11-20T12:20:58.698 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.655+0000 7f889a59c700 1 -- 172.21.15.189:0/2540100393 learned_addr learned my addr 172.21.15.189:0/2540100393 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:20:58.699 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.655+0000 7f889ad9d700 1 --2- 172.21.15.189:0/2540100393 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f889c0a8840 0x7f889c135cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:58.699 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.655+0000 7f889b59e700 1 --2- 172.21.15.189:0/2540100393 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f889c13b000 0x7f889c14d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:58.699 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.655+0000 7f889a59c700 1 -- 172.21.15.189:0/2540100393 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f889c13b000 msgr2=0x7f889c14d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:58.699 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.655+0000 7f889a59c700 1 --2- 172.21.15.189:0/2540100393 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f889c13b000 0x7f889c14d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:58.699 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.655+0000 7f889a59c700 1 -- 172.21.15.189:0/2540100393 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f889c0a8840 msgr2=0x7f889c135cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:58.700 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.655+0000 7f889a59c700 1 --2- 172.21.15.189:0/2540100393 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f889c0a8840 0x7f889c135cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:58.700 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.655+0000 7f889a59c700 1 -- 172.21.15.189:0/2540100393 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889c13bb50 con 0x7f889c13a680 2020-11-20T12:20:58.700 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.655+0000 7f889a59c700 1 --2- 172.21.15.189:0/2540100393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f889c13a680 0x7f889c0aac70 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f889000eba0 tx=0x7f8890006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:58.700 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.655+0000 7f8882ffd700 1 -- 172.21.15.189:0/2540100393 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889000c8e0 con 0x7f889c13a680 2020-11-20T12:20:58.700 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.655+0000 7f8882ffd700 1 -- 172.21.15.189:0/2540100393 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f889000ca40 con 0x7f889c13a680 2020-11-20T12:20:58.701 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.655+0000 7f88a0a4b700 1 -- 172.21.15.189:0/2540100393 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f889c135000 con 0x7f889c13a680 2020-11-20T12:20:58.701 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.655+0000 7f8882ffd700 1 -- 172.21.15.189:0/2540100393 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889000cdc0 con 0x7f889c13a680 2020-11-20T12:20:58.701 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.655+0000 7f88a0a4b700 1 -- 172.21.15.189:0/2540100393 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f889c0ab160 con 0x7f889c13a680 2020-11-20T12:20:58.701 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.655+0000 7f88a0a4b700 1 -- 172.21.15.189:0/2540100393 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f889c0ad820 con 0x7f889c13a680 2020-11-20T12:20:58.701 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.659+0000 7f8882ffd700 1 -- 172.21.15.189:0/2540100393 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f8890012600 con 0x7f889c13a680 2020-11-20T12:20:58.702 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.659+0000 7f8882ffd700 1 --2- 172.21.15.189:0/2540100393 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f88780557c0 0x7f8878057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:20:58.702 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.659+0000 7f889ad9d700 1 --2- 172.21.15.189:0/2540100393 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f88780557c0 0x7f8878057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:20:58.702 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.659+0000 7f8882ffd700 1 -- 172.21.15.189:0/2540100393 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 7105+0+0 (secure 0 0 0) 0x7f889006ee60 con 0x7f889c13a680 2020-11-20T12:20:58.702 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.659+0000 7f889ad9d700 1 --2- 172.21.15.189:0/2540100393 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f88780557c0 0x7f8878057c80 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f888c000f40 tx=0x7f888c009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:20:58.702 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.667+0000 7f8882ffd700 1 -- 172.21.15.189:0/2540100393 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f88900448f0 con 0x7f889c13a680 2020-11-20T12:20:58.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.855+0000 7f88a0a4b700 1 -- 172.21.15.189:0/2540100393 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f889c0a9230 con 0x7f889c13a680 2020-11-20T12:20:58.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.855+0000 7f8882ffd700 1 -- 172.21.15.189:0/2540100393 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+1791 (secure 0 0 0) 0x7f8890047f10 con 0x7f889c13a680 2020-11-20T12:20:58.863 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:20:58.864 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:20:58.811448+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5746":{"gid":5746,"name":"a","rank":0,"incarnation":62,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.189:6835/3538437301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3538437301},{"type":"v1","addr":"172.21.15.189:6835","nonce":3538437301}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:20:58.866 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.859+0000 7f88a0a4b700 1 -- 172.21.15.189:0/2540100393 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f88780557c0 msgr2=0x7f8878057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:58.866 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.859+0000 7f88a0a4b700 1 --2- 172.21.15.189:0/2540100393 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f88780557c0 0x7f8878057c80 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f888c000f40 tx=0x7f888c009040).stop 2020-11-20T12:20:58.866 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.859+0000 7f88a0a4b700 1 -- 172.21.15.189:0/2540100393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f889c13a680 msgr2=0x7f889c0aac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:20:58.866 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.859+0000 7f88a0a4b700 1 --2- 172.21.15.189:0/2540100393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f889c13a680 0x7f889c0aac70 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f889000eba0 tx=0x7f8890006df0).stop 2020-11-20T12:20:58.867 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.859+0000 7f88a0a4b700 1 -- 172.21.15.189:0/2540100393 shutdown_connections 2020-11-20T12:20:58.867 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.859+0000 7f88a0a4b700 1 --2- 172.21.15.189:0/2540100393 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f88780557c0 0x7f8878057c80 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:58.867 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.859+0000 7f88a0a4b700 1 --2- 172.21.15.189:0/2540100393 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f889c0a8840 0x7f889c135cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:58.867 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.859+0000 7f88a0a4b700 1 --2- 172.21.15.189:0/2540100393 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f889c13b000 0x7f889c14d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:58.867 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.859+0000 7f88a0a4b700 1 --2- 172.21.15.189:0/2540100393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f889c13a680 0x7f889c0aac70 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:20:58.868 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.859+0000 7f88a0a4b700 1 -- 172.21.15.189:0/2540100393 >> 172.21.15.189:0/2540100393 conn(0x7f889c060170 msgr2=0x7f889c0a70e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:20:58.868 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.859+0000 7f88a0a4b700 1 -- 172.21.15.189:0/2540100393 shutdown_connections 2020-11-20T12:20:58.868 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:20:58.859+0000 7f88a0a4b700 1 -- 172.21.15.189:0/2540100393 wait complete. 2020-11-20T12:20:58.868 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 63 2020-11-20T12:20:58.881 INFO:tasks.mds_thrash.fs.[None]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 63, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2020-11-20T12:04:54.012110+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 63, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5746": { "addr": "172.21.15.189:6835/3538437301", "addrs": { "addrvec": [ { "addr": "172.21.15.189:6834", "nonce": 3538437301, "type": "v2" }, { "addr": "172.21.15.189:6835", "nonce": 3538437301, "type": "v1" } ] }, "export_targets": [], "features": 4540138297136906239, "flags": 0, "gid": 5746, "incarnation": 62, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2020-11-20T12:20:58.811448+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5746 } } } ], "standbys": [] } 2020-11-20T12:21:00.883 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:01.014 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.007+0000 7fad44b35700 1 Processor -- start 2020-11-20T12:21:01.015 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.007+0000 7fad44b35700 1 -- start start 2020-11-20T12:21:01.017 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.011+0000 7fad44b35700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fad40060170 0x7fad40137110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:01.017 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.011+0000 7fad44b35700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fad400a3c60 0x7fad40137600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:01.018 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.011+0000 7fad44b35700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad400aa920 con 0x7fad40060550 2020-11-20T12:21:01.018 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.011+0000 7fad44b35700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad40066a10 con 0x7fad40060170 2020-11-20T12:21:01.018 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.011+0000 7fad44b35700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad400a7350 con 0x7fad400a3c60 2020-11-20T12:21:01.018 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.011+0000 7fad3ffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fad400a3c60 0x7fad40137600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:01.018 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.011+0000 7fad3effd700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fad40060550 0x7fad400a3550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53516/0 (socket says 172.21.15.189:53516) 2020-11-20T12:21:01.019 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.011+0000 7fad3effd700 1 -- 172.21.15.189:0/1632750240 learned_addr learned my addr 172.21.15.189:0/1632750240 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:21:01.019 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.011+0000 7fad3f7fe700 1 --2- 172.21.15.189:0/1632750240 >> v2:172.21.15.47:3300/0 conn(0x7fad40060170 0x7fad40137110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:01.021 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.011+0000 7fad3e7fc700 1 -- 172.21.15.189:0/1632750240 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1964438537 0 0) 0x7fad400aa920 con 0x7fad40060550 2020-11-20T12:21:01.021 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.011+0000 7fad3e7fc700 1 -- 172.21.15.189:0/1632750240 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad2c003240 con 0x7fad40060550 2020-11-20T12:21:01.021 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.011+0000 7fad3f7fe700 1 -- 172.21.15.189:0/1632750240 >> v2:172.21.15.189:3300/0 conn(0x7fad400a3c60 msgr2=0x7fad40137600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:01.021 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.011+0000 7fad3f7fe700 1 --2- 172.21.15.189:0/1632750240 >> v2:172.21.15.189:3300/0 conn(0x7fad400a3c60 0x7fad40137600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:01.022 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.011+0000 7fad3f7fe700 1 -- 172.21.15.189:0/1632750240 >> v1:172.21.15.189:6789/0 conn(0x7fad40060550 legacy=0x7fad400a3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:01.022 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.011+0000 7fad3f7fe700 1 -- 172.21.15.189:0/1632750240 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad400662d0 con 0x7fad40060170 2020-11-20T12:21:01.022 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.011+0000 7fad3ffff700 1 --2- 172.21.15.189:0/1632750240 >> v2:172.21.15.189:3300/0 conn(0x7fad400a3c60 0x7fad40137600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:21:01.022 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.011+0000 7fad3f7fe700 1 --2- 172.21.15.189:0/1632750240 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad40060170 0x7fad40137110 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fad40066e90 tx=0x7fad30006df0).ready entity=mon.0 client_cookie=7617e72d5e36dd08 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:01.022 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.011+0000 7fad3e7fc700 1 -- 172.21.15.189:0/1632750240 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad30009640 con 0x7fad40060170 2020-11-20T12:21:01.023 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.011+0000 7fad3e7fc700 1 -- 172.21.15.189:0/1632750240 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad300097a0 con 0x7fad40060170 2020-11-20T12:21:01.023 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.011+0000 7fad3e7fc700 1 -- 172.21.15.189:0/1632750240 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad30009b20 con 0x7fad40060170 2020-11-20T12:21:01.023 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad44b35700 1 -- 172.21.15.189:0/1632750240 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad40060170 msgr2=0x7fad40137110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:01.023 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad44b35700 1 --2- 172.21.15.189:0/1632750240 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad40060170 0x7fad40137110 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fad40066e90 tx=0x7fad30006df0).stop 2020-11-20T12:21:01.023 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad44b35700 1 -- 172.21.15.189:0/1632750240 shutdown_connections 2020-11-20T12:21:01.024 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad44b35700 1 --2- 172.21.15.189:0/1632750240 >> v2:172.21.15.189:3300/0 conn(0x7fad400a3c60 0x7fad40137600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:01.024 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad44b35700 1 --2- 172.21.15.189:0/1632750240 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad40060170 0x7fad40137110 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:01.024 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad44b35700 1 -- 172.21.15.189:0/1632750240 >> 172.21.15.189:0/1632750240 conn(0x7fad40062620 msgr2=0x7fad40060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:01.024 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad44b35700 1 -- 172.21.15.189:0/1632750240 shutdown_connections 2020-11-20T12:21:01.024 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad44b35700 1 -- 172.21.15.189:0/1632750240 wait complete. 2020-11-20T12:21:01.025 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad44b35700 1 Processor -- start 2020-11-20T12:21:01.025 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad44b35700 1 -- start start 2020-11-20T12:21:01.027 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad44b35700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad40060170 0x7fad4013aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:01.027 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad44b35700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad40060550 0x7fad40146bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:01.028 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad44b35700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fad4009cdb0 0x7fad4009e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:01.028 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad44b35700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad40146480 con 0x7fad40060170 2020-11-20T12:21:01.028 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad44b35700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad40146720 con 0x7fad40060550 2020-11-20T12:21:01.028 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad44b35700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad401470b0 con 0x7fad4009cdb0 2020-11-20T12:21:01.028 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad3effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad40060550 0x7fad40146bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:01.029 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad3effd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad40060550 0x7fad40146bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51900/0 (socket says 172.21.15.189:51900) 2020-11-20T12:21:01.029 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad3effd700 1 -- 172.21.15.189:0/3539068485 learned_addr learned my addr 172.21.15.189:0/3539068485 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:21:01.029 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad3f7fe700 1 --2- 172.21.15.189:0/3539068485 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad40060170 0x7fad4013aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:01.029 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad3ffff700 1 --2- 172.21.15.189:0/3539068485 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fad4009cdb0 0x7fad4009e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:01.029 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad3effd700 1 -- 172.21.15.189:0/3539068485 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fad4009cdb0 msgr2=0x7fad4009e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:01.030 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad3effd700 1 --2- 172.21.15.189:0/3539068485 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fad4009cdb0 0x7fad4009e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:01.030 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad3effd700 1 -- 172.21.15.189:0/3539068485 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad40060170 msgr2=0x7fad4013aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:01.030 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad3effd700 1 --2- 172.21.15.189:0/3539068485 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad40060170 0x7fad4013aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:01.030 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad3effd700 1 -- 172.21.15.189:0/3539068485 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4009e900 con 0x7fad40060550 2020-11-20T12:21:01.030 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad3effd700 1 --2- 172.21.15.189:0/3539068485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad40060550 0x7fad40146bc0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7fad3400c6a0 tx=0x7fad34007ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:01.031 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad1f7fe700 1 -- 172.21.15.189:0/3539068485 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad340072e0 con 0x7fad40060550 2020-11-20T12:21:01.031 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad1f7fe700 1 -- 172.21.15.189:0/3539068485 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad34007440 con 0x7fad40060550 2020-11-20T12:21:01.031 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad44b35700 1 -- 172.21.15.189:0/3539068485 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad401498b0 con 0x7fad40060550 2020-11-20T12:21:01.031 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad1f7fe700 1 -- 172.21.15.189:0/3539068485 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad34005820 con 0x7fad40060550 2020-11-20T12:21:01.032 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.015+0000 7fad44b35700 1 -- 172.21.15.189:0/3539068485 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad400a3350 con 0x7fad40060550 2020-11-20T12:21:01.032 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.019+0000 7fad44b35700 1 -- 172.21.15.189:0/3539068485 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad401468e0 con 0x7fad40060550 2020-11-20T12:21:01.032 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.019+0000 7fad1f7fe700 1 -- 172.21.15.189:0/3539068485 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fad34010070 con 0x7fad40060550 2020-11-20T12:21:01.032 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.019+0000 7fad1f7fe700 1 --2- 172.21.15.189:0/3539068485 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fad180557c0 0x7fad18057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:01.032 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.019+0000 7fad1f7fe700 1 -- 172.21.15.189:0/3539068485 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 7105+0+0 (secure 0 0 0) 0x7fad3406e7c0 con 0x7fad40060550 2020-11-20T12:21:01.033 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.019+0000 7fad3f7fe700 1 --2- 172.21.15.189:0/3539068485 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fad180557c0 0x7fad18057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:01.033 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.019+0000 7fad3f7fe700 1 --2- 172.21.15.189:0/3539068485 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fad180557c0 0x7fad18057c80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fad3000b150 tx=0x7fad30015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:01.033 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.023+0000 7fad1f7fe700 1 -- 172.21.15.189:0/3539068485 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fad340442d0 con 0x7fad40060550 2020-11-20T12:21:01.245 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.239+0000 7fad44b35700 1 -- 172.21.15.189:0/3539068485 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad40061710 con 0x7fad40060550 2020-11-20T12:21:01.246 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.239+0000 7fad1f7fe700 1 -- 172.21.15.189:0/3539068485 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+1788 (secure 0 0 0) 0x7fad340478f0 con 0x7fad40060550 2020-11-20T12:21:01.246 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:21:01.246 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:21:00.823729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5746":{"gid":5746,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6835/3538437301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3538437301},{"type":"v1","addr":"172.21.15.189:6835","nonce":3538437301}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:21:01.248 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.243+0000 7fad44b35700 1 -- 172.21.15.189:0/3539068485 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fad180557c0 msgr2=0x7fad18057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:01.248 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.243+0000 7fad44b35700 1 --2- 172.21.15.189:0/3539068485 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fad180557c0 0x7fad18057c80 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fad3000b150 tx=0x7fad30015000).stop 2020-11-20T12:21:01.249 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.243+0000 7fad44b35700 1 -- 172.21.15.189:0/3539068485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad40060550 msgr2=0x7fad40146bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:01.249 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.243+0000 7fad44b35700 1 --2- 172.21.15.189:0/3539068485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad40060550 0x7fad40146bc0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7fad3400c6a0 tx=0x7fad34007ac0).stop 2020-11-20T12:21:01.249 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.243+0000 7fad44b35700 1 -- 172.21.15.189:0/3539068485 shutdown_connections 2020-11-20T12:21:01.249 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.243+0000 7fad44b35700 1 --2- 172.21.15.189:0/3539068485 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fad180557c0 0x7fad18057c80 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:01.249 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.243+0000 7fad44b35700 1 --2- 172.21.15.189:0/3539068485 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fad40060170 0x7fad4013aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:01.250 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.243+0000 7fad44b35700 1 --2- 172.21.15.189:0/3539068485 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fad4009cdb0 0x7fad4009e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:01.250 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.243+0000 7fad44b35700 1 --2- 172.21.15.189:0/3539068485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad40060550 0x7fad40146bc0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:01.250 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.243+0000 7fad44b35700 1 -- 172.21.15.189:0/3539068485 >> 172.21.15.189:0/3539068485 conn(0x7fad40062620 msgr2=0x7fad4005ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:01.250 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.243+0000 7fad44b35700 1 -- 172.21.15.189:0/3539068485 shutdown_connections 2020-11-20T12:21:01.250 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:01.243+0000 7fad44b35700 1 -- 172.21.15.189:0/3539068485 wait complete. 2020-11-20T12:21:01.251 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 65 2020-11-20T12:21:01.268 INFO:tasks.mds_thrash.fs.[None]:mds.a has gained rank=0, replacing gid=5520 2020-11-20T12:21:01.269 INFO:tasks.mds_thrash.fs.[None]:waiting for 3 secs before reviving mds.b 2020-11-20T12:21:02.679 INFO:tasks.ceph.mds.a.smithi189.stderr:2020-11-20T12:21:02.671+0000 7f1c1773c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:21:04.270 INFO:tasks.mds_thrash.fs.[None]:reviving mds.b 2020-11-20T12:21:04.271 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T12:21:04.271 INFO:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T12:21:04.280 INFO:tasks.ceph.mds.b:Started 2020-11-20T12:21:04.281 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:04.351 INFO:tasks.ceph.mds.b.smithi047.stdout:starting mds.b at 2020-11-20T12:21:04.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.439+0000 7ffac6b1c700 1 Processor -- start 2020-11-20T12:21:04.446 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.439+0000 7ffac6b1c700 1 -- start start 2020-11-20T12:21:04.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.439+0000 7ffac6b1c700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ffac00623c0 0x7ffac00677f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:04.447 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac6b1c700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ffac013aac0 0x7ffac013af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:04.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac6b1c700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac013b430 con 0x7ffac01340e0 2020-11-20T12:21:04.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac6b1c700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffac00a8e60 con 0x7ffac013aac0 2020-11-20T12:21:04.448 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac6b1c700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac00a9c60 con 0x7ffac00623c0 2020-11-20T12:21:04.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac5b1a700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ffac00623c0 0x7ffac00677f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:04.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac5319700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ffac013aac0 0x7ffac013af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:04.449 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac5319700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7ffac013aac0 0x7ffac013af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.189:58134/0 (socket says 172.21.15.189:58134) 2020-11-20T12:21:04.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac5b1a700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ffac00623c0 0x7ffac00677f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.189:44940/0 (socket says 172.21.15.189:44940) 2020-11-20T12:21:04.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac5319700 1 -- 172.21.15.189:0/3984427596 learned_addr learned my addr 172.21.15.189:0/3984427596 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:21:04.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac631b700 1 --1- 172.21.15.189:0/3984427596 >> v1:172.21.15.47:6789/0 conn(0x7ffac01340e0 0x7ffac0134590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.189:48194/0 (socket says 172.21.15.189:48194) 2020-11-20T12:21:04.450 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac5319700 1 -- 172.21.15.189:0/3984427596 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ffac00623c0 msgr2=0x7ffac00677f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:04.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac5319700 1 --2- 172.21.15.189:0/3984427596 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ffac00623c0 0x7ffac00677f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:04.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac5319700 1 -- 172.21.15.189:0/3984427596 >> v1:172.21.15.47:6789/0 conn(0x7ffac01340e0 legacy=0x7ffac0134590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:04.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac5319700 1 -- 172.21.15.189:0/3984427596 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac0067e90 con 0x7ffac013aac0 2020-11-20T12:21:04.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac5319700 1 --2- 172.21.15.189:0/3984427596 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffac013aac0 0x7ffac013af40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7ffab4006fd0 tx=0x7ffab4006eb0).ready entity=mon.0 client_cookie=b88ddbd2f0814e7d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:04.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac4b18700 1 -- 172.21.15.189:0/3984427596 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab4009810 con 0x7ffac013aac0 2020-11-20T12:21:04.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac4b18700 1 -- 172.21.15.189:0/3984427596 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab4009970 con 0x7ffac013aac0 2020-11-20T12:21:04.458 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac4b18700 1 -- 172.21.15.189:0/3984427596 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab4009cf0 con 0x7ffac013aac0 2020-11-20T12:21:04.459 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac6b1c700 1 -- 172.21.15.189:0/3984427596 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffac013aac0 msgr2=0x7ffac013af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:04.459 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac6b1c700 1 --2- 172.21.15.189:0/3984427596 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffac013aac0 0x7ffac013af40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7ffab4006fd0 tx=0x7ffab4006eb0).stop 2020-11-20T12:21:04.459 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac6b1c700 1 -- 172.21.15.189:0/3984427596 shutdown_connections 2020-11-20T12:21:04.459 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac6b1c700 1 --2- 172.21.15.189:0/3984427596 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ffac00623c0 0x7ffac00677f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:04.460 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac6b1c700 1 --2- 172.21.15.189:0/3984427596 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffac013aac0 0x7ffac013af40 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:04.460 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac6b1c700 1 -- 172.21.15.189:0/3984427596 >> 172.21.15.189:0/3984427596 conn(0x7ffac0065fe0 msgr2=0x7ffac0061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:04.460 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac6b1c700 1 -- 172.21.15.189:0/3984427596 shutdown_connections 2020-11-20T12:21:04.460 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac6b1c700 1 -- 172.21.15.189:0/3984427596 wait complete. 2020-11-20T12:21:04.460 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac6b1c700 1 Processor -- start 2020-11-20T12:21:04.461 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.443+0000 7ffac6b1c700 1 -- start start 2020-11-20T12:21:04.461 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.447+0000 7ffac6b1c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac00623c0 0x7ffac014d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:04.461 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.447+0000 7ffac6b1c700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffac01340e0 0x7ffac0139610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:04.461 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.447+0000 7ffac6b1c700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ffac013aac0 0x7ffac00a3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:04.461 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.447+0000 7ffac6b1c700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac014cdf0 con 0x7ffac01340e0 2020-11-20T12:21:04.462 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.447+0000 7ffac6b1c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac014d020 con 0x7ffac00623c0 2020-11-20T12:21:04.462 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.447+0000 7ffac6b1c700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac014dc50 con 0x7ffac013aac0 2020-11-20T12:21:04.462 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.447+0000 7ffac5b1a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac00623c0 0x7ffac014d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:04.462 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.447+0000 7ffac5b1a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac00623c0 0x7ffac014d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51912/0 (socket says 172.21.15.189:51912) 2020-11-20T12:21:04.462 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.447+0000 7ffac5b1a700 1 -- 172.21.15.189:0/845075853 learned_addr learned my addr 172.21.15.189:0/845075853 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:21:04.463 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.447+0000 7ffac5319700 1 --2- 172.21.15.189:0/845075853 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffac01340e0 0x7ffac0139610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:04.463 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.447+0000 7ffac631b700 1 --2- 172.21.15.189:0/845075853 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ffac013aac0 0x7ffac00a3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:04.463 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.447+0000 7ffac5b1a700 1 -- 172.21.15.189:0/845075853 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ffac013aac0 msgr2=0x7ffac00a3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:04.463 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.447+0000 7ffac5b1a700 1 --2- 172.21.15.189:0/845075853 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ffac013aac0 0x7ffac00a3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:04.463 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.447+0000 7ffac5b1a700 1 -- 172.21.15.189:0/845075853 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffac01340e0 msgr2=0x7ffac0139610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:04.464 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.447+0000 7ffac5b1a700 1 --2- 172.21.15.189:0/845075853 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffac01340e0 0x7ffac0139610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:04.464 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.447+0000 7ffac5b1a700 1 -- 172.21.15.189:0/845075853 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac0139b00 con 0x7ffac00623c0 2020-11-20T12:21:04.464 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.447+0000 7ffac5319700 1 --2- 172.21.15.189:0/845075853 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffac01340e0 0x7ffac0139610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:21:04.464 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.447+0000 7ffac5b1a700 1 --2- 172.21.15.189:0/845075853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac00623c0 0x7ffac014d760 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7ffab000ef80 tx=0x7ffab00091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:04.464 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.447+0000 7ffabdffb700 1 -- 172.21.15.189:0/845075853 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab000cda0 con 0x7ffac00623c0 2020-11-20T12:21:04.465 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.447+0000 7ffabdffb700 1 -- 172.21.15.189:0/845075853 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab00123f0 con 0x7ffac00623c0 2020-11-20T12:21:04.465 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.447+0000 7ffac6b1c700 1 -- 172.21.15.189:0/845075853 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac01339d0 con 0x7ffac00623c0 2020-11-20T12:21:04.465 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.447+0000 7ffabdffb700 1 -- 172.21.15.189:0/845075853 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab0012770 con 0x7ffac00623c0 2020-11-20T12:21:04.465 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.447+0000 7ffac6b1c700 1 -- 172.21.15.189:0/845075853 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac00a45c0 con 0x7ffac00623c0 2020-11-20T12:21:04.465 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.447+0000 7ffac6b1c700 1 -- 172.21.15.189:0/845075853 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffac00663c0 con 0x7ffac00623c0 2020-11-20T12:21:04.466 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.447+0000 7ffabdffb700 1 -- 172.21.15.189:0/845075853 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7ffab0022070 con 0x7ffac00623c0 2020-11-20T12:21:04.466 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.451+0000 7ffabdffb700 1 --2- 172.21.15.189:0/845075853 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ffa980557c0 0x7ffa98057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:04.466 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.451+0000 7ffabdffb700 1 -- 172.21.15.189:0/845075853 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 7105+0+0 (secure 0 0 0) 0x7ffab006f380 con 0x7ffac00623c0 2020-11-20T12:21:04.466 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.451+0000 7ffac5319700 1 --2- 172.21.15.189:0/845075853 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ffa980557c0 0x7ffa98057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:04.466 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.451+0000 7ffac5319700 1 --2- 172.21.15.189:0/845075853 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ffa980557c0 0x7ffa98057c80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7ffab4006ef0 tx=0x7ffab4015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:04.467 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.455+0000 7ffabdffb700 1 -- 172.21.15.189:0/845075853 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ffab0044e10 con 0x7ffac00623c0 2020-11-20T12:21:04.670 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.663+0000 7ffac6b1c700 1 -- 172.21.15.189:0/845075853 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffac00446d0 con 0x7ffac00623c0 2020-11-20T12:21:04.670 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.663+0000 7ffabdffb700 1 -- 172.21.15.189:0/845075853 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+1788 (secure 0 0 0) 0x7ffab0016090 con 0x7ffac00623c0 2020-11-20T12:21:04.671 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:21:04.671 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:21:00.823729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5746":{"gid":5746,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6835/3538437301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3538437301},{"type":"v1","addr":"172.21.15.189:6835","nonce":3538437301}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:21:04.673 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.667+0000 7ffac6b1c700 1 -- 172.21.15.189:0/845075853 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ffa980557c0 msgr2=0x7ffa98057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:04.673 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.667+0000 7ffac6b1c700 1 --2- 172.21.15.189:0/845075853 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ffa980557c0 0x7ffa98057c80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7ffab4006ef0 tx=0x7ffab4015000).stop 2020-11-20T12:21:04.673 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.667+0000 7ffac6b1c700 1 -- 172.21.15.189:0/845075853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac00623c0 msgr2=0x7ffac014d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:04.673 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.667+0000 7ffac6b1c700 1 --2- 172.21.15.189:0/845075853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac00623c0 0x7ffac014d760 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7ffab000ef80 tx=0x7ffab00091d0).stop 2020-11-20T12:21:04.674 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.667+0000 7ffac6b1c700 1 -- 172.21.15.189:0/845075853 shutdown_connections 2020-11-20T12:21:04.674 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.667+0000 7ffac6b1c700 1 --2- 172.21.15.189:0/845075853 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7ffac013aac0 0x7ffac00a3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:04.674 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.667+0000 7ffac6b1c700 1 --2- 172.21.15.189:0/845075853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac00623c0 0x7ffac014d760 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:04.674 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.667+0000 7ffac6b1c700 1 --2- 172.21.15.189:0/845075853 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7ffa980557c0 0x7ffa98057c80 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:04.674 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.667+0000 7ffac6b1c700 1 --2- 172.21.15.189:0/845075853 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7ffac01340e0 0x7ffac0139610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:04.675 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.667+0000 7ffac6b1c700 1 -- 172.21.15.189:0/845075853 >> 172.21.15.189:0/845075853 conn(0x7ffac0065fe0 msgr2=0x7ffac00a08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:04.675 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.667+0000 7ffac6b1c700 1 -- 172.21.15.189:0/845075853 shutdown_connections 2020-11-20T12:21:04.675 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:04.667+0000 7ffac6b1c700 1 -- 172.21.15.189:0/845075853 wait complete. 2020-11-20T12:21:04.675 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 65 2020-11-20T12:21:04.686 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2020-11-20T12:21:06.688 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:06.810 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.803+0000 7f91dc09b700 1 Processor -- start 2020-11-20T12:21:06.810 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.803+0000 7f91dc09b700 1 -- start start 2020-11-20T12:21:06.813 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.803+0000 7f91dc09b700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f91d4068020 0x7f91d41339f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:06.814 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.803+0000 7f91dc09b700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f91d40a9290 0x7f91d4133ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:06.814 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.803+0000 7f91dc09b700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d40663e0 con 0x7f91d40a9e20 2020-11-20T12:21:06.814 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.803+0000 7f91dc09b700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91d4067b80 con 0x7f91d4068020 2020-11-20T12:21:06.814 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.803+0000 7f91dc09b700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d40a9970 con 0x7f91d40a9290 2020-11-20T12:21:06.815 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d9e37700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f91d40a9e20 0x7f91d40a83f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53542/0 (socket says 172.21.15.189:53542) 2020-11-20T12:21:06.815 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d9e37700 1 -- 172.21.15.189:0/4217377523 learned_addr learned my addr 172.21.15.189:0/4217377523 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:21:06.815 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d9636700 1 --2- 172.21.15.189:0/4217377523 >> v2:172.21.15.47:3300/0 conn(0x7f91d4068020 0x7f91d41339f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:06.815 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91da638700 1 --2- 172.21.15.189:0/4217377523 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f91d40a9290 0x7f91d4133ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:06.815 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d8e35700 1 -- 172.21.15.189:0/4217377523 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745137210 0 0) 0x7f91d40663e0 con 0x7f91d40a9e20 2020-11-20T12:21:06.816 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d8e35700 1 -- 172.21.15.189:0/4217377523 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91bc003240 con 0x7f91d40a9e20 2020-11-20T12:21:06.816 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d8e35700 1 -- 172.21.15.189:0/4217377523 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2332494018 0 0) 0x7f91bc003240 con 0x7f91d40a9e20 2020-11-20T12:21:06.816 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d8e35700 1 -- 172.21.15.189:0/4217377523 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f91d40a9290 msgr2=0x7f91d4133ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:06.816 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d8e35700 1 --2- 172.21.15.189:0/4217377523 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f91d40a9290 0x7f91d4133ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:06.816 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d8e35700 1 -- 172.21.15.189:0/4217377523 >> v2:172.21.15.47:3300/0 conn(0x7f91d4068020 msgr2=0x7f91d41339f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:06.817 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d8e35700 1 --2- 172.21.15.189:0/4217377523 >> v2:172.21.15.47:3300/0 conn(0x7f91d4068020 0x7f91d41339f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:06.817 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d8e35700 1 -- 172.21.15.189:0/4217377523 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d4066680 con 0x7f91d40a9e20 2020-11-20T12:21:06.817 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d8e35700 1 -- 172.21.15.189:0/4217377523 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f91d00014e0 con 0x7f91d40a9e20 2020-11-20T12:21:06.817 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d8e35700 1 -- 172.21.15.189:0/4217377523 >> v1:172.21.15.189:6789/0 conn(0x7f91d40a9e20 legacy=0x7f91d40a83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:06.817 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d8e35700 1 --2- 172.21.15.189:0/4217377523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91bc003730 0x7f91bc003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:06.818 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d8e35700 1 --2- 172.21.15.189:0/4217377523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91bc0044e0 0x7f91bc004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:06.818 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d8e35700 1 --2- 172.21.15.189:0/4217377523 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f91d40a9290 0x7f91bc004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:06.818 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d8e35700 1 -- 172.21.15.189:0/4217377523 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d4067b80 con 0x7f91bc0044e0 2020-11-20T12:21:06.818 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d8e35700 1 -- 172.21.15.189:0/4217377523 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d40a9970 con 0x7f91bc003730 2020-11-20T12:21:06.818 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d8e35700 1 -- 172.21.15.189:0/4217377523 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91bc004270 con 0x7f91d40a9290 2020-11-20T12:21:06.819 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d9e37700 1 --2- 172.21.15.189:0/4217377523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91bc003730 0x7f91bc003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:06.819 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91da638700 1 --2- 172.21.15.189:0/4217377523 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f91d40a9290 0x7f91bc004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:06.819 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d9636700 1 --2- 172.21.15.189:0/4217377523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91bc0044e0 0x7f91bc004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:06.819 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d9e37700 1 -- 172.21.15.189:0/4217377523 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f91d40a9290 msgr2=0x7f91bc004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:06.819 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d9e37700 1 --2- 172.21.15.189:0/4217377523 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f91d40a9290 0x7f91bc004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:06.820 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d9e37700 1 -- 172.21.15.189:0/4217377523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91bc0044e0 msgr2=0x7f91bc004960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:06.820 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d9e37700 1 --2- 172.21.15.189:0/4217377523 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91bc0044e0 0x7f91bc004960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:06.820 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d9e37700 1 -- 172.21.15.189:0/4217377523 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91bc003240 con 0x7f91bc003730 2020-11-20T12:21:06.820 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91da638700 1 --2- 172.21.15.189:0/4217377523 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f91d40a9290 0x7f91bc004e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:21:06.820 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d9e37700 1 --2- 172.21.15.189:0/4217377523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91bc003730 0x7f91bc003b50 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7f91d000c440 tx=0x7f91d0007800).ready entity=mon.1 client_cookie=a5185685ad5209a0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:06.821 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d8e35700 1 -- 172.21.15.189:0/4217377523 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d000ce00 con 0x7f91bc003730 2020-11-20T12:21:06.821 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d8e35700 1 -- 172.21.15.189:0/4217377523 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d00072e0 con 0x7f91bc003730 2020-11-20T12:21:06.821 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91d8e35700 1 -- 172.21.15.189:0/4217377523 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d0005820 con 0x7f91bc003730 2020-11-20T12:21:06.824 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91dc09b700 1 -- 172.21.15.189:0/4217377523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91bc003730 msgr2=0x7f91bc003b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:06.824 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91dc09b700 1 --2- 172.21.15.189:0/4217377523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91bc003730 0x7f91bc003b50 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7f91d000c440 tx=0x7f91d0007800).stop 2020-11-20T12:21:06.824 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91da638700 1 -- 172.21.15.189:0/4217377523 reap_dead start 2020-11-20T12:21:06.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91dc09b700 1 -- 172.21.15.189:0/4217377523 shutdown_connections 2020-11-20T12:21:06.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91dc09b700 1 -- 172.21.15.189:0/4217377523 >> 172.21.15.189:0/4217377523 conn(0x7f91d40627f0 msgr2=0x7f91d4065870 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:06.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91dc09b700 1 -- 172.21.15.189:0/4217377523 shutdown_connections 2020-11-20T12:21:06.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.807+0000 7f91dc09b700 1 -- 172.21.15.189:0/4217377523 wait complete. 2020-11-20T12:21:06.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.811+0000 7f91dc09b700 1 Processor -- start 2020-11-20T12:21:06.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.811+0000 7f91dc09b700 1 -- start start 2020-11-20T12:21:06.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.811+0000 7f91dc09b700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f91d413c440 0x7f91d413d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:06.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.811+0000 7f91dc09b700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91d4136070 0x7f91d413d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:06.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.811+0000 7f91dc09b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91d41518a0 0x7f91d4151d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:06.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.811+0000 7f91dc09b700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d4137a10 con 0x7f91d4136070 2020-11-20T12:21:06.827 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.811+0000 7f91dc09b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d413de30 con 0x7f91d41518a0 2020-11-20T12:21:06.827 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.811+0000 7f91dc09b700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91d40ac3d0 con 0x7f91d413c440 2020-11-20T12:21:06.827 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.811+0000 7f91da638700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91d41518a0 0x7f91d4151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:06.827 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.811+0000 7f91da638700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91d41518a0 0x7f91d4151d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51936/0 (socket says 172.21.15.189:51936) 2020-11-20T12:21:06.827 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.811+0000 7f91da638700 1 -- 172.21.15.189:0/1205321821 learned_addr learned my addr 172.21.15.189:0/1205321821 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:21:06.828 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.811+0000 7f91d9e37700 1 --2- 172.21.15.189:0/1205321821 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f91d413c440 0x7f91d413d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:06.828 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.811+0000 7f91d9636700 1 --2- 172.21.15.189:0/1205321821 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91d4136070 0x7f91d413d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:06.828 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.811+0000 7f91da638700 1 -- 172.21.15.189:0/1205321821 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f91d413c440 msgr2=0x7f91d413d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:06.828 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.811+0000 7f91da638700 1 --2- 172.21.15.189:0/1205321821 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f91d413c440 0x7f91d413d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:06.828 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.811+0000 7f91da638700 1 -- 172.21.15.189:0/1205321821 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91d4136070 msgr2=0x7f91d413d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:06.829 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.811+0000 7f91da638700 1 --2- 172.21.15.189:0/1205321821 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91d4136070 0x7f91d413d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:06.829 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.811+0000 7f91da638700 1 -- 172.21.15.189:0/1205321821 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d40ac670 con 0x7f91d41518a0 2020-11-20T12:21:06.829 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.811+0000 7f91d9636700 1 --2- 172.21.15.189:0/1205321821 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91d4136070 0x7f91d413d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:21:06.829 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.811+0000 7f91da638700 1 --2- 172.21.15.189:0/1205321821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91d41518a0 0x7f91d4151d20 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7f91d40a7f10 tx=0x7f91cc008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:06.829 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.811+0000 7f91c1ffb700 1 -- 172.21.15.189:0/1205321821 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91cc00e960 con 0x7f91d41518a0 2020-11-20T12:21:06.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.811+0000 7f91c1ffb700 1 -- 172.21.15.189:0/1205321821 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91cc00eae0 con 0x7f91d41518a0 2020-11-20T12:21:06.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.811+0000 7f91dc09b700 1 -- 172.21.15.189:0/1205321821 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d413c820 con 0x7f91d41518a0 2020-11-20T12:21:06.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.811+0000 7f91c1ffb700 1 -- 172.21.15.189:0/1205321821 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91cc00eea0 con 0x7f91d41518a0 2020-11-20T12:21:06.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.811+0000 7f91dc09b700 1 -- 172.21.15.189:0/1205321821 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d40ac050 con 0x7f91d41518a0 2020-11-20T12:21:06.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.815+0000 7f91dc09b700 1 -- 172.21.15.189:0/1205321821 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91d40ac800 con 0x7f91d41518a0 2020-11-20T12:21:06.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.815+0000 7f91c1ffb700 1 -- 172.21.15.189:0/1205321821 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f91cc01e030 con 0x7f91d41518a0 2020-11-20T12:21:06.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.815+0000 7f91c1ffb700 1 --2- 172.21.15.189:0/1205321821 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f91ac0557e0 0x7f91ac057ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:06.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.815+0000 7f91d9e37700 1 --2- 172.21.15.189:0/1205321821 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f91ac0557e0 0x7f91ac057ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:06.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.815+0000 7f91c1ffb700 1 -- 172.21.15.189:0/1205321821 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 7105+0+0 (secure 0 0 0) 0x7f91cc06e680 con 0x7f91d41518a0 2020-11-20T12:21:06.832 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.815+0000 7f91d9e37700 1 --2- 172.21.15.189:0/1205321821 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f91ac0557e0 0x7f91ac057ca0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f91d4059e90 tx=0x7f91d0015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:06.832 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:06.819+0000 7f91c1ffb700 1 -- 172.21.15.189:0/1205321821 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f91cc0441c0 con 0x7f91d41518a0 2020-11-20T12:21:07.031 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:07.027+0000 7f91dc09b700 1 -- 172.21.15.189:0/1205321821 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91d4044f60 con 0x7f91d41518a0 2020-11-20T12:21:07.032 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:07.027+0000 7f91c1ffb700 1 -- 172.21.15.189:0/1205321821 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+1788 (secure 0 0 0) 0x7f91cc0477e0 con 0x7f91d41518a0 2020-11-20T12:21:07.032 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:21:07.032 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:21:00.823729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5746":{"gid":5746,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6835/3538437301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3538437301},{"type":"v1","addr":"172.21.15.189:6835","nonce":3538437301}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:21:07.034 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:07.027+0000 7f91dc09b700 1 -- 172.21.15.189:0/1205321821 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f91ac0557e0 msgr2=0x7f91ac057ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:07.035 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:07.027+0000 7f91dc09b700 1 --2- 172.21.15.189:0/1205321821 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f91ac0557e0 0x7f91ac057ca0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f91d4059e90 tx=0x7f91d0015000).stop 2020-11-20T12:21:07.035 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:07.027+0000 7f91dc09b700 1 -- 172.21.15.189:0/1205321821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91d41518a0 msgr2=0x7f91d4151d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:07.035 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:07.027+0000 7f91dc09b700 1 --2- 172.21.15.189:0/1205321821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91d41518a0 0x7f91d4151d20 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7f91d40a7f10 tx=0x7f91cc008040).stop 2020-11-20T12:21:07.035 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:07.027+0000 7f91dc09b700 1 -- 172.21.15.189:0/1205321821 shutdown_connections 2020-11-20T12:21:07.036 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:07.027+0000 7f91dc09b700 1 --2- 172.21.15.189:0/1205321821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91d41518a0 0x7f91d4151d20 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:07.036 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:07.027+0000 7f91dc09b700 1 --2- 172.21.15.189:0/1205321821 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f91d413c440 0x7f91d413d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:07.036 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:07.027+0000 7f91dc09b700 1 --2- 172.21.15.189:0/1205321821 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f91ac0557e0 0x7f91ac057ca0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:07.036 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:07.027+0000 7f91dc09b700 1 --2- 172.21.15.189:0/1205321821 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f91d4136070 0x7f91d413d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:07.036 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:07.027+0000 7f91dc09b700 1 -- 172.21.15.189:0/1205321821 >> 172.21.15.189:0/1205321821 conn(0x7f91d40627f0 msgr2=0x7f91d40adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:07.037 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:07.027+0000 7f91dc09b700 1 -- 172.21.15.189:0/1205321821 shutdown_connections 2020-11-20T12:21:07.037 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:07.027+0000 7f91dc09b700 1 -- 172.21.15.189:0/1205321821 wait complete. 2020-11-20T12:21:07.038 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 65 2020-11-20T12:21:07.047 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2020-11-20T12:21:09.048 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:09.210 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.202+0000 7f7291532700 1 Processor -- start 2020-11-20T12:21:09.211 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.202+0000 7f7291532700 1 -- start start 2020-11-20T12:21:09.213 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.202+0000 7f7291532700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f728c0ad000 0x7f728c13e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:09.213 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.202+0000 7f7291532700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f728c13a6e0 0x7f728c13ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:09.213 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.202+0000 7f7291532700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f728c067730 con 0x7f728c0ad3e0 2020-11-20T12:21:09.214 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.202+0000 7f7291532700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f728c138190 con 0x7f728c0ad000 2020-11-20T12:21:09.214 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.202+0000 7f7291532700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f728c13dd40 con 0x7f728c13a6e0 2020-11-20T12:21:09.214 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.202+0000 7f7290d31700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f728c13a6e0 0x7f728c13ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:09.215 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.206+0000 7f7290d31700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f728c13a6e0 0x7f728c13ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51944/0 (socket says 172.21.15.189:51944) 2020-11-20T12:21:09.215 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.206+0000 7f7290d31700 1 -- 172.21.15.189:0/2396064943 learned_addr learned my addr 172.21.15.189:0/2396064943 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:21:09.215 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.206+0000 7f728bfff700 1 --2- 172.21.15.189:0/2396064943 >> v2:172.21.15.47:3300/0 conn(0x7f728c0ad000 0x7f728c13e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:09.216 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.206+0000 7f7290d31700 1 -- 172.21.15.189:0/2396064943 >> v2:172.21.15.47:3300/0 conn(0x7f728c0ad000 msgr2=0x7f728c13e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:09.222 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.206+0000 7f7290d31700 1 --2- 172.21.15.189:0/2396064943 >> v2:172.21.15.47:3300/0 conn(0x7f728c0ad000 0x7f728c13e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:09.223 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.206+0000 7f7290d31700 1 -- 172.21.15.189:0/2396064943 >> v1:172.21.15.47:6789/0 conn(0x7f728c0ad3e0 legacy=0x7f728c066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:09.223 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.206+0000 7f7290d31700 1 -- 172.21.15.189:0/2396064943 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f728c13dfe0 con 0x7f728c13a6e0 2020-11-20T12:21:09.223 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.206+0000 7f728bfff700 1 --2- 172.21.15.189:0/2396064943 >> v2:172.21.15.47:3300/0 conn(0x7f728c0ad000 0x7f728c13e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:21:09.223 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.206+0000 7f7290d31700 1 --2- 172.21.15.189:0/2396064943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f728c13a6e0 0x7f728c13ad00 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f728c068780 tx=0x7f7284009290).ready entity=mon.1 client_cookie=d2150b837efd067e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:09.224 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.206+0000 7f728b7fe700 1 -- 172.21.15.189:0/2396064943 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728400bbc0 con 0x7f728c13a6e0 2020-11-20T12:21:09.224 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.206+0000 7f728b7fe700 1 -- 172.21.15.189:0/2396064943 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f728400bd20 con 0x7f728c13a6e0 2020-11-20T12:21:09.224 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.206+0000 7f728b7fe700 1 -- 172.21.15.189:0/2396064943 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7284011610 con 0x7f728c13a6e0 2020-11-20T12:21:09.224 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.206+0000 7f7291532700 1 -- 172.21.15.189:0/2396064943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f728c13a6e0 msgr2=0x7f728c13ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:09.224 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.206+0000 7f7291532700 1 --2- 172.21.15.189:0/2396064943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f728c13a6e0 0x7f728c13ad00 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f728c068780 tx=0x7f7284009290).stop 2020-11-20T12:21:09.225 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.206+0000 7f7291532700 1 -- 172.21.15.189:0/2396064943 shutdown_connections 2020-11-20T12:21:09.225 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.206+0000 7f7291532700 1 --2- 172.21.15.189:0/2396064943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f728c13a6e0 0x7f728c13ad00 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:09.225 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.206+0000 7f7291532700 1 --2- 172.21.15.189:0/2396064943 >> v2:172.21.15.47:3300/0 conn(0x7f728c0ad000 0x7f728c13e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:09.225 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.206+0000 7f7291532700 1 -- 172.21.15.189:0/2396064943 >> 172.21.15.189:0/2396064943 conn(0x7f728c062080 msgr2=0x7f728c062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:09.226 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.206+0000 7f7291532700 1 -- 172.21.15.189:0/2396064943 shutdown_connections 2020-11-20T12:21:09.226 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.206+0000 7f7291532700 1 -- 172.21.15.189:0/2396064943 wait complete. 2020-11-20T12:21:09.226 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.206+0000 7f7291532700 1 Processor -- start 2020-11-20T12:21:09.226 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.206+0000 7f7291532700 1 -- start start 2020-11-20T12:21:09.226 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.210+0000 7f7291532700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f728c0ad000 0x7f728c140a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:09.227 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.210+0000 7f7291532700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f728c0ad3e0 0x7f728c0a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:09.227 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.210+0000 7f7291532700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f728c0a31b0 0x7f728c0a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:09.227 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.210+0000 7f7291532700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728c0a5110 con 0x7f728c0ad000 2020-11-20T12:21:09.227 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.210+0000 7f7291532700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728c141890 con 0x7f728c0ad3e0 2020-11-20T12:21:09.227 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.210+0000 7f7291532700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f728c0a5300 con 0x7f728c0a31b0 2020-11-20T12:21:09.228 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.210+0000 7f7283fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f728c0ad3e0 0x7f728c0a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:09.228 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.210+0000 7f7283fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f728c0ad3e0 0x7f728c0a2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51948/0 (socket says 172.21.15.189:51948) 2020-11-20T12:21:09.228 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.210+0000 7f7283fff700 1 -- 172.21.15.189:0/1502563619 learned_addr learned my addr 172.21.15.189:0/1502563619 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:21:09.228 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.210+0000 7f728bfff700 1 --2- 172.21.15.189:0/1502563619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f728c0ad000 0x7f728c140a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:09.228 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.210+0000 7f7290d31700 1 --2- 172.21.15.189:0/1502563619 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f728c0a31b0 0x7f728c0a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:09.229 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.210+0000 7f7283fff700 1 -- 172.21.15.189:0/1502563619 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f728c0a31b0 msgr2=0x7f728c0a35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:09.229 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.210+0000 7f7283fff700 1 --2- 172.21.15.189:0/1502563619 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f728c0a31b0 0x7f728c0a35f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:09.229 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.210+0000 7f7283fff700 1 -- 172.21.15.189:0/1502563619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f728c0ad000 msgr2=0x7f728c140a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:09.230 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.210+0000 7f7283fff700 1 --2- 172.21.15.189:0/1502563619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f728c0ad000 0x7f728c140a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:09.230 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.210+0000 7f7283fff700 1 -- 172.21.15.189:0/1502563619 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f728c0a55a0 con 0x7f728c0ad3e0 2020-11-20T12:21:09.230 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.210+0000 7f728bfff700 1 --2- 172.21.15.189:0/1502563619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f728c0ad000 0x7f728c140a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:21:09.230 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.210+0000 7f7283fff700 1 --2- 172.21.15.189:0/1502563619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f728c0ad3e0 0x7f728c0a2580 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7f7274001e70 tx=0x7f7274007c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:09.230 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.210+0000 7f7288ff9700 1 -- 172.21.15.189:0/1502563619 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7274007020 con 0x7f728c0ad3e0 2020-11-20T12:21:09.231 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.210+0000 7f7288ff9700 1 -- 172.21.15.189:0/1502563619 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7274007180 con 0x7f728c0ad3e0 2020-11-20T12:21:09.231 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.210+0000 7f7291532700 1 -- 172.21.15.189:0/1502563619 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f728c0a5840 con 0x7f728c0ad3e0 2020-11-20T12:21:09.231 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.210+0000 7f7288ff9700 1 -- 172.21.15.189:0/1502563619 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f727400e560 con 0x7f728c0ad3e0 2020-11-20T12:21:09.231 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.210+0000 7f7291532700 1 -- 172.21.15.189:0/1502563619 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f728c14d030 con 0x7f728c0ad3e0 2020-11-20T12:21:09.231 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.210+0000 7f7291532700 1 -- 172.21.15.189:0/1502563619 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f728c0acdb0 con 0x7f728c0ad3e0 2020-11-20T12:21:09.232 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.210+0000 7f7288ff9700 1 -- 172.21.15.189:0/1502563619 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f7274010070 con 0x7f728c0ad3e0 2020-11-20T12:21:09.232 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.214+0000 7f7288ff9700 1 --2- 172.21.15.189:0/1502563619 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f72680557f0 0x7f7268057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:09.232 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.214+0000 7f7288ff9700 1 -- 172.21.15.189:0/1502563619 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 7105+0+0 (secure 0 0 0) 0x7f727406dea0 con 0x7f728c0ad3e0 2020-11-20T12:21:09.232 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.214+0000 7f728bfff700 1 --2- 172.21.15.189:0/1502563619 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f72680557f0 0x7f7268057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:09.232 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.214+0000 7f728bfff700 1 --2- 172.21.15.189:0/1502563619 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f72680557f0 0x7f7268057cb0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f727c00bac0 tx=0x7f727c008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:09.233 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.218+0000 7f7288ff9700 1 -- 172.21.15.189:0/1502563619 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7274042fd0 con 0x7f728c0ad3e0 2020-11-20T12:21:09.460 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.454+0000 7f7291532700 1 -- 172.21.15.189:0/1502563619 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f728c0a59d0 con 0x7f728c0ad3e0 2020-11-20T12:21:09.461 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.454+0000 7f7288ff9700 1 -- 172.21.15.189:0/1502563619 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f728c0a59d0 con 0x7f728c0ad3e0 2020-11-20T12:21:09.461 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:21:09.461 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5297,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.47:6833/3025803398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3025803398},{"type":"v1","addr":"172.21.15.47:6833","nonce":3025803398}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:21:00.823729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5746":{"gid":5746,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6835/3538437301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3538437301},{"type":"v1","addr":"172.21.15.189:6835","nonce":3538437301}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:21:09.464 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.454+0000 7f7291532700 1 -- 172.21.15.189:0/1502563619 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f72680557f0 msgr2=0x7f7268057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:09.464 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.454+0000 7f7291532700 1 --2- 172.21.15.189:0/1502563619 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f72680557f0 0x7f7268057cb0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f727c00bac0 tx=0x7f727c008040).stop 2020-11-20T12:21:09.465 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.454+0000 7f7291532700 1 -- 172.21.15.189:0/1502563619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f728c0ad3e0 msgr2=0x7f728c0a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:09.465 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.454+0000 7f7291532700 1 --2- 172.21.15.189:0/1502563619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f728c0ad3e0 0x7f728c0a2580 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7f7274001e70 tx=0x7f7274007c50).stop 2020-11-20T12:21:09.465 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.454+0000 7f7291532700 1 -- 172.21.15.189:0/1502563619 shutdown_connections 2020-11-20T12:21:09.466 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.454+0000 7f7291532700 1 --2- 172.21.15.189:0/1502563619 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f72680557f0 0x7f7268057cb0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:09.466 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.454+0000 7f7291532700 1 --2- 172.21.15.189:0/1502563619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f728c0ad000 0x7f728c140a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:09.467 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.454+0000 7f7291532700 1 --2- 172.21.15.189:0/1502563619 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f728c0a31b0 0x7f728c0a35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:09.467 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.454+0000 7f7291532700 1 --2- 172.21.15.189:0/1502563619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f728c0ad3e0 0x7f728c0a2580 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:09.467 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.454+0000 7f7291532700 1 -- 172.21.15.189:0/1502563619 >> 172.21.15.189:0/1502563619 conn(0x7f728c062080 msgr2=0x7f728c05ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:09.468 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.454+0000 7f7291532700 1 -- 172.21.15.189:0/1502563619 shutdown_connections 2020-11-20T12:21:09.468 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:09.454+0000 7f7291532700 1 -- 172.21.15.189:0/1502563619 wait complete. 2020-11-20T12:21:09.469 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 66 2020-11-20T12:21:09.477 INFO:tasks.mds_thrash.fs.[None]:mds.b reported in up:standby state 2020-11-20T12:21:09.478 INFO:tasks.mds_thrash.fs.[None]:waiting for 21 secs before thrashing 2020-11-20T12:21:26.274 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:21:26.276 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:21:30.479 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:30.604 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd5c8a7700 1 Processor -- start 2020-11-20T12:21:30.605 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd5c8a7700 1 -- start start 2020-11-20T12:21:30.607 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd5c8a7700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcd580ad3e0 0x7fcd58066d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:30.607 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd5c8a7700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fcd5813a6e0 0x7fcd5813ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:30.607 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd5c8a7700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd58067730 con 0x7fcd580ad000 2020-11-20T12:21:30.608 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd5c8a7700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd58066050 con 0x7fcd5813a6e0 2020-11-20T12:21:30.608 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd5c8a7700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd58138190 con 0x7fcd580ad3e0 2020-11-20T12:21:30.608 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd56d9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fcd580ad000 0x7fcd5813e480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53576/0 (socket says 172.21.15.189:53576) 2020-11-20T12:21:30.608 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd56d9d700 1 -- 172.21.15.189:0/2923714797 learned_addr learned my addr 172.21.15.189:0/2923714797 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:21:30.608 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd5659c700 1 --2- 172.21.15.189:0/2923714797 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcd580ad3e0 0x7fcd58066d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:30.609 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd5759e700 1 --2- 172.21.15.189:0/2923714797 >> v2:172.21.15.47:3300/0 conn(0x7fcd5813a6e0 0x7fcd5813ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:30.609 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd55d9b700 1 -- 172.21.15.189:0/2923714797 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3295492528 0 0) 0x7fcd58067730 con 0x7fcd580ad000 2020-11-20T12:21:30.609 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd55d9b700 1 -- 172.21.15.189:0/2923714797 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd44003240 con 0x7fcd580ad000 2020-11-20T12:21:30.609 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd55d9b700 1 -- 172.21.15.189:0/2923714797 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 187808583 0 0) 0x7fcd44003240 con 0x7fcd580ad000 2020-11-20T12:21:30.610 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd55d9b700 1 -- 172.21.15.189:0/2923714797 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcd580ad3e0 msgr2=0x7fcd58066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:30.610 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd55d9b700 1 --2- 172.21.15.189:0/2923714797 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcd580ad3e0 0x7fcd58066d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.610 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd55d9b700 1 -- 172.21.15.189:0/2923714797 >> v2:172.21.15.47:3300/0 conn(0x7fcd5813a6e0 msgr2=0x7fcd5813ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:30.610 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd55d9b700 1 --2- 172.21.15.189:0/2923714797 >> v2:172.21.15.47:3300/0 conn(0x7fcd5813a6e0 0x7fcd5813ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.610 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd55d9b700 1 -- 172.21.15.189:0/2923714797 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd5813dd40 con 0x7fcd580ad000 2020-11-20T12:21:30.611 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd55d9b700 1 -- 172.21.15.189:0/2923714797 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7fcd480014e0 con 0x7fcd580ad000 2020-11-20T12:21:30.611 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd55d9b700 1 -- 172.21.15.189:0/2923714797 >> v1:172.21.15.189:6789/0 conn(0x7fcd580ad000 legacy=0x7fcd5813e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:30.611 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd55d9b700 1 --2- 172.21.15.189:0/2923714797 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcd580ad3e0 0x7fcd44003730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:30.611 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd55d9b700 1 --2- 172.21.15.189:0/2923714797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd44003c40 0x7fcd440044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:30.611 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd55d9b700 1 --2- 172.21.15.189:0/2923714797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcd440049d0 0x7fcd44004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:30.612 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd55d9b700 1 -- 172.21.15.189:0/2923714797 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd58138190 con 0x7fcd440049d0 2020-11-20T12:21:30.612 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd55d9b700 1 -- 172.21.15.189:0/2923714797 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd44004180 con 0x7fcd44003c40 2020-11-20T12:21:30.612 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd55d9b700 1 -- 172.21.15.189:0/2923714797 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd44004340 con 0x7fcd580ad3e0 2020-11-20T12:21:30.612 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd5659c700 1 --2- 172.21.15.189:0/2923714797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd44003c40 0x7fcd440044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:30.612 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd5759e700 1 --2- 172.21.15.189:0/2923714797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcd440049d0 0x7fcd44004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:30.613 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd56d9d700 1 --2- 172.21.15.189:0/2923714797 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcd580ad3e0 0x7fcd44003730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:30.613 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd5659c700 1 -- 172.21.15.189:0/2923714797 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcd580ad3e0 msgr2=0x7fcd44003730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:30.613 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd5659c700 1 --2- 172.21.15.189:0/2923714797 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcd580ad3e0 0x7fcd44003730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.613 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd5659c700 1 -- 172.21.15.189:0/2923714797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcd440049d0 msgr2=0x7fcd44004e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:30.613 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd5659c700 1 --2- 172.21.15.189:0/2923714797 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcd440049d0 0x7fcd44004e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd5659c700 1 -- 172.21.15.189:0/2923714797 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd5813dd40 con 0x7fcd44003c40 2020-11-20T12:21:30.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd56d9d700 1 --2- 172.21.15.189:0/2923714797 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcd580ad3e0 0x7fcd44003730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:21:30.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.598+0000 7fcd5659c700 1 --2- 172.21.15.189:0/2923714797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd44003c40 0x7fcd440044e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7fcd4c00bc00 tx=0x7fcd4c006df0).ready entity=mon.1 client_cookie=2ba29c6053074574 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:30.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd55d9b700 1 -- 172.21.15.189:0/2923714797 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4c00b310 con 0x7fcd44003c40 2020-11-20T12:21:30.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd55d9b700 1 -- 172.21.15.189:0/2923714797 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd4c00eae0 con 0x7fcd44003c40 2020-11-20T12:21:30.615 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd55d9b700 1 -- 172.21.15.189:0/2923714797 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4c00ee60 con 0x7fcd44003c40 2020-11-20T12:21:30.618 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd5c8a7700 1 -- 172.21.15.189:0/2923714797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd44003c40 msgr2=0x7fcd440044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:30.618 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd5c8a7700 1 --2- 172.21.15.189:0/2923714797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd44003c40 0x7fcd440044e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7fcd4c00bc00 tx=0x7fcd4c006df0).stop 2020-11-20T12:21:30.618 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd5759e700 1 -- 172.21.15.189:0/2923714797 reap_dead start 2020-11-20T12:21:30.618 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd5c8a7700 1 -- 172.21.15.189:0/2923714797 shutdown_connections 2020-11-20T12:21:30.619 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd5c8a7700 1 -- 172.21.15.189:0/2923714797 >> 172.21.15.189:0/2923714797 conn(0x7fcd58062080 msgr2=0x7fcd58062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:30.619 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd5c8a7700 1 -- 172.21.15.189:0/2923714797 shutdown_connections 2020-11-20T12:21:30.619 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd5c8a7700 1 -- 172.21.15.189:0/2923714797 wait complete. 2020-11-20T12:21:30.619 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd5c8a7700 1 Processor -- start 2020-11-20T12:21:30.619 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd5c8a7700 1 -- start start 2020-11-20T12:21:30.620 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd5c8a7700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcd44003c40 0x7fcd580aae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:30.620 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd5c8a7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd58140950 0x7fcd580a97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:30.620 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd5c8a7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcd580a9cc0 0x7fcd5814cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:30.620 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd5c8a7700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd58140dd0 con 0x7fcd580a9cc0 2020-11-20T12:21:30.620 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd5c8a7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd580a1df0 con 0x7fcd58140950 2020-11-20T12:21:30.621 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd5c8a7700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd580a2050 con 0x7fcd44003c40 2020-11-20T12:21:30.621 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd5659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd58140950 0x7fcd580a97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:30.621 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd5659c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd58140950 0x7fcd580a97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51968/0 (socket says 172.21.15.189:51968) 2020-11-20T12:21:30.621 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd5659c700 1 -- 172.21.15.189:0/109733762 learned_addr learned my addr 172.21.15.189:0/109733762 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:21:30.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd5759e700 1 --2- 172.21.15.189:0/109733762 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcd580a9cc0 0x7fcd5814cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:30.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd56d9d700 1 --2- 172.21.15.189:0/109733762 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcd44003c40 0x7fcd580aae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:30.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd5659c700 1 -- 172.21.15.189:0/109733762 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcd44003c40 msgr2=0x7fcd580aae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:30.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd5659c700 1 --2- 172.21.15.189:0/109733762 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcd44003c40 0x7fcd580aae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.622 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd5659c700 1 -- 172.21.15.189:0/109733762 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcd580a9cc0 msgr2=0x7fcd5814cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:30.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd5659c700 1 --2- 172.21.15.189:0/109733762 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcd580a9cc0 0x7fcd5814cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd5659c700 1 -- 172.21.15.189:0/109733762 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd580a22f0 con 0x7fcd58140950 2020-11-20T12:21:30.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd56d9d700 1 --2- 172.21.15.189:0/109733762 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcd44003c40 0x7fcd580aae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:21:30.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.602+0000 7fcd5759e700 1 --2- 172.21.15.189:0/109733762 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcd580a9cc0 0x7fcd5814cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:21:30.623 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.606+0000 7fcd5659c700 1 --2- 172.21.15.189:0/109733762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd58140950 0x7fcd580a97d0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7fcd4c006d00 tx=0x7fcd4c019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:30.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.606+0000 7fcd3effd700 1 -- 172.21.15.189:0/109733762 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4c01e420 con 0x7fcd58140950 2020-11-20T12:21:30.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.606+0000 7fcd3effd700 1 -- 172.21.15.189:0/109733762 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd4c01e5a0 con 0x7fcd58140950 2020-11-20T12:21:30.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.606+0000 7fcd5c8a7700 1 -- 172.21.15.189:0/109733762 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd580aa120 con 0x7fcd58140950 2020-11-20T12:21:30.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.606+0000 7fcd3effd700 1 -- 172.21.15.189:0/109733762 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4c01e790 con 0x7fcd58140950 2020-11-20T12:21:30.624 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.606+0000 7fcd5c8a7700 1 -- 172.21.15.189:0/109733762 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd58141600 con 0x7fcd58140950 2020-11-20T12:21:30.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.606+0000 7fcd5c8a7700 1 -- 172.21.15.189:0/109733762 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd580a2480 con 0x7fcd58140950 2020-11-20T12:21:30.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.606+0000 7fcd3effd700 1 -- 172.21.15.189:0/109733762 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fcd4c01ebc0 con 0x7fcd58140950 2020-11-20T12:21:30.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.606+0000 7fcd3effd700 1 --2- 172.21.15.189:0/109733762 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcd340557c0 0x7fcd34057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:30.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.606+0000 7fcd3effd700 1 -- 172.21.15.189:0/109733762 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 7105+0+0 (secure 0 0 0) 0x7fcd4c070cc0 con 0x7fcd58140950 2020-11-20T12:21:30.625 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.606+0000 7fcd56d9d700 1 --2- 172.21.15.189:0/109733762 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcd340557c0 0x7fcd34057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:30.626 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.606+0000 7fcd56d9d700 1 --2- 172.21.15.189:0/109733762 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcd340557c0 0x7fcd34057c80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fcd48000dd0 tx=0x7fcd48009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:30.626 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.610+0000 7fcd3effd700 1 -- 172.21.15.189:0/109733762 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcd4c046750 con 0x7fcd58140950 2020-11-20T12:21:30.825 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.818+0000 7fcd5c8a7700 1 -- 172.21.15.189:0/109733762 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd580446d0 con 0x7fcd58140950 2020-11-20T12:21:30.826 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.818+0000 7fcd3effd700 1 -- 172.21.15.189:0/109733762 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fcd4c049d70 con 0x7fcd58140950 2020-11-20T12:21:30.826 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:21:30.827 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5297,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.47:6833/3025803398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3025803398},{"type":"v1","addr":"172.21.15.47:6833","nonce":3025803398}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:21:00.823729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5746":{"gid":5746,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6835/3538437301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3538437301},{"type":"v1","addr":"172.21.15.189:6835","nonce":3538437301}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:21:30.828 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.822+0000 7fcd5c8a7700 1 -- 172.21.15.189:0/109733762 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcd340557c0 msgr2=0x7fcd34057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:30.829 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.822+0000 7fcd5c8a7700 1 --2- 172.21.15.189:0/109733762 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcd340557c0 0x7fcd34057c80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fcd48000dd0 tx=0x7fcd48009040).stop 2020-11-20T12:21:30.829 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.822+0000 7fcd5c8a7700 1 -- 172.21.15.189:0/109733762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd58140950 msgr2=0x7fcd580a97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:30.829 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.822+0000 7fcd5c8a7700 1 --2- 172.21.15.189:0/109733762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd58140950 0x7fcd580a97d0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7fcd4c006d00 tx=0x7fcd4c019000).stop 2020-11-20T12:21:30.829 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.822+0000 7fcd5c8a7700 1 -- 172.21.15.189:0/109733762 shutdown_connections 2020-11-20T12:21:30.829 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.822+0000 7fcd5c8a7700 1 --2- 172.21.15.189:0/109733762 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fcd340557c0 0x7fcd34057c80 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.822+0000 7fcd5c8a7700 1 --2- 172.21.15.189:0/109733762 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fcd580a9cc0 0x7fcd5814cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.822+0000 7fcd5c8a7700 1 --2- 172.21.15.189:0/109733762 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fcd44003c40 0x7fcd580aae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.822+0000 7fcd5c8a7700 1 --2- 172.21.15.189:0/109733762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd58140950 0x7fcd580a97d0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:30.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.822+0000 7fcd5c8a7700 1 -- 172.21.15.189:0/109733762 >> 172.21.15.189:0/109733762 conn(0x7fcd58062080 msgr2=0x7fcd5813aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:30.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.822+0000 7fcd5c8a7700 1 -- 172.21.15.189:0/109733762 shutdown_connections 2020-11-20T12:21:30.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:30.822+0000 7fcd5c8a7700 1 -- 172.21.15.189:0/109733762 wait complete. 2020-11-20T12:21:30.832 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 66 2020-11-20T12:21:30.843 INFO:tasks.mds_thrash.fs.[None]:kill mds.a (rank=0) 2020-11-20T12:21:30.843 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T12:21:30.844 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:21:30.852 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T12:21:30.853 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:31.034 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.026+0000 7f00af59e700 1 Processor -- start 2020-11-20T12:21:31.034 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.026+0000 7f00af59e700 1 -- start start 2020-11-20T12:21:31.037 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.026+0000 7f00af59e700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f00b013e600 0x7f00b00a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:31.037 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.026+0000 7f00af59e700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f00b00a18f0 0x7f00b0137430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:31.038 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.026+0000 7f00af59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00b00a1e70 con 0x7f00b00aa360 2020-11-20T12:21:31.038 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.026+0000 7f00af59e700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00b00ac450 con 0x7f00b013e600 2020-11-20T12:21:31.038 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.026+0000 7f00af59e700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00b00a2110 con 0x7f00b00a18f0 2020-11-20T12:21:31.038 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.026+0000 7f00aed9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f00b00a18f0 0x7f00b0137430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:31.039 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.026+0000 7f00ae59c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f00b00aa360 0x7f00b013def0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53598/0 (socket says 172.21.15.189:53598) 2020-11-20T12:21:31.039 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.026+0000 7f00aed9d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f00b00a18f0 0x7f00b0137430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51978/0 (socket says 172.21.15.189:51978) 2020-11-20T12:21:31.039 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.026+0000 7f00ae59c700 1 -- 172.21.15.189:0/4210986414 learned_addr learned my addr 172.21.15.189:0/4210986414 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:21:31.039 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.026+0000 7f00add9b700 1 --2- 172.21.15.189:0/4210986414 >> v2:172.21.15.47:3300/0 conn(0x7f00b013e600 0x7f00b00a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:31.039 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.026+0000 7f00aed9d700 1 -- 172.21.15.189:0/4210986414 >> v2:172.21.15.47:3300/0 conn(0x7f00b013e600 msgr2=0x7f00b00a0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:31.040 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.026+0000 7f00aed9d700 1 --2- 172.21.15.189:0/4210986414 >> v2:172.21.15.47:3300/0 conn(0x7f00b013e600 0x7f00b00a0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:31.040 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.026+0000 7f00ad59a700 1 -- 172.21.15.189:0/4210986414 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 932525333 0 0) 0x7f00b00a1e70 con 0x7f00b00aa360 2020-11-20T12:21:31.040 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.026+0000 7f00aed9d700 1 -- 172.21.15.189:0/4210986414 >> v1:172.21.15.189:6789/0 conn(0x7f00b00aa360 legacy=0x7f00b013def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:31.040 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.030+0000 7f00aed9d700 1 -- 172.21.15.189:0/4210986414 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b013da50 con 0x7f00b00a18f0 2020-11-20T12:21:31.040 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.030+0000 7f00aed9d700 1 --2- 172.21.15.189:0/4210986414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00b00a18f0 0x7f00b0137430 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7f00b00ac1e0 tx=0x7f00a80091d0).ready entity=mon.1 client_cookie=f95861a84d8a4cc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:31.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.030+0000 7f00ad59a700 1 -- 172.21.15.189:0/4210986414 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a800ba70 con 0x7f00b00a18f0 2020-11-20T12:21:31.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.030+0000 7f00ad59a700 1 -- 172.21.15.189:0/4210986414 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a800bbd0 con 0x7f00b00a18f0 2020-11-20T12:21:31.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.030+0000 7f00ad59a700 1 -- 172.21.15.189:0/4210986414 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a8011400 con 0x7f00b00a18f0 2020-11-20T12:21:31.041 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.030+0000 7f00af59e700 1 -- 172.21.15.189:0/4210986414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00b00a18f0 msgr2=0x7f00b0137430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:31.042 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.030+0000 7f00af59e700 1 --2- 172.21.15.189:0/4210986414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00b00a18f0 0x7f00b0137430 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7f00b00ac1e0 tx=0x7f00a80091d0).stop 2020-11-20T12:21:31.042 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.030+0000 7f00af59e700 1 -- 172.21.15.189:0/4210986414 shutdown_connections 2020-11-20T12:21:31.042 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.030+0000 7f00af59e700 1 --2- 172.21.15.189:0/4210986414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00b00a18f0 0x7f00b0137430 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:31.042 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.030+0000 7f00af59e700 1 --2- 172.21.15.189:0/4210986414 >> v2:172.21.15.47:3300/0 conn(0x7f00b013e600 0x7f00b00a0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:31.042 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.030+0000 7f00af59e700 1 -- 172.21.15.189:0/4210986414 >> 172.21.15.189:0/4210986414 conn(0x7f00b0060170 msgr2=0x7f00b0060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:31.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.030+0000 7f00af59e700 1 -- 172.21.15.189:0/4210986414 shutdown_connections 2020-11-20T12:21:31.043 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.030+0000 7f00af59e700 1 -- 172.21.15.189:0/4210986414 wait complete. 2020-11-20T12:21:31.045 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.030+0000 7f00af59e700 1 Processor -- start 2020-11-20T12:21:31.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.030+0000 7f00af59e700 1 -- start start 2020-11-20T12:21:31.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.034+0000 7f00af59e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00b00a18f0 0x7f00b013f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:31.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.034+0000 7f00af59e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f00b00aa360 0x7f00b0134440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:31.046 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.034+0000 7f00af59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00b013e600 0x7f00b0149a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:31.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.034+0000 7f00af59e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b01493e0 con 0x7f00b00a18f0 2020-11-20T12:21:31.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.034+0000 7f00af59e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b013ff50 con 0x7f00b013e600 2020-11-20T12:21:31.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.034+0000 7f00af59e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00b01495d0 con 0x7f00b00aa360 2020-11-20T12:21:31.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.034+0000 7f00aed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00b013e600 0x7f00b0149a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:31.047 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.034+0000 7f00aed9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00b013e600 0x7f00b0149a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51984/0 (socket says 172.21.15.189:51984) 2020-11-20T12:21:31.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.034+0000 7f00aed9d700 1 -- 172.21.15.189:0/2189772817 learned_addr learned my addr 172.21.15.189:0/2189772817 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:21:31.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.034+0000 7f00ae59c700 1 --2- 172.21.15.189:0/2189772817 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00b00a18f0 0x7f00b013f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:31.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.034+0000 7f00add9b700 1 --2- 172.21.15.189:0/2189772817 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f00b00aa360 0x7f00b0134440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:31.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.034+0000 7f00aed9d700 1 -- 172.21.15.189:0/2189772817 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f00b00aa360 msgr2=0x7f00b0134440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:31.048 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.034+0000 7f00aed9d700 1 --2- 172.21.15.189:0/2189772817 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f00b00aa360 0x7f00b0134440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:31.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.034+0000 7f00aed9d700 1 -- 172.21.15.189:0/2189772817 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00b00a18f0 msgr2=0x7f00b013f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:31.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.034+0000 7f00aed9d700 1 --2- 172.21.15.189:0/2189772817 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00b00a18f0 0x7f00b013f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:31.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.034+0000 7f00aed9d700 1 -- 172.21.15.189:0/2189772817 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b0141880 con 0x7f00b013e600 2020-11-20T12:21:31.049 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.034+0000 7f00aed9d700 1 --2- 172.21.15.189:0/2189772817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00b013e600 0x7f00b0149a70 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f00a8009210 tx=0x7f00a8016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:31.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.034+0000 7f009e7fc700 1 -- 172.21.15.189:0/2189772817 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a801c5f0 con 0x7f00b013e600 2020-11-20T12:21:31.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.034+0000 7f009e7fc700 1 -- 172.21.15.189:0/2189772817 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a801c750 con 0x7f00b013e600 2020-11-20T12:21:31.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.034+0000 7f009e7fc700 1 -- 172.21.15.189:0/2189772817 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a801cad0 con 0x7f00b013e600 2020-11-20T12:21:31.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.034+0000 7f00af59e700 1 -- 172.21.15.189:0/2189772817 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00b0138570 con 0x7f00b013e600 2020-11-20T12:21:31.050 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.034+0000 7f00af59e700 1 -- 172.21.15.189:0/2189772817 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00b014a180 con 0x7f00b013e600 2020-11-20T12:21:31.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.038+0000 7f00af59e700 1 -- 172.21.15.189:0/2189772817 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00b013fda0 con 0x7f00b013e600 2020-11-20T12:21:31.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.038+0000 7f009e7fc700 1 -- 172.21.15.189:0/2189772817 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f00a8024030 con 0x7f00b013e600 2020-11-20T12:21:31.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.038+0000 7f009e7fc700 1 --2- 172.21.15.189:0/2189772817 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f00840557f0 0x7f0084057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:31.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.038+0000 7f009e7fc700 1 -- 172.21.15.189:0/2189772817 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 7105+0+0 (secure 0 0 0) 0x7f00a8070720 con 0x7f00b013e600 2020-11-20T12:21:31.051 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.038+0000 7f00ae59c700 1 --2- 172.21.15.189:0/2189772817 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f00840557f0 0x7f0084057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:31.052 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.038+0000 7f00ae59c700 1 --2- 172.21.15.189:0/2189772817 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f00840557f0 0x7f0084057cb0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f00a00011a0 tx=0x7f00a0009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:31.052 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.042+0000 7f009e7fc700 1 -- 172.21.15.189:0/2189772817 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f00a80462e0 con 0x7f00b013e600 2020-11-20T12:21:31.247 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.238+0000 7f00af59e700 1 -- 172.21.15.189:0/2189772817 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00b0149790 con 0x7f00b013e600 2020-11-20T12:21:31.249 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.238+0000 7f009e7fc700 1 -- 172.21.15.189:0/2189772817 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f00a8022020 con 0x7f00b013e600 2020-11-20T12:21:31.249 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:21:31.250 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5297,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.47:6833/3025803398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3025803398},{"type":"v1","addr":"172.21.15.47:6833","nonce":3025803398}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:21:00.823729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5746":{"gid":5746,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6835/3538437301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3538437301},{"type":"v1","addr":"172.21.15.189:6835","nonce":3538437301}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:21:31.252 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.242+0000 7f00af59e700 1 -- 172.21.15.189:0/2189772817 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f00840557f0 msgr2=0x7f0084057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:31.252 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.242+0000 7f00af59e700 1 --2- 172.21.15.189:0/2189772817 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f00840557f0 0x7f0084057cb0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f00a00011a0 tx=0x7f00a0009040).stop 2020-11-20T12:21:31.253 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.242+0000 7f00af59e700 1 -- 172.21.15.189:0/2189772817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00b013e600 msgr2=0x7f00b0149a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:31.253 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.242+0000 7f00af59e700 1 --2- 172.21.15.189:0/2189772817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00b013e600 0x7f00b0149a70 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f00a8009210 tx=0x7f00a8016000).stop 2020-11-20T12:21:31.253 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.242+0000 7f00af59e700 1 -- 172.21.15.189:0/2189772817 shutdown_connections 2020-11-20T12:21:31.253 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.242+0000 7f00af59e700 1 --2- 172.21.15.189:0/2189772817 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f00840557f0 0x7f0084057cb0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:31.254 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.242+0000 7f00af59e700 1 --2- 172.21.15.189:0/2189772817 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f00b00a18f0 0x7f00b013f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:31.254 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.242+0000 7f00af59e700 1 --2- 172.21.15.189:0/2189772817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00b013e600 0x7f00b0149a70 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:31.254 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.242+0000 7f00af59e700 1 --2- 172.21.15.189:0/2189772817 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f00b00aa360 0x7f00b0134440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:31.255 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.242+0000 7f00af59e700 1 -- 172.21.15.189:0/2189772817 >> 172.21.15.189:0/2189772817 conn(0x7f00b0060170 msgr2=0x7f00b00ab7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:31.255 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.242+0000 7f00af59e700 1 -- 172.21.15.189:0/2189772817 shutdown_connections 2020-11-20T12:21:31.255 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:31.242+0000 7f00af59e700 1 -- 172.21.15.189:0/2189772817 wait complete. 2020-11-20T12:21:31.255 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 66 2020-11-20T12:21:31.269 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:21:33.270 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:33.419 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.410+0000 7fec269da700 1 Processor -- start 2020-11-20T12:21:33.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.410+0000 7fec269da700 1 -- start start 2020-11-20T12:21:33.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.410+0000 7fec269da700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fec20062220 0x7fec2013a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:33.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.410+0000 7fec269da700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fec20062600 0x7fec20066fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:33.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.410+0000 7fec269da700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec200a9bb0 con 0x7fec20137750 2020-11-20T12:21:33.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.410+0000 7fec269da700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec200674c0 con 0x7fec20062600 2020-11-20T12:21:33.421 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.410+0000 7fec269da700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec2013ab40 con 0x7fec20062220 2020-11-20T12:21:33.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.410+0000 7fec259d8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fec20062220 0x7fec2013a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:33.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.410+0000 7fec261d9700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fec20137750 0x7fec20060a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53622/0 (socket says 172.21.15.189:53622) 2020-11-20T12:21:33.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.410+0000 7fec261d9700 1 -- 172.21.15.189:0/1292625708 learned_addr learned my addr 172.21.15.189:0/1292625708 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:21:33.423 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.410+0000 7fec251d7700 1 --2- 172.21.15.189:0/1292625708 >> v2:172.21.15.47:3300/0 conn(0x7fec20062600 0x7fec20066fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:33.423 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.410+0000 7fec249d6700 1 -- 172.21.15.189:0/1292625708 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1329935786 0 0) 0x7fec10001450 con 0x7fec20137750 2020-11-20T12:21:33.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.410+0000 7fec259d8700 1 -- 172.21.15.189:0/1292625708 >> v2:172.21.15.47:3300/0 conn(0x7fec20062600 msgr2=0x7fec20066fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:33.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.410+0000 7fec259d8700 1 --2- 172.21.15.189:0/1292625708 >> v2:172.21.15.47:3300/0 conn(0x7fec20062600 0x7fec20066fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:33.429 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.410+0000 7fec259d8700 1 -- 172.21.15.189:0/1292625708 >> v1:172.21.15.189:6789/0 conn(0x7fec20137750 legacy=0x7fec20060a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:33.430 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.410+0000 7fec259d8700 1 -- 172.21.15.189:0/1292625708 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec2013ade0 con 0x7fec20062220 2020-11-20T12:21:33.430 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.414+0000 7fec259d8700 1 --2- 172.21.15.189:0/1292625708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec20062220 0x7fec2013a4a0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7fec200ae8f0 tx=0x7fec18006df0).ready entity=mon.1 client_cookie=c51f08c55c78b2a1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:33.430 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.414+0000 7fec249d6700 1 -- 172.21.15.189:0/1292625708 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec18009840 con 0x7fec20062220 2020-11-20T12:21:33.431 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.414+0000 7fec249d6700 1 -- 172.21.15.189:0/1292625708 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec180099a0 con 0x7fec20062220 2020-11-20T12:21:33.431 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.414+0000 7fec249d6700 1 -- 172.21.15.189:0/1292625708 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec18009d20 con 0x7fec20062220 2020-11-20T12:21:33.431 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.414+0000 7fec269da700 1 -- 172.21.15.189:0/1292625708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec20062220 msgr2=0x7fec2013a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:33.432 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.414+0000 7fec269da700 1 --2- 172.21.15.189:0/1292625708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec20062220 0x7fec2013a4a0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7fec200ae8f0 tx=0x7fec18006df0).stop 2020-11-20T12:21:33.432 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.414+0000 7fec269da700 1 -- 172.21.15.189:0/1292625708 shutdown_connections 2020-11-20T12:21:33.433 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.414+0000 7fec269da700 1 --2- 172.21.15.189:0/1292625708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec20062220 0x7fec2013a4a0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:33.433 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.414+0000 7fec269da700 1 --2- 172.21.15.189:0/1292625708 >> v2:172.21.15.47:3300/0 conn(0x7fec20062600 0x7fec20066fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:33.433 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.414+0000 7fec269da700 1 -- 172.21.15.189:0/1292625708 >> 172.21.15.189:0/1292625708 conn(0x7fec20065fe0 msgr2=0x7fec20061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:33.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.414+0000 7fec269da700 1 -- 172.21.15.189:0/1292625708 shutdown_connections 2020-11-20T12:21:33.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.414+0000 7fec269da700 1 -- 172.21.15.189:0/1292625708 wait complete. 2020-11-20T12:21:33.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.414+0000 7fec269da700 1 Processor -- start 2020-11-20T12:21:33.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.414+0000 7fec269da700 1 -- start start 2020-11-20T12:21:33.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.414+0000 7fec269da700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec20062600 0x7fec200a2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:33.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.414+0000 7fec269da700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec20137750 0x7fec20132930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:33.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.414+0000 7fec269da700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fec2013dfb0 0x7fec20132e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:33.436 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.414+0000 7fec269da700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec20133310 con 0x7fec20137750 2020-11-20T12:21:33.436 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.414+0000 7fec269da700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec200a4490 con 0x7fec20062600 2020-11-20T12:21:33.436 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.414+0000 7fec269da700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec2013e860 con 0x7fec2013dfb0 2020-11-20T12:21:33.437 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.414+0000 7fec259d8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec20062600 0x7fec200a2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:33.437 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.414+0000 7fec259d8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec20062600 0x7fec200a2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52002/0 (socket says 172.21.15.189:52002) 2020-11-20T12:21:33.437 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.414+0000 7fec259d8700 1 -- 172.21.15.189:0/508616712 learned_addr learned my addr 172.21.15.189:0/508616712 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:21:33.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.414+0000 7fec251d7700 1 --2- 172.21.15.189:0/508616712 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec20137750 0x7fec20132930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:33.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.414+0000 7fec261d9700 1 --2- 172.21.15.189:0/508616712 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fec2013dfb0 0x7fec20132e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:33.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.418+0000 7fec259d8700 1 -- 172.21.15.189:0/508616712 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fec2013dfb0 msgr2=0x7fec20132e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:33.439 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.418+0000 7fec259d8700 1 --2- 172.21.15.189:0/508616712 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fec2013dfb0 0x7fec20132e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:33.439 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.418+0000 7fec259d8700 1 -- 172.21.15.189:0/508616712 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec20137750 msgr2=0x7fec20132930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:33.439 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.418+0000 7fec259d8700 1 --2- 172.21.15.189:0/508616712 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec20137750 0x7fec20132930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:33.439 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.418+0000 7fec259d8700 1 -- 172.21.15.189:0/508616712 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec200a3d50 con 0x7fec20062600 2020-11-20T12:21:33.440 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.418+0000 7fec259d8700 1 --2- 172.21.15.189:0/508616712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec20062600 0x7fec200a2340 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7fec18006e30 tx=0x7fec18015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:33.440 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.418+0000 7fec1dffb700 1 -- 172.21.15.189:0/508616712 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec18011e00 con 0x7fec20062600 2020-11-20T12:21:33.440 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.418+0000 7fec1dffb700 1 -- 172.21.15.189:0/508616712 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1801e460 con 0x7fec20062600 2020-11-20T12:21:33.441 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.418+0000 7fec269da700 1 -- 172.21.15.189:0/508616712 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec200a3ff0 con 0x7fec20062600 2020-11-20T12:21:33.441 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.418+0000 7fec1dffb700 1 -- 172.21.15.189:0/508616712 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1801e7e0 con 0x7fec20062600 2020-11-20T12:21:33.441 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.418+0000 7fec269da700 1 -- 172.21.15.189:0/508616712 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec2013e430 con 0x7fec20062600 2020-11-20T12:21:33.442 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.418+0000 7fec269da700 1 -- 172.21.15.189:0/508616712 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec200663c0 con 0x7fec20062600 2020-11-20T12:21:33.442 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.418+0000 7fec1dffb700 1 -- 172.21.15.189:0/508616712 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fec18025030 con 0x7fec20062600 2020-11-20T12:21:33.442 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.418+0000 7fec1dffb700 1 --2- 172.21.15.189:0/508616712 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7febfc0557f0 0x7febfc057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:33.443 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.418+0000 7fec1dffb700 1 -- 172.21.15.189:0/508616712 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 7105+0+0 (secure 0 0 0) 0x7fec18070220 con 0x7fec20062600 2020-11-20T12:21:33.443 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.418+0000 7fec251d7700 1 --2- 172.21.15.189:0/508616712 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7febfc0557f0 0x7febfc057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:33.443 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.422+0000 7fec251d7700 1 --2- 172.21.15.189:0/508616712 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7febfc0557f0 0x7febfc057cb0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fec200a3740 tx=0x7fec08006df0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:33.444 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.426+0000 7fec1dffb700 1 -- 172.21.15.189:0/508616712 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fec18045c50 con 0x7fec20062600 2020-11-20T12:21:33.640 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.634+0000 7fec269da700 1 -- 172.21.15.189:0/508616712 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec20065ca0 con 0x7fec20062600 2020-11-20T12:21:33.641 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.634+0000 7fec1dffb700 1 -- 172.21.15.189:0/508616712 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fec18023070 con 0x7fec20062600 2020-11-20T12:21:33.642 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:21:33.642 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5297,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.47:6833/3025803398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3025803398},{"type":"v1","addr":"172.21.15.47:6833","nonce":3025803398}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:21:00.823729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5746":{"gid":5746,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6835/3538437301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3538437301},{"type":"v1","addr":"172.21.15.189:6835","nonce":3538437301}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:21:33.643 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.634+0000 7fec269da700 1 -- 172.21.15.189:0/508616712 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7febfc0557f0 msgr2=0x7febfc057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:33.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.634+0000 7fec269da700 1 --2- 172.21.15.189:0/508616712 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7febfc0557f0 0x7febfc057cb0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fec200a3740 tx=0x7fec08006df0).stop 2020-11-20T12:21:33.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.634+0000 7fec269da700 1 -- 172.21.15.189:0/508616712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec20062600 msgr2=0x7fec200a2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:33.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.634+0000 7fec269da700 1 --2- 172.21.15.189:0/508616712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec20062600 0x7fec200a2340 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7fec18006e30 tx=0x7fec18015000).stop 2020-11-20T12:21:33.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.634+0000 7fec269da700 1 -- 172.21.15.189:0/508616712 shutdown_connections 2020-11-20T12:21:33.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.634+0000 7fec269da700 1 --2- 172.21.15.189:0/508616712 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fec2013dfb0 0x7fec20132e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:33.645 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.634+0000 7fec269da700 1 --2- 172.21.15.189:0/508616712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec20062600 0x7fec200a2340 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:33.646 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.634+0000 7fec269da700 1 --2- 172.21.15.189:0/508616712 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7febfc0557f0 0x7febfc057cb0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:33.646 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.634+0000 7fec269da700 1 --2- 172.21.15.189:0/508616712 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fec20137750 0x7fec20132930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:33.646 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.634+0000 7fec269da700 1 -- 172.21.15.189:0/508616712 >> 172.21.15.189:0/508616712 conn(0x7fec20065fe0 msgr2=0x7fec20136e40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:33.646 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.634+0000 7fec269da700 1 -- 172.21.15.189:0/508616712 shutdown_connections 2020-11-20T12:21:33.647 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:33.634+0000 7fec269da700 1 -- 172.21.15.189:0/508616712 wait complete. 2020-11-20T12:21:33.647 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 66 2020-11-20T12:21:33.662 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:21:35.663 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:35.796 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.790+0000 7efff9ecd700 1 Processor -- start 2020-11-20T12:21:35.797 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.790+0000 7efff9ecd700 1 -- start start 2020-11-20T12:21:35.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.790+0000 7efff9ecd700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efff413ab80 0x7efff413e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:35.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.790+0000 7efff9ecd700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efff413e6c0 0x7efff413cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:35.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.790+0000 7efff9ecd700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efff413b300 con 0x7efff4066de0 2020-11-20T12:21:35.800 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.790+0000 7efff9ecd700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efff413dd30 con 0x7efff413ab80 2020-11-20T12:21:35.801 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.790+0000 7efff9ecd700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efff413aff0 con 0x7efff413e6c0 2020-11-20T12:21:35.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.790+0000 7efff3fff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efff413e6c0 0x7efff413cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:35.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.790+0000 7efff37fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7efff4066de0 0x7efff40671c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.189:48298/0 (socket says 172.21.15.189:48298) 2020-11-20T12:21:35.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.790+0000 7efff37fe700 1 -- 172.21.15.189:0/106675370 learned_addr learned my addr 172.21.15.189:0/106675370 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:21:35.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.790+0000 7efff2ffd700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7efff413ab80 0x7efff413e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:35.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.790+0000 7efff3fff700 1 -- 172.21.15.189:0/106675370 >> v2:172.21.15.47:3300/0 conn(0x7efff413ab80 msgr2=0x7efff413e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:35.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.790+0000 7efff3fff700 1 --2- 172.21.15.189:0/106675370 >> v2:172.21.15.47:3300/0 conn(0x7efff413ab80 0x7efff413e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:35.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.790+0000 7efff3fff700 1 -- 172.21.15.189:0/106675370 >> v1:172.21.15.47:6789/0 conn(0x7efff4066de0 legacy=0x7efff40671c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:35.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.790+0000 7efff3fff700 1 -- 172.21.15.189:0/106675370 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff413d390 con 0x7efff413e6c0 2020-11-20T12:21:35.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.790+0000 7efff3fff700 1 --2- 172.21.15.189:0/106675370 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efff413e6c0 0x7efff413cea0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7effec009cc0 tx=0x7effec0091d0).ready entity=mon.2 client_cookie=c1593a488ca2919 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:35.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.790+0000 7efff27fc700 1 -- 172.21.15.189:0/106675370 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effec00bb20 con 0x7efff413e6c0 2020-11-20T12:21:35.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.790+0000 7efff27fc700 1 -- 172.21.15.189:0/106675370 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effec00bc80 con 0x7efff413e6c0 2020-11-20T12:21:35.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.790+0000 7efff27fc700 1 -- 172.21.15.189:0/106675370 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effec011400 con 0x7efff413e6c0 2020-11-20T12:21:35.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.790+0000 7efff9ecd700 1 -- 172.21.15.189:0/106675370 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efff413e6c0 msgr2=0x7efff413cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:35.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.790+0000 7efff9ecd700 1 --2- 172.21.15.189:0/106675370 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efff413e6c0 0x7efff413cea0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7effec009cc0 tx=0x7effec0091d0).stop 2020-11-20T12:21:35.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff9ecd700 1 -- 172.21.15.189:0/106675370 shutdown_connections 2020-11-20T12:21:35.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff9ecd700 1 --2- 172.21.15.189:0/106675370 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efff413e6c0 0x7efff413cea0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:35.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff9ecd700 1 --2- 172.21.15.189:0/106675370 >> v2:172.21.15.47:3300/0 conn(0x7efff413ab80 0x7efff413e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:35.806 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff9ecd700 1 -- 172.21.15.189:0/106675370 >> 172.21.15.189:0/106675370 conn(0x7efff4061f60 msgr2=0x7efff4062340 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:35.806 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff9ecd700 1 -- 172.21.15.189:0/106675370 shutdown_connections 2020-11-20T12:21:35.806 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff9ecd700 1 -- 172.21.15.189:0/106675370 wait complete. 2020-11-20T12:21:35.808 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff9ecd700 1 Processor -- start 2020-11-20T12:21:35.809 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff9ecd700 1 -- start start 2020-11-20T12:21:35.809 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff9ecd700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efff4066de0 0x7efff40acb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:35.809 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff9ecd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efff413ab80 0x7efff40a2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:35.809 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff9ecd700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efff413e6c0 0x7efff40a2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:35.810 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff9ecd700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff40a3610 con 0x7efff413e6c0 2020-11-20T12:21:35.810 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff9ecd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff40a1d60 con 0x7efff413ab80 2020-11-20T12:21:35.810 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff9ecd700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efff4149640 con 0x7efff4066de0 2020-11-20T12:21:35.810 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff2ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efff413ab80 0x7efff40a2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:35.810 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff2ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efff413ab80 0x7efff40a2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52022/0 (socket says 172.21.15.189:52022) 2020-11-20T12:21:35.811 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff2ffd700 1 -- 172.21.15.189:0/1161998619 learned_addr learned my addr 172.21.15.189:0/1161998619 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:21:35.811 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff3fff700 1 --2- 172.21.15.189:0/1161998619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efff413e6c0 0x7efff40a2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:35.811 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff37fe700 1 --2- 172.21.15.189:0/1161998619 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efff4066de0 0x7efff40acb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:35.811 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff3fff700 1 -- 172.21.15.189:0/1161998619 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efff4066de0 msgr2=0x7efff40acb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:35.811 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff3fff700 1 --2- 172.21.15.189:0/1161998619 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efff4066de0 0x7efff40acb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:35.812 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff3fff700 1 -- 172.21.15.189:0/1161998619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efff413ab80 msgr2=0x7efff40a2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:35.812 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff3fff700 1 --2- 172.21.15.189:0/1161998619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efff413ab80 0x7efff40a2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:35.812 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff3fff700 1 -- 172.21.15.189:0/1161998619 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff40a5350 con 0x7efff413e6c0 2020-11-20T12:21:35.812 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff3fff700 1 --2- 172.21.15.189:0/1161998619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efff413e6c0 0x7efff40a2200 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7effec009cc0 tx=0x7effec016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:35.812 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7effd37fe700 1 -- 172.21.15.189:0/1161998619 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effec010780 con 0x7efff413e6c0 2020-11-20T12:21:35.813 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7effd37fe700 1 -- 172.21.15.189:0/1161998619 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effec0108e0 con 0x7efff413e6c0 2020-11-20T12:21:35.813 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff9ecd700 1 -- 172.21.15.189:0/1161998619 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efff40a5540 con 0x7efff413e6c0 2020-11-20T12:21:35.813 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7effd37fe700 1 -- 172.21.15.189:0/1161998619 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effec010c60 con 0x7efff413e6c0 2020-11-20T12:21:35.813 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.794+0000 7efff9ecd700 1 -- 172.21.15.189:0/1161998619 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efff40a26f0 con 0x7efff413e6c0 2020-11-20T12:21:35.813 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.798+0000 7efff9ecd700 1 -- 172.21.15.189:0/1161998619 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efff40a1f20 con 0x7efff413e6c0 2020-11-20T12:21:35.814 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.798+0000 7effd37fe700 1 -- 172.21.15.189:0/1161998619 <== mon.0 v2:172.21.15.47:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7effec017070 con 0x7efff413e6c0 2020-11-20T12:21:35.814 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.798+0000 7effd37fe700 1 --2- 172.21.15.189:0/1161998619 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7effcc0557f0 0x7effcc057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:35.814 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.798+0000 7effd37fe700 1 -- 172.21.15.189:0/1161998619 <== mon.0 v2:172.21.15.47:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 7105+0+0 (secure 0 0 0) 0x7effec071a10 con 0x7efff413e6c0 2020-11-20T12:21:35.814 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.798+0000 7efff37fe700 1 --2- 172.21.15.189:0/1161998619 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7effcc0557f0 0x7effcc057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:35.814 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.798+0000 7efff37fe700 1 --2- 172.21.15.189:0/1161998619 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7effcc0557f0 0x7effcc057cb0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7effe40045d0 tx=0x7effe4007b80).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:35.817 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:35.810+0000 7effd37fe700 1 -- 172.21.15.189:0/1161998619 <== mon.0 v2:172.21.15.47:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7effec047480 con 0x7efff413e6c0 2020-11-20T12:21:36.019 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:36.010+0000 7efff9ecd700 1 -- 172.21.15.189:0/1161998619 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efff40a3410 con 0x7efff413e6c0 2020-11-20T12:21:36.020 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:36.014+0000 7effd37fe700 1 -- 172.21.15.189:0/1161998619 <== mon.0 v2:172.21.15.47:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2140 (secure 0 0 0) 0x7effec015070 con 0x7efff413e6c0 2020-11-20T12:21:36.020 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:21:36.020 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5297,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.47:6833/3025803398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3025803398},{"type":"v1","addr":"172.21.15.47:6833","nonce":3025803398}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:21:00.823729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5746":{"gid":5746,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6835/3538437301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3538437301},{"type":"v1","addr":"172.21.15.189:6835","nonce":3538437301}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:21:36.022 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:36.014+0000 7efff9ecd700 1 -- 172.21.15.189:0/1161998619 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7effcc0557f0 msgr2=0x7effcc057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:36.022 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:36.014+0000 7efff9ecd700 1 --2- 172.21.15.189:0/1161998619 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7effcc0557f0 0x7effcc057cb0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7effe40045d0 tx=0x7effe4007b80).stop 2020-11-20T12:21:36.023 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:36.014+0000 7efff9ecd700 1 -- 172.21.15.189:0/1161998619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efff413e6c0 msgr2=0x7efff40a2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:36.023 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:36.014+0000 7efff9ecd700 1 --2- 172.21.15.189:0/1161998619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efff413e6c0 0x7efff40a2200 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7effec009cc0 tx=0x7effec016000).stop 2020-11-20T12:21:36.023 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:36.014+0000 7efff9ecd700 1 -- 172.21.15.189:0/1161998619 shutdown_connections 2020-11-20T12:21:36.023 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:36.014+0000 7efff9ecd700 1 --2- 172.21.15.189:0/1161998619 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7effcc0557f0 0x7effcc057cb0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:36.023 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:36.014+0000 7efff9ecd700 1 --2- 172.21.15.189:0/1161998619 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7efff413e6c0 0x7efff40a2200 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:36.024 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:36.014+0000 7efff9ecd700 1 --2- 172.21.15.189:0/1161998619 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7efff4066de0 0x7efff40acb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:36.024 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:36.014+0000 7efff9ecd700 1 --2- 172.21.15.189:0/1161998619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efff413ab80 0x7efff40a2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:36.024 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:36.014+0000 7efff9ecd700 1 -- 172.21.15.189:0/1161998619 >> 172.21.15.189:0/1161998619 conn(0x7efff4061f60 msgr2=0x7efff40b0570 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:36.024 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:36.014+0000 7efff9ecd700 1 -- 172.21.15.189:0/1161998619 shutdown_connections 2020-11-20T12:21:36.025 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:36.014+0000 7efff9ecd700 1 -- 172.21.15.189:0/1161998619 wait complete. 2020-11-20T12:21:36.025 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 66 2020-11-20T12:21:36.041 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:21:38.044 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:38.184 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc7160700 1 Processor -- start 2020-11-20T12:21:38.184 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc7160700 1 -- start start 2020-11-20T12:21:38.185 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc7160700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2cc00a9800 0x7f2cc013ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:38.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc7160700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2cc013f260 0x7f2cc013b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:38.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc7160700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2cc013be50 0x7f2cc0060170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:38.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc7160700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc0065ec0 con 0x7f2cc00a9800 2020-11-20T12:21:38.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc7160700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc00676a0 con 0x7f2cc013be50 2020-11-20T12:21:38.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc7160700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc0067c50 con 0x7f2cc013f260 2020-11-20T12:21:38.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc56fd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2cc013be50 0x7f2cc0060170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:38.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc4efc700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f2cc00a9800 0x7f2cc013ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:38.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc56fd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2cc013be50 0x7f2cc0060170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52032/0 (socket says 172.21.15.189:52032) 2020-11-20T12:21:38.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc56fd700 1 -- 172.21.15.189:0/2056950243 learned_addr learned my addr 172.21.15.189:0/2056950243 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:21:38.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cbffff700 1 --2- 172.21.15.189:0/2056950243 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2cc013f260 0x7f2cc013b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:38.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc56fd700 1 -- 172.21.15.189:0/2056950243 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2cc013f260 msgr2=0x7f2cc013b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:38.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc56fd700 1 --2- 172.21.15.189:0/2056950243 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2cc013f260 0x7f2cc013b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:38.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc56fd700 1 -- 172.21.15.189:0/2056950243 >> v2:172.21.15.47:3300/0 conn(0x7f2cc00a9800 msgr2=0x7f2cc013ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:38.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc56fd700 1 --2- 172.21.15.189:0/2056950243 >> v2:172.21.15.47:3300/0 conn(0x7f2cc00a9800 0x7f2cc013ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:38.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc56fd700 1 -- 172.21.15.189:0/2056950243 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc0067160 con 0x7f2cc013be50 2020-11-20T12:21:38.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc56fd700 1 --2- 172.21.15.189:0/2056950243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cc013be50 0x7f2cc0060170 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7f2cc00a0820 tx=0x7f2cb8009290).ready entity=mon.1 client_cookie=2da722cc391f2518 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:38.190 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cbf7fe700 1 -- 172.21.15.189:0/2056950243 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb800bba0 con 0x7f2cc013be50 2020-11-20T12:21:38.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cbf7fe700 1 -- 172.21.15.189:0/2056950243 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb800bd00 con 0x7f2cc013be50 2020-11-20T12:21:38.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cbf7fe700 1 -- 172.21.15.189:0/2056950243 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb8011610 con 0x7f2cc013be50 2020-11-20T12:21:38.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc7160700 1 -- 172.21.15.189:0/2056950243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cc013be50 msgr2=0x7f2cc0060170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:38.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc7160700 1 --2- 172.21.15.189:0/2056950243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cc013be50 0x7f2cc0060170 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7f2cc00a0820 tx=0x7f2cb8009290).stop 2020-11-20T12:21:38.191 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc7160700 1 -- 172.21.15.189:0/2056950243 shutdown_connections 2020-11-20T12:21:38.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc7160700 1 --2- 172.21.15.189:0/2056950243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cc013be50 0x7f2cc0060170 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:38.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc7160700 1 --2- 172.21.15.189:0/2056950243 >> v2:172.21.15.47:3300/0 conn(0x7f2cc00a9800 0x7f2cc013ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:38.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc7160700 1 --2- 172.21.15.189:0/2056950243 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2cc013f260 0x7f2cc013b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:38.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc7160700 1 -- 172.21.15.189:0/2056950243 >> 172.21.15.189:0/2056950243 conn(0x7f2cc0061f60 msgr2=0x7f2cc0062370 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:38.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc7160700 1 -- 172.21.15.189:0/2056950243 shutdown_connections 2020-11-20T12:21:38.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.178+0000 7f2cc7160700 1 -- 172.21.15.189:0/2056950243 wait complete. 2020-11-20T12:21:38.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.182+0000 7f2cc7160700 1 Processor -- start 2020-11-20T12:21:38.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.182+0000 7f2cc7160700 1 -- start start 2020-11-20T12:21:38.195 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.182+0000 7f2cc7160700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cc00a9800 0x7f2cc009d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:38.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.186+0000 7f2cc7160700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2cc013be50 0x7f2cc00a0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:38.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.186+0000 7f2cc7160700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2cc013f260 0x7f2cc009caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:38.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.186+0000 7f2cc7160700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc0150e00 con 0x7f2cc013be50 2020-11-20T12:21:38.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.186+0000 7f2cc7160700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc009fae0 con 0x7f2cc00a9800 2020-11-20T12:21:38.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.186+0000 7f2cc7160700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc009fd80 con 0x7f2cc013f260 2020-11-20T12:21:38.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.186+0000 7f2cc4efc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cc00a9800 0x7f2cc009d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:38.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.186+0000 7f2cc4efc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cc00a9800 0x7f2cc009d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52034/0 (socket says 172.21.15.189:52034) 2020-11-20T12:21:38.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.186+0000 7f2cc4efc700 1 -- 172.21.15.189:0/1218772049 learned_addr learned my addr 172.21.15.189:0/1218772049 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:21:38.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.186+0000 7f2cbffff700 1 --2- 172.21.15.189:0/1218772049 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2cc013be50 0x7f2cc00a0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:38.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.186+0000 7f2cc56fd700 1 --2- 172.21.15.189:0/1218772049 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2cc013f260 0x7f2cc009caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:38.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.186+0000 7f2cc4efc700 1 -- 172.21.15.189:0/1218772049 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2cc013f260 msgr2=0x7f2cc009caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:38.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.186+0000 7f2cc4efc700 1 --2- 172.21.15.189:0/1218772049 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2cc013f260 0x7f2cc009caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:38.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.186+0000 7f2cc4efc700 1 -- 172.21.15.189:0/1218772049 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2cc013be50 msgr2=0x7f2cc00a0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:38.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.186+0000 7f2cc4efc700 1 --2- 172.21.15.189:0/1218772049 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2cc013be50 0x7f2cc00a0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:38.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.186+0000 7f2cc4efc700 1 -- 172.21.15.189:0/1218772049 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc009c600 con 0x7f2cc00a9800 2020-11-20T12:21:38.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.186+0000 7f2cc4efc700 1 --2- 172.21.15.189:0/1218772049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cc00a9800 0x7f2cc009d610 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7f2cb400eba0 tx=0x7f2cb4006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:38.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.186+0000 7f2cbcff9700 1 -- 172.21.15.189:0/1218772049 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb400c8e0 con 0x7f2cc00a9800 2020-11-20T12:21:38.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.186+0000 7f2cbcff9700 1 -- 172.21.15.189:0/1218772049 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb400ca40 con 0x7f2cc00a9800 2020-11-20T12:21:38.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.186+0000 7f2cc7160700 1 -- 172.21.15.189:0/1218772049 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc00a0570 con 0x7f2cc00a9800 2020-11-20T12:21:38.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.186+0000 7f2cbcff9700 1 -- 172.21.15.189:0/1218772049 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb400cdc0 con 0x7f2cc00a9800 2020-11-20T12:21:38.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.186+0000 7f2cc7160700 1 -- 172.21.15.189:0/1218772049 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc009f860 con 0x7f2cc00a9800 2020-11-20T12:21:38.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.186+0000 7f2cbcff9700 1 -- 172.21.15.189:0/1218772049 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f2cb4012600 con 0x7f2cc00a9800 2020-11-20T12:21:38.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.186+0000 7f2cc7160700 1 -- 172.21.15.189:0/1218772049 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc0067340 con 0x7f2cc00a9800 2020-11-20T12:21:38.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.186+0000 7f2cbcff9700 1 --2- 172.21.15.189:0/1218772049 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2c980557c0 0x7f2c98057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:38.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.186+0000 7f2cbcff9700 1 -- 172.21.15.189:0/1218772049 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 7105+0+0 (secure 0 0 0) 0x7f2cb406edb0 con 0x7f2cc00a9800 2020-11-20T12:21:38.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.190+0000 7f2cbffff700 1 --2- 172.21.15.189:0/1218772049 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2c980557c0 0x7f2c98057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:38.202 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.190+0000 7f2cbffff700 1 --2- 172.21.15.189:0/1218772049 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2c980557c0 0x7f2c98057c80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f2cc013ba70 tx=0x7f2cac006df0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:38.202 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.194+0000 7f2cbcff9700 1 -- 172.21.15.189:0/1218772049 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2cb4044840 con 0x7f2cc00a9800 2020-11-20T12:21:38.403 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.394+0000 7f2cc7160700 1 -- 172.21.15.189:0/1218772049 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2cc0044f60 con 0x7f2cc00a9800 2020-11-20T12:21:38.404 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.398+0000 7f2cbcff9700 1 -- 172.21.15.189:0/1218772049 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f2cb406e070 con 0x7f2cc00a9800 2020-11-20T12:21:38.404 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:21:38.404 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5297,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.47:6833/3025803398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3025803398},{"type":"v1","addr":"172.21.15.47:6833","nonce":3025803398}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:21:00.823729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5746":{"gid":5746,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6835/3538437301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3538437301},{"type":"v1","addr":"172.21.15.189:6835","nonce":3538437301}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:21:38.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.398+0000 7f2cc7160700 1 -- 172.21.15.189:0/1218772049 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2c980557c0 msgr2=0x7f2c98057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:38.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.398+0000 7f2cc7160700 1 --2- 172.21.15.189:0/1218772049 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2c980557c0 0x7f2c98057c80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f2cc013ba70 tx=0x7f2cac006df0).stop 2020-11-20T12:21:38.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.398+0000 7f2cc7160700 1 -- 172.21.15.189:0/1218772049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cc00a9800 msgr2=0x7f2cc009d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:38.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.398+0000 7f2cc7160700 1 --2- 172.21.15.189:0/1218772049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cc00a9800 0x7f2cc009d610 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7f2cb400eba0 tx=0x7f2cb4006df0).stop 2020-11-20T12:21:38.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.398+0000 7f2cc7160700 1 -- 172.21.15.189:0/1218772049 shutdown_connections 2020-11-20T12:21:38.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.398+0000 7f2cc7160700 1 --2- 172.21.15.189:0/1218772049 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2cc013f260 0x7f2cc009caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:38.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.398+0000 7f2cc7160700 1 --2- 172.21.15.189:0/1218772049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cc00a9800 0x7f2cc009d610 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:38.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.398+0000 7f2cc7160700 1 --2- 172.21.15.189:0/1218772049 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2c980557c0 0x7f2c98057c80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:38.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.398+0000 7f2cc7160700 1 --2- 172.21.15.189:0/1218772049 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2cc013be50 0x7f2cc00a0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:38.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.398+0000 7f2cc7160700 1 -- 172.21.15.189:0/1218772049 >> 172.21.15.189:0/1218772049 conn(0x7f2cc0061f60 msgr2=0x7f2cc00608c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:38.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.398+0000 7f2cc7160700 1 -- 172.21.15.189:0/1218772049 shutdown_connections 2020-11-20T12:21:38.409 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:38.398+0000 7f2cc7160700 1 -- 172.21.15.189:0/1218772049 wait complete. 2020-11-20T12:21:38.409 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 66 2020-11-20T12:21:38.418 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:21:40.420 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:40.557 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.550+0000 7f88cd6a1700 1 Processor -- start 2020-11-20T12:21:40.558 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.550+0000 7f88cd6a1700 1 -- start start 2020-11-20T12:21:40.559 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.550+0000 7f88cd6a1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f88c80a9950 0x7f88c8065fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:40.559 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.550+0000 7f88cd6a1700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f88c80664d0 0x7f88c8134330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:40.560 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.550+0000 7f88cd6a1700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c80a8820 con 0x7f88c80a43c0 2020-11-20T12:21:40.560 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.550+0000 7f88cd6a1700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c8066ec0 con 0x7f88c80664d0 2020-11-20T12:21:40.560 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.550+0000 7f88cd6a1700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c80a8ac0 con 0x7f88c80a9950 2020-11-20T12:21:40.560 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.550+0000 7f88c6ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f88c80a9950 0x7f88c8065fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:40.560 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.550+0000 7f88c6ffd700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f88c80a9950 0x7f88c8065fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52046/0 (socket says 172.21.15.189:52046) 2020-11-20T12:21:40.561 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.550+0000 7f88c6ffd700 1 -- 172.21.15.189:0/2312644502 learned_addr learned my addr 172.21.15.189:0/2312644502 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:21:40.561 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.550+0000 7f88be7fc700 1 --2- 172.21.15.189:0/2312644502 >> v2:172.21.15.47:3300/0 conn(0x7f88c80664d0 0x7f88c8134330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:40.561 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.550+0000 7f88c6ffd700 1 -- 172.21.15.189:0/2312644502 >> v2:172.21.15.47:3300/0 conn(0x7f88c80664d0 msgr2=0x7f88c8134330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:40.561 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.550+0000 7f88c6ffd700 1 --2- 172.21.15.189:0/2312644502 >> v2:172.21.15.47:3300/0 conn(0x7f88c80664d0 0x7f88c8134330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:40.562 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.554+0000 7f88c6ffd700 1 -- 172.21.15.189:0/2312644502 >> v1:172.21.15.47:6789/0 conn(0x7f88c80a43c0 legacy=0x7f88c8134820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:40.562 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.554+0000 7f88c6ffd700 1 -- 172.21.15.189:0/2312644502 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c80a8d60 con 0x7f88c80a9950 2020-11-20T12:21:40.562 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.554+0000 7f88c6ffd700 1 --2- 172.21.15.189:0/2312644502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88c80a9950 0x7f88c8065fe0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7f88c80a9720 tx=0x7f88b8006eb0).ready entity=mon.1 client_cookie=569627325d778b83 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:40.563 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.554+0000 7f88c67fc700 1 -- 172.21.15.189:0/2312644502 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b8009810 con 0x7f88c80a9950 2020-11-20T12:21:40.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.554+0000 7f88c67fc700 1 -- 172.21.15.189:0/2312644502 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b8009970 con 0x7f88c80a9950 2020-11-20T12:21:40.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.554+0000 7f88c67fc700 1 -- 172.21.15.189:0/2312644502 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b8009cf0 con 0x7f88c80a9950 2020-11-20T12:21:40.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.554+0000 7f88cd6a1700 1 -- 172.21.15.189:0/2312644502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88c80a9950 msgr2=0x7f88c8065fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:40.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.554+0000 7f88cd6a1700 1 --2- 172.21.15.189:0/2312644502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88c80a9950 0x7f88c8065fe0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7f88c80a9720 tx=0x7f88b8006eb0).stop 2020-11-20T12:21:40.564 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.554+0000 7f88cd6a1700 1 -- 172.21.15.189:0/2312644502 shutdown_connections 2020-11-20T12:21:40.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.554+0000 7f88cd6a1700 1 --2- 172.21.15.189:0/2312644502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88c80a9950 0x7f88c8065fe0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:40.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.554+0000 7f88cd6a1700 1 --2- 172.21.15.189:0/2312644502 >> v2:172.21.15.47:3300/0 conn(0x7f88c80664d0 0x7f88c8134330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:40.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.554+0000 7f88cd6a1700 1 -- 172.21.15.189:0/2312644502 >> 172.21.15.189:0/2312644502 conn(0x7f88c8061f60 msgr2=0x7f88c8062370 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:40.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.554+0000 7f88cd6a1700 1 -- 172.21.15.189:0/2312644502 shutdown_connections 2020-11-20T12:21:40.565 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.554+0000 7f88cd6a1700 1 -- 172.21.15.189:0/2312644502 wait complete. 2020-11-20T12:21:40.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.558+0000 7f88cd6a1700 1 Processor -- start 2020-11-20T12:21:40.566 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.558+0000 7f88cd6a1700 1 -- start start 2020-11-20T12:21:40.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.558+0000 7f88cd6a1700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f88c80664d0 0x7f88c80ac020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:40.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.558+0000 7f88cd6a1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88c80a43c0 0x7f88c814d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:40.568 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.558+0000 7f88cd6a1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88c80a9950 0x7f88c81360a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:40.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.558+0000 7f88cd6a1700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c814cda0 con 0x7f88c80a43c0 2020-11-20T12:21:40.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.558+0000 7f88cd6a1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c814cfd0 con 0x7f88c80a9950 2020-11-20T12:21:40.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.558+0000 7f88cd6a1700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c814d270 con 0x7f88c80664d0 2020-11-20T12:21:40.569 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.558+0000 7f88c77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88c80a9950 0x7f88c81360a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:40.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.558+0000 7f88c77fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88c80a9950 0x7f88c81360a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52050/0 (socket says 172.21.15.189:52050) 2020-11-20T12:21:40.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.558+0000 7f88c77fe700 1 -- 172.21.15.189:0/1621057447 learned_addr learned my addr 172.21.15.189:0/1621057447 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:21:40.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.558+0000 7f88be7fc700 1 --2- 172.21.15.189:0/1621057447 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88c80a43c0 0x7f88c814d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:40.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.558+0000 7f88c6ffd700 1 --2- 172.21.15.189:0/1621057447 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f88c80664d0 0x7f88c80ac020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:40.570 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.558+0000 7f88c77fe700 1 -- 172.21.15.189:0/1621057447 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f88c80664d0 msgr2=0x7f88c80ac020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:40.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.558+0000 7f88c77fe700 1 --2- 172.21.15.189:0/1621057447 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f88c80664d0 0x7f88c80ac020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:40.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.558+0000 7f88c77fe700 1 -- 172.21.15.189:0/1621057447 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88c80a43c0 msgr2=0x7f88c814d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:40.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.558+0000 7f88c77fe700 1 --2- 172.21.15.189:0/1621057447 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88c80a43c0 0x7f88c814d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:40.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.558+0000 7f88c77fe700 1 -- 172.21.15.189:0/1621057447 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c814dc00 con 0x7f88c80a9950 2020-11-20T12:21:40.571 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.558+0000 7f88c77fe700 1 --2- 172.21.15.189:0/1621057447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88c80a9950 0x7f88c81360a0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7f88c000ca90 tx=0x7f88c0007ee0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:40.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.558+0000 7f88bf7fe700 1 -- 172.21.15.189:0/1621057447 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c000a7f0 con 0x7f88c80a9950 2020-11-20T12:21:40.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.558+0000 7f88bf7fe700 1 -- 172.21.15.189:0/1621057447 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c000a950 con 0x7f88c80a9950 2020-11-20T12:21:40.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.558+0000 7f88cd6a1700 1 -- 172.21.15.189:0/1621057447 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c81337e0 con 0x7f88c80a9950 2020-11-20T12:21:40.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.558+0000 7f88bf7fe700 1 -- 172.21.15.189:0/1621057447 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c000acd0 con 0x7f88c80a9950 2020-11-20T12:21:40.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.558+0000 7f88cd6a1700 1 -- 172.21.15.189:0/1621057447 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c81367b0 con 0x7f88c80a9950 2020-11-20T12:21:40.572 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.562+0000 7f88cd6a1700 1 -- 172.21.15.189:0/1621057447 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c814d430 con 0x7f88c80a9950 2020-11-20T12:21:40.573 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.562+0000 7f88bf7fe700 1 -- 172.21.15.189:0/1621057447 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f88c0019030 con 0x7f88c80a9950 2020-11-20T12:21:40.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.562+0000 7f88bf7fe700 1 --2- 172.21.15.189:0/1621057447 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f88a40557c0 0x7f88a4057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:40.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.562+0000 7f88bf7fe700 1 -- 172.21.15.189:0/1621057447 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 7105+0+0 (secure 0 0 0) 0x7f88c006e270 con 0x7f88c80a9950 2020-11-20T12:21:40.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.562+0000 7f88c6ffd700 1 --2- 172.21.15.189:0/1621057447 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f88a40557c0 0x7f88a4057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:40.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.562+0000 7f88c6ffd700 1 --2- 172.21.15.189:0/1621057447 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f88a40557c0 0x7f88a4057c80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f88b800beb0 tx=0x7f88b8015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:40.575 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.566+0000 7f88bf7fe700 1 -- 172.21.15.189:0/1621057447 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f88c0043db0 con 0x7f88c80a9950 2020-11-20T12:21:40.776 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.770+0000 7f88cd6a1700 1 -- 172.21.15.189:0/1621057447 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88c8044f60 con 0x7f88c80a9950 2020-11-20T12:21:40.777 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.770+0000 7f88bf7fe700 1 -- 172.21.15.189:0/1621057447 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f88c00473d0 con 0x7f88c80a9950 2020-11-20T12:21:40.778 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:21:40.778 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5297,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.47:6833/3025803398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3025803398},{"type":"v1","addr":"172.21.15.47:6833","nonce":3025803398}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:21:00.823729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5746":{"gid":5746,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6835/3538437301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3538437301},{"type":"v1","addr":"172.21.15.189:6835","nonce":3538437301}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:21:40.780 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.770+0000 7f88cd6a1700 1 -- 172.21.15.189:0/1621057447 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f88a40557c0 msgr2=0x7f88a4057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:40.780 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.770+0000 7f88cd6a1700 1 --2- 172.21.15.189:0/1621057447 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f88a40557c0 0x7f88a4057c80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f88b800beb0 tx=0x7f88b8015000).stop 2020-11-20T12:21:40.780 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.770+0000 7f88cd6a1700 1 -- 172.21.15.189:0/1621057447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88c80a9950 msgr2=0x7f88c81360a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:40.780 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.770+0000 7f88cd6a1700 1 --2- 172.21.15.189:0/1621057447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88c80a9950 0x7f88c81360a0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7f88c000ca90 tx=0x7f88c0007ee0).stop 2020-11-20T12:21:40.781 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.774+0000 7f88cd6a1700 1 -- 172.21.15.189:0/1621057447 shutdown_connections 2020-11-20T12:21:40.781 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.774+0000 7f88cd6a1700 1 --2- 172.21.15.189:0/1621057447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88c80a9950 0x7f88c81360a0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:40.781 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.774+0000 7f88cd6a1700 1 --2- 172.21.15.189:0/1621057447 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f88c80664d0 0x7f88c80ac020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:40.781 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.774+0000 7f88cd6a1700 1 --2- 172.21.15.189:0/1621057447 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f88a40557c0 0x7f88a4057c80 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:40.781 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.774+0000 7f88cd6a1700 1 --2- 172.21.15.189:0/1621057447 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f88c80a43c0 0x7f88c814d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:40.782 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.774+0000 7f88cd6a1700 1 -- 172.21.15.189:0/1621057447 >> 172.21.15.189:0/1621057447 conn(0x7f88c8061f60 msgr2=0x7f88c8060170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:40.782 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.774+0000 7f88cd6a1700 1 -- 172.21.15.189:0/1621057447 shutdown_connections 2020-11-20T12:21:40.782 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:40.774+0000 7f88cd6a1700 1 -- 172.21.15.189:0/1621057447 wait complete. 2020-11-20T12:21:40.782 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 66 2020-11-20T12:21:40.794 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:21:42.797 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:42.932 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.922+0000 7f904359e700 1 Processor -- start 2020-11-20T12:21:42.933 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.922+0000 7f904359e700 1 -- start start 2020-11-20T12:21:42.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.922+0000 7f904359e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f904413bc80 0x7f904414a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:42.934 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.922+0000 7f904359e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9044066800 con 0x7f9044066ed0 2020-11-20T12:21:42.935 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.922+0000 7f904359e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9044066aa0 con 0x7f90440a8840 2020-11-20T12:21:42.935 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.922+0000 7f904359e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9044067450 con 0x7f904413bc80 2020-11-20T12:21:42.936 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.922+0000 7f904259c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f90440a8840 0x7f904413eb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53680/0 (socket says 172.21.15.189:53680) 2020-11-20T12:21:42.937 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.922+0000 7f904259c700 1 -- 172.21.15.189:0/4174713448 learned_addr learned my addr 172.21.15.189:0/4174713448 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:21:42.937 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.922+0000 7f9042d9d700 1 --2- 172.21.15.189:0/4174713448 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f904413bc80 0x7f904414a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:42.938 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f904159a700 1 -- 172.21.15.189:0/4174713448 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754919602 0 0) 0x7f9044066aa0 con 0x7f90440a8840 2020-11-20T12:21:42.938 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f904159a700 1 -- 172.21.15.189:0/4174713448 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9028003240 con 0x7f90440a8840 2020-11-20T12:21:42.939 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f904159a700 1 -- 172.21.15.189:0/4174713448 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1399497473 0 0) 0x7f9044066800 con 0x7f9044066ed0 2020-11-20T12:21:42.939 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f904159a700 1 -- 172.21.15.189:0/4174713448 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9044066aa0 con 0x7f9044066ed0 2020-11-20T12:21:42.940 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f9042d9d700 1 -- 172.21.15.189:0/4174713448 >> v1:172.21.15.189:6789/0 conn(0x7f90440a8840 legacy=0x7f904413eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:42.941 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f9042d9d700 1 -- 172.21.15.189:0/4174713448 >> v1:172.21.15.47:6789/0 conn(0x7f9044066ed0 legacy=0x7f904413b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:42.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f9042d9d700 1 -- 172.21.15.189:0/4174713448 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90440676f0 con 0x7f904413bc80 2020-11-20T12:21:42.942 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f9042d9d700 1 --2- 172.21.15.189:0/4174713448 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f904413bc80 0x7f904414a5f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f903400b9d0 tx=0x7f9034006df0).ready entity=mon.2 client_cookie=bbf37d66ed90c2c1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:42.944 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f904159a700 1 -- 172.21.15.189:0/4174713448 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9034009710 con 0x7f904413bc80 2020-11-20T12:21:42.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f904159a700 1 -- 172.21.15.189:0/4174713448 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9034009870 con 0x7f904413bc80 2020-11-20T12:21:42.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f904159a700 1 -- 172.21.15.189:0/4174713448 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9034009bf0 con 0x7f904413bc80 2020-11-20T12:21:42.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f904359e700 1 -- 172.21.15.189:0/4174713448 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f904413bc80 msgr2=0x7f904414a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:42.945 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f904359e700 1 --2- 172.21.15.189:0/4174713448 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f904413bc80 0x7f904414a5f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f903400b9d0 tx=0x7f9034006df0).stop 2020-11-20T12:21:42.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f904359e700 1 -- 172.21.15.189:0/4174713448 shutdown_connections 2020-11-20T12:21:42.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f904359e700 1 --2- 172.21.15.189:0/4174713448 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f904413bc80 0x7f904414a5f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:42.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f904359e700 1 -- 172.21.15.189:0/4174713448 >> 172.21.15.189:0/4174713448 conn(0x7f9044060170 msgr2=0x7f9044060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:42.946 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f904359e700 1 -- 172.21.15.189:0/4174713448 shutdown_connections 2020-11-20T12:21:42.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f904359e700 1 -- 172.21.15.189:0/4174713448 wait complete. 2020-11-20T12:21:42.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f904359e700 1 Processor -- start 2020-11-20T12:21:42.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f904359e700 1 -- start start 2020-11-20T12:21:42.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f904359e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9044066ed0 0x7f9044145b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:42.947 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f904359e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f90440a8840 0x7f904409cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:42.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f904359e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f904413bc80 0x7f90440a6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:42.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f904359e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904409d010 con 0x7f904413bc80 2020-11-20T12:21:42.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f904359e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90440a5340 con 0x7f9044066ed0 2020-11-20T12:21:42.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f904359e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f904409e7d0 con 0x7f90440a8840 2020-11-20T12:21:42.948 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f904259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9044066ed0 0x7f9044145b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:42.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f904259c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9044066ed0 0x7f9044145b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52062/0 (socket says 172.21.15.189:52062) 2020-11-20T12:21:42.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f904259c700 1 -- 172.21.15.189:0/3990653704 learned_addr learned my addr 172.21.15.189:0/3990653704 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:21:42.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f9042d9d700 1 --2- 172.21.15.189:0/3990653704 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f904413bc80 0x7f90440a6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:42.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.926+0000 7f9041d9b700 1 --2- 172.21.15.189:0/3990653704 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f90440a8840 0x7f904409cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:42.949 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.930+0000 7f904259c700 1 -- 172.21.15.189:0/3990653704 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f90440a8840 msgr2=0x7f904409cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:42.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.930+0000 7f904259c700 1 --2- 172.21.15.189:0/3990653704 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f90440a8840 0x7f904409cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:42.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.930+0000 7f904259c700 1 -- 172.21.15.189:0/3990653704 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f904413bc80 msgr2=0x7f90440a6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:42.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.930+0000 7f904259c700 1 --2- 172.21.15.189:0/3990653704 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f904413bc80 0x7f90440a6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:42.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.930+0000 7f904259c700 1 -- 172.21.15.189:0/3990653704 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904409ea70 con 0x7f9044066ed0 2020-11-20T12:21:42.950 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.930+0000 7f904259c700 1 --2- 172.21.15.189:0/3990653704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9044066ed0 0x7f9044145b80 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7f9038007fd0 tx=0x7f9038007a80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:42.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.930+0000 7f90327fc700 1 -- 172.21.15.189:0/3990653704 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90380072e0 con 0x7f9044066ed0 2020-11-20T12:21:42.951 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.930+0000 7f90327fc700 1 -- 172.21.15.189:0/3990653704 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9038007440 con 0x7f9044066ed0 2020-11-20T12:21:42.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.930+0000 7f904359e700 1 -- 172.21.15.189:0/3990653704 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f904409e3e0 con 0x7f9044066ed0 2020-11-20T12:21:42.952 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.930+0000 7f90327fc700 1 -- 172.21.15.189:0/3990653704 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9038005800 con 0x7f9044066ed0 2020-11-20T12:21:42.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.930+0000 7f904359e700 1 -- 172.21.15.189:0/3990653704 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90440a6840 con 0x7f9044066ed0 2020-11-20T12:21:42.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.930+0000 7f904359e700 1 -- 172.21.15.189:0/3990653704 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9044146070 con 0x7f9044066ed0 2020-11-20T12:21:42.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.930+0000 7f90327fc700 1 -- 172.21.15.189:0/3990653704 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f9038010070 con 0x7f9044066ed0 2020-11-20T12:21:42.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.930+0000 7f90327fc700 1 --2- 172.21.15.189:0/3990653704 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f90180557c0 0x7f9018057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:42.953 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.930+0000 7f90327fc700 1 -- 172.21.15.189:0/3990653704 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 7105+0+0 (secure 0 0 0) 0x7f903806e210 con 0x7f9044066ed0 2020-11-20T12:21:42.954 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.930+0000 7f9041d9b700 1 --2- 172.21.15.189:0/3990653704 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f90180557c0 0x7f9018057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:42.954 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.930+0000 7f9041d9b700 1 --2- 172.21.15.189:0/3990653704 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f90180557c0 0x7f9018057c80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f90441462c0 tx=0x7f903c009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:42.954 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:42.938+0000 7f90327fc700 1 -- 172.21.15.189:0/3990653704 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9038043ca0 con 0x7f9044066ed0 2020-11-20T12:21:43.146 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:43.138+0000 7f904359e700 1 -- 172.21.15.189:0/3990653704 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f904409ec00 con 0x7f9044066ed0 2020-11-20T12:21:43.147 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:43.138+0000 7f90327fc700 1 -- 172.21.15.189:0/3990653704 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f90380472c0 con 0x7f9044066ed0 2020-11-20T12:21:43.147 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:21:43.147 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5297,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.47:6833/3025803398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3025803398},{"type":"v1","addr":"172.21.15.47:6833","nonce":3025803398}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:21:00.823729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5746":{"gid":5746,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6835/3538437301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3538437301},{"type":"v1","addr":"172.21.15.189:6835","nonce":3538437301}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:21:43.150 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:43.142+0000 7f904359e700 1 -- 172.21.15.189:0/3990653704 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f90180557c0 msgr2=0x7f9018057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:43.150 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:43.142+0000 7f904359e700 1 --2- 172.21.15.189:0/3990653704 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f90180557c0 0x7f9018057c80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f90441462c0 tx=0x7f903c009040).stop 2020-11-20T12:21:43.150 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:43.142+0000 7f904359e700 1 -- 172.21.15.189:0/3990653704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9044066ed0 msgr2=0x7f9044145b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:43.151 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:43.142+0000 7f904359e700 1 --2- 172.21.15.189:0/3990653704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9044066ed0 0x7f9044145b80 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7f9038007fd0 tx=0x7f9038007a80).stop 2020-11-20T12:21:43.151 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:43.142+0000 7f904359e700 1 -- 172.21.15.189:0/3990653704 shutdown_connections 2020-11-20T12:21:43.151 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:43.142+0000 7f904359e700 1 --2- 172.21.15.189:0/3990653704 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f90180557c0 0x7f9018057c80 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:43.151 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:43.142+0000 7f904359e700 1 --2- 172.21.15.189:0/3990653704 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f904413bc80 0x7f90440a6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:43.151 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:43.142+0000 7f904359e700 1 --2- 172.21.15.189:0/3990653704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9044066ed0 0x7f9044145b80 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:43.152 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:43.142+0000 7f904359e700 1 --2- 172.21.15.189:0/3990653704 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f90440a8840 0x7f904409cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:43.152 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:43.142+0000 7f904359e700 1 -- 172.21.15.189:0/3990653704 >> 172.21.15.189:0/3990653704 conn(0x7f9044060170 msgr2=0x7f90440a7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:43.152 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:43.142+0000 7f904359e700 1 -- 172.21.15.189:0/3990653704 shutdown_connections 2020-11-20T12:21:43.152 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:43.142+0000 7f904359e700 1 -- 172.21.15.189:0/3990653704 wait complete. 2020-11-20T12:21:43.153 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 66 2020-11-20T12:21:43.167 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:21:45.169 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:45.305 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.298+0000 7fb331ae9700 1 Processor -- start 2020-11-20T12:21:45.305 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.298+0000 7fb331ae9700 1 -- start start 2020-11-20T12:21:45.307 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.298+0000 7fb331ae9700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fb32c0a8840 0x7fb32c13eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:45.307 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.298+0000 7fb331ae9700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb32c13f030 0x7fb32c1427a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:45.307 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.298+0000 7fb331ae9700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb32c066800 con 0x7fb32c066ed0 2020-11-20T12:21:45.307 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.298+0000 7fb331ae9700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb32c13f4a0 con 0x7fb32c0a8840 2020-11-20T12:21:45.308 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.298+0000 7fb331ae9700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb32c066aa0 con 0x7fb32c13f030 2020-11-20T12:21:45.308 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.298+0000 7fb3312e8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb32c13f030 0x7fb32c1427a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:45.308 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.298+0000 7fb32bfff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb32c066ed0 0x7fb32c142090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53698/0 (socket says 172.21.15.189:53698) 2020-11-20T12:21:45.308 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.298+0000 7fb32bfff700 1 -- 172.21.15.189:0/2268979614 learned_addr learned my addr 172.21.15.189:0/2268979614 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:21:45.308 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.298+0000 7fb3312e8700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb32c13f030 0x7fb32c1427a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52072/0 (socket says 172.21.15.189:52072) 2020-11-20T12:21:45.309 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.298+0000 7fb330ae7700 1 --2- 172.21.15.189:0/2268979614 >> v2:172.21.15.47:3300/0 conn(0x7fb32c0a8840 0x7fb32c13eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:45.309 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.298+0000 7fb3312e8700 1 -- 172.21.15.189:0/2268979614 >> v2:172.21.15.47:3300/0 conn(0x7fb32c0a8840 msgr2=0x7fb32c13eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:45.309 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.298+0000 7fb3312e8700 1 --2- 172.21.15.189:0/2268979614 >> v2:172.21.15.47:3300/0 conn(0x7fb32c0a8840 0x7fb32c13eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:45.309 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.298+0000 7fb3312e8700 1 -- 172.21.15.189:0/2268979614 >> v1:172.21.15.189:6789/0 conn(0x7fb32c066ed0 legacy=0x7fb32c142090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:45.309 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.298+0000 7fb3312e8700 1 -- 172.21.15.189:0/2268979614 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32c067450 con 0x7fb32c13f030 2020-11-20T12:21:45.310 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.298+0000 7fb32b7fe700 1 -- 172.21.15.189:0/2268979614 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275892079 0 0) 0x7fb32c066800 con 0x7fb32c066ed0 2020-11-20T12:21:45.310 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.298+0000 7fb330ae7700 1 --2- 172.21.15.189:0/2268979614 >> v2:172.21.15.47:3300/0 conn(0x7fb32c0a8840 0x7fb32c13eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:21:45.310 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.298+0000 7fb3312e8700 1 --2- 172.21.15.189:0/2268979614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb32c13f030 0x7fb32c1427a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7fb32c065b90 tx=0x7fb3240091d0).ready entity=mon.1 client_cookie=5f8c8e660b8105bd server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:45.310 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.298+0000 7fb32b7fe700 1 -- 172.21.15.189:0/2268979614 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32400bbc0 con 0x7fb32c13f030 2020-11-20T12:21:45.310 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.298+0000 7fb32b7fe700 1 -- 172.21.15.189:0/2268979614 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb32400bd20 con 0x7fb32c13f030 2020-11-20T12:21:45.311 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.298+0000 7fb32b7fe700 1 -- 172.21.15.189:0/2268979614 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb324011610 con 0x7fb32c13f030 2020-11-20T12:21:45.313 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb331ae9700 1 -- 172.21.15.189:0/2268979614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb32c13f030 msgr2=0x7fb32c1427a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:45.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb331ae9700 1 --2- 172.21.15.189:0/2268979614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb32c13f030 0x7fb32c1427a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7fb32c065b90 tx=0x7fb3240091d0).stop 2020-11-20T12:21:45.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb331ae9700 1 -- 172.21.15.189:0/2268979614 shutdown_connections 2020-11-20T12:21:45.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb331ae9700 1 --2- 172.21.15.189:0/2268979614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb32c13f030 0x7fb32c1427a0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:45.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb331ae9700 1 --2- 172.21.15.189:0/2268979614 >> v2:172.21.15.47:3300/0 conn(0x7fb32c0a8840 0x7fb32c13eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:45.314 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb331ae9700 1 -- 172.21.15.189:0/2268979614 >> 172.21.15.189:0/2268979614 conn(0x7fb32c060170 msgr2=0x7fb32c060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:45.315 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb331ae9700 1 -- 172.21.15.189:0/2268979614 shutdown_connections 2020-11-20T12:21:45.315 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb331ae9700 1 -- 172.21.15.189:0/2268979614 wait complete. 2020-11-20T12:21:45.315 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb331ae9700 1 Processor -- start 2020-11-20T12:21:45.315 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb331ae9700 1 -- start start 2020-11-20T12:21:45.315 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb331ae9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb32c066ed0 0x7fb32c0a2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:45.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb331ae9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb32c0a8840 0x7fb32c09e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:45.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb331ae9700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb32c13f030 0x7fb32c0a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:45.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb331ae9700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c09da40 con 0x7fb32c066ed0 2020-11-20T12:21:45.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb331ae9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c13c8b0 con 0x7fb32c0a8840 2020-11-20T12:21:45.316 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb331ae9700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c0a3120 con 0x7fb32c13f030 2020-11-20T12:21:45.317 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb32bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb32c0a8840 0x7fb32c09e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:45.317 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb32bfff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb32c0a8840 0x7fb32c09e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52080/0 (socket says 172.21.15.189:52080) 2020-11-20T12:21:45.317 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb32bfff700 1 -- 172.21.15.189:0/3077713831 learned_addr learned my addr 172.21.15.189:0/3077713831 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:21:45.317 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb330ae7700 1 --2- 172.21.15.189:0/3077713831 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb32c066ed0 0x7fb32c0a2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:45.317 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb3312e8700 1 --2- 172.21.15.189:0/3077713831 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb32c13f030 0x7fb32c0a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:45.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb32bfff700 1 -- 172.21.15.189:0/3077713831 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb32c13f030 msgr2=0x7fb32c0a1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:45.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb32bfff700 1 --2- 172.21.15.189:0/3077713831 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb32c13f030 0x7fb32c0a1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:45.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb32bfff700 1 -- 172.21.15.189:0/3077713831 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb32c066ed0 msgr2=0x7fb32c0a2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:45.318 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb32bfff700 1 --2- 172.21.15.189:0/3077713831 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb32c066ed0 0x7fb32c0a2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:45.319 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb32bfff700 1 -- 172.21.15.189:0/3077713831 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32c0a33c0 con 0x7fb32c0a8840 2020-11-20T12:21:45.319 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb32bfff700 1 --2- 172.21.15.189:0/3077713831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb32c0a8840 0x7fb32c09e690 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7fb31c00c720 tx=0x7fb31c007ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:45.319 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb328ff9700 1 -- 172.21.15.189:0/3077713831 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c0072e0 con 0x7fb32c0a8840 2020-11-20T12:21:45.319 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb328ff9700 1 -- 172.21.15.189:0/3077713831 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb31c007440 con 0x7fb32c0a8840 2020-11-20T12:21:45.320 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb331ae9700 1 -- 172.21.15.189:0/3077713831 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb32c0a3660 con 0x7fb32c0a8840 2020-11-20T12:21:45.320 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb328ff9700 1 -- 172.21.15.189:0/3077713831 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c005800 con 0x7fb32c0a8840 2020-11-20T12:21:45.320 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.302+0000 7fb331ae9700 1 -- 172.21.15.189:0/3077713831 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb32c09c910 con 0x7fb32c0a8840 2020-11-20T12:21:45.320 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.306+0000 7fb331ae9700 1 -- 172.21.15.189:0/3077713831 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb32c0a3b90 con 0x7fb32c0a8840 2020-11-20T12:21:45.320 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.306+0000 7fb328ff9700 1 -- 172.21.15.189:0/3077713831 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fb31c01a060 con 0x7fb32c0a8840 2020-11-20T12:21:45.321 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.306+0000 7fb328ff9700 1 --2- 172.21.15.189:0/3077713831 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb3080557c0 0x7fb308057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:45.321 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.306+0000 7fb328ff9700 1 -- 172.21.15.189:0/3077713831 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 7105+0+0 (secure 0 0 0) 0x7fb31c06e210 con 0x7fb32c0a8840 2020-11-20T12:21:45.321 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.306+0000 7fb330ae7700 1 --2- 172.21.15.189:0/3077713831 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb3080557c0 0x7fb308057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:45.322 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.306+0000 7fb330ae7700 1 --2- 172.21.15.189:0/3077713831 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb3080557c0 0x7fb308057c80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fb31400ba80 tx=0x7fb314008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:45.322 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.314+0000 7fb328ff9700 1 -- 172.21.15.189:0/3077713831 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb31c043ca0 con 0x7fb32c0a8840 2020-11-20T12:21:45.523 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.518+0000 7fb331ae9700 1 -- 172.21.15.189:0/3077713831 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb32c0446d0 con 0x7fb32c0a8840 2020-11-20T12:21:45.524 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.518+0000 7fb328ff9700 1 -- 172.21.15.189:0/3077713831 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fb31c018070 con 0x7fb32c0a8840 2020-11-20T12:21:45.525 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:21:45.525 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5297,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.47:6833/3025803398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3025803398},{"type":"v1","addr":"172.21.15.47:6833","nonce":3025803398}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:21:00.823729+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5746":{"gid":5746,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6835/3538437301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3538437301},{"type":"v1","addr":"172.21.15.189:6835","nonce":3538437301}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:21:45.528 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.518+0000 7fb331ae9700 1 -- 172.21.15.189:0/3077713831 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb3080557c0 msgr2=0x7fb308057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:45.528 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.518+0000 7fb331ae9700 1 --2- 172.21.15.189:0/3077713831 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb3080557c0 0x7fb308057c80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fb31400ba80 tx=0x7fb314008040).stop 2020-11-20T12:21:45.528 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.518+0000 7fb331ae9700 1 -- 172.21.15.189:0/3077713831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb32c0a8840 msgr2=0x7fb32c09e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:45.528 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.518+0000 7fb331ae9700 1 --2- 172.21.15.189:0/3077713831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb32c0a8840 0x7fb32c09e690 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7fb31c00c720 tx=0x7fb31c007ac0).stop 2020-11-20T12:21:45.528 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.518+0000 7fb331ae9700 1 -- 172.21.15.189:0/3077713831 shutdown_connections 2020-11-20T12:21:45.529 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.518+0000 7fb331ae9700 1 --2- 172.21.15.189:0/3077713831 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fb3080557c0 0x7fb308057c80 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:45.529 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.518+0000 7fb331ae9700 1 --2- 172.21.15.189:0/3077713831 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fb32c066ed0 0x7fb32c0a2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:45.529 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.518+0000 7fb331ae9700 1 --2- 172.21.15.189:0/3077713831 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fb32c13f030 0x7fb32c0a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:45.529 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.518+0000 7fb331ae9700 1 --2- 172.21.15.189:0/3077713831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb32c0a8840 0x7fb32c09e690 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:45.530 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.518+0000 7fb331ae9700 1 -- 172.21.15.189:0/3077713831 >> 172.21.15.189:0/3077713831 conn(0x7fb32c060170 msgr2=0x7fb32c05ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:45.530 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.522+0000 7fb331ae9700 1 -- 172.21.15.189:0/3077713831 shutdown_connections 2020-11-20T12:21:45.530 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:45.522+0000 7fb331ae9700 1 -- 172.21.15.189:0/3077713831 wait complete. 2020-11-20T12:21:45.530 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 66 2020-11-20T12:21:45.544 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2020-11-20T12:21:47.545 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:47.676 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.670+0000 7f75e88ff700 1 Processor -- start 2020-11-20T12:21:47.676 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.670+0000 7f75e88ff700 1 -- start start 2020-11-20T12:21:47.678 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.670+0000 7f75e88ff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75e40aa680 0x7f75e40a70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:47.678 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.670+0000 7f75e88ff700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f75e40a75c0 0x7f75e40a9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:47.679 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.670+0000 7f75e88ff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e40a90c0 con 0x7f75e40a9b60 2020-11-20T12:21:47.679 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.670+0000 7f75e88ff700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75e40ab630 con 0x7f75e40a75c0 2020-11-20T12:21:47.679 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.670+0000 7f75e88ff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75e40a9ff0 con 0x7f75e40aa680 2020-11-20T12:21:47.679 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.670+0000 7f75e1d9b700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f75e40a75c0 0x7f75e40a9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:47.679 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.670+0000 7f75e2d9d700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f75e40a9b60 0x7f75e4060170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.189:48376/0 (socket says 172.21.15.189:48376) 2020-11-20T12:21:47.680 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.670+0000 7f75e2d9d700 1 -- 172.21.15.189:0/1346174127 learned_addr learned my addr 172.21.15.189:0/1346174127 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:21:47.680 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.670+0000 7f75e259c700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75e40aa680 0x7f75e40a70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:47.680 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.670+0000 7f75e159a700 1 -- 172.21.15.189:0/1346174127 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3589197300 0 0) 0x7f75e40a90c0 con 0x7f75e40a9b60 2020-11-20T12:21:47.680 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.670+0000 7f75e159a700 1 -- 172.21.15.189:0/1346174127 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75c8003240 con 0x7f75e40a9b60 2020-11-20T12:21:47.680 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.670+0000 7f75e1d9b700 1 -- 172.21.15.189:0/1346174127 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75e40aa680 msgr2=0x7f75e40a70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:47.681 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.670+0000 7f75e1d9b700 1 --2- 172.21.15.189:0/1346174127 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75e40aa680 0x7f75e40a70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:47.681 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.670+0000 7f75e1d9b700 1 -- 172.21.15.189:0/1346174127 >> v1:172.21.15.47:6789/0 conn(0x7f75e40a9b60 legacy=0x7f75e4060170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:47.681 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.670+0000 7f75e1d9b700 1 -- 172.21.15.189:0/1346174127 --> v2:172.21.15.47:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e40ab890 con 0x7f75e40a75c0 2020-11-20T12:21:47.681 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.670+0000 7f75e259c700 1 --2- 172.21.15.189:0/1346174127 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75e40aa680 0x7f75e40a70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:21:47.681 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.670+0000 7f75e1d9b700 1 --2- 172.21.15.189:0/1346174127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75e40a75c0 0x7f75e40a9670 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f75dc00b9a0 tx=0x7f75dc006df0).ready entity=mon.0 client_cookie=6ebb91147d772741 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:47.684 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.670+0000 7f75e159a700 1 -- 172.21.15.189:0/1346174127 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75dc009710 con 0x7f75e40a75c0 2020-11-20T12:21:47.684 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.670+0000 7f75e159a700 1 -- 172.21.15.189:0/1346174127 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75dc009870 con 0x7f75e40a75c0 2020-11-20T12:21:47.685 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.670+0000 7f75e159a700 1 -- 172.21.15.189:0/1346174127 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75dc009bf0 con 0x7f75e40a75c0 2020-11-20T12:21:47.685 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.674+0000 7f75e88ff700 1 -- 172.21.15.189:0/1346174127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75e40a75c0 msgr2=0x7f75e40a9670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:47.685 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.674+0000 7f75e88ff700 1 --2- 172.21.15.189:0/1346174127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75e40a75c0 0x7f75e40a9670 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f75dc00b9a0 tx=0x7f75dc006df0).stop 2020-11-20T12:21:47.685 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.674+0000 7f75e88ff700 1 -- 172.21.15.189:0/1346174127 shutdown_connections 2020-11-20T12:21:47.685 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.674+0000 7f75e88ff700 1 --2- 172.21.15.189:0/1346174127 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75e40aa680 0x7f75e40a70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:47.686 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.674+0000 7f75e88ff700 1 --2- 172.21.15.189:0/1346174127 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75e40a75c0 0x7f75e40a9670 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:47.686 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.674+0000 7f75e88ff700 1 -- 172.21.15.189:0/1346174127 >> 172.21.15.189:0/1346174127 conn(0x7f75e4061f60 msgr2=0x7f75e4062340 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:47.686 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.674+0000 7f75e88ff700 1 -- 172.21.15.189:0/1346174127 shutdown_connections 2020-11-20T12:21:47.686 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.674+0000 7f75e88ff700 1 -- 172.21.15.189:0/1346174127 wait complete. 2020-11-20T12:21:47.686 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.674+0000 7f75e88ff700 1 Processor -- start 2020-11-20T12:21:47.687 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.674+0000 7f75e88ff700 1 -- start start 2020-11-20T12:21:47.687 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.674+0000 7f75e88ff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75e40a75c0 0x7f75e4139bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:47.687 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.674+0000 7f75e88ff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75e40a9b60 0x7f75e4147d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:47.687 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.674+0000 7f75e88ff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75e40aa680 0x7f75e4137b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:47.687 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.674+0000 7f75e88ff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e41397b0 con 0x7f75e40a75c0 2020-11-20T12:21:47.688 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.674+0000 7f75e88ff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e41485b0 con 0x7f75e40aa680 2020-11-20T12:21:47.688 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.674+0000 7f75e88ff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75e41376b0 con 0x7f75e40a9b60 2020-11-20T12:21:47.688 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.674+0000 7f75e2d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75e40aa680 0x7f75e4137b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:47.688 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.674+0000 7f75e2d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75e40aa680 0x7f75e4137b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52096/0 (socket says 172.21.15.189:52096) 2020-11-20T12:21:47.688 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.674+0000 7f75e259c700 1 --2- 172.21.15.189:0/2294579139 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75e40a75c0 0x7f75e4139bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:47.689 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.674+0000 7f75e2d9d700 1 -- 172.21.15.189:0/2294579139 learned_addr learned my addr 172.21.15.189:0/2294579139 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:21:47.689 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.674+0000 7f75e1d9b700 1 --2- 172.21.15.189:0/2294579139 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75e40a9b60 0x7f75e4147d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:47.689 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.674+0000 7f75e2d9d700 1 -- 172.21.15.189:0/2294579139 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75e40a9b60 msgr2=0x7f75e4147d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:47.689 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.674+0000 7f75e2d9d700 1 --2- 172.21.15.189:0/2294579139 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75e40a9b60 0x7f75e4147d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:47.689 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.674+0000 7f75e2d9d700 1 -- 172.21.15.189:0/2294579139 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75e40a75c0 msgr2=0x7f75e4139bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:47.690 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.674+0000 7f75e2d9d700 1 --2- 172.21.15.189:0/2294579139 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75e40a75c0 0x7f75e4139bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:47.690 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.674+0000 7f75e2d9d700 1 -- 172.21.15.189:0/2294579139 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e4148220 con 0x7f75e40aa680 2020-11-20T12:21:47.690 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.678+0000 7f75e2d9d700 1 --2- 172.21.15.189:0/2294579139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75e40aa680 0x7f75e4137b50 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f75cc00c6e0 tx=0x7f75cc007a20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:47.690 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.678+0000 7f75da7fc700 1 -- 172.21.15.189:0/2294579139 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75cc0072e0 con 0x7f75e40aa680 2020-11-20T12:21:47.691 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.678+0000 7f75da7fc700 1 -- 172.21.15.189:0/2294579139 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75cc007440 con 0x7f75e40aa680 2020-11-20T12:21:47.691 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.678+0000 7f75e88ff700 1 -- 172.21.15.189:0/2294579139 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75e4138f90 con 0x7f75e40aa680 2020-11-20T12:21:47.691 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.678+0000 7f75da7fc700 1 -- 172.21.15.189:0/2294579139 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75cc005800 con 0x7f75e40aa680 2020-11-20T12:21:47.691 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.678+0000 7f75e88ff700 1 -- 172.21.15.189:0/2294579139 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75e4138260 con 0x7f75e40aa680 2020-11-20T12:21:47.692 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.678+0000 7f75e88ff700 1 -- 172.21.15.189:0/2294579139 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75e40a8f20 con 0x7f75e40aa680 2020-11-20T12:21:47.693 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.678+0000 7f75da7fc700 1 -- 172.21.15.189:0/2294579139 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f75cc010070 con 0x7f75e40aa680 2020-11-20T12:21:47.693 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.682+0000 7f75da7fc700 1 --2- 172.21.15.189:0/2294579139 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f75b40557c0 0x7f75b4057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:47.693 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.682+0000 7f75e259c700 1 --2- 172.21.15.189:0/2294579139 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f75b40557c0 0x7f75b4057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:47.694 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.682+0000 7f75da7fc700 1 -- 172.21.15.189:0/2294579139 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 7105+0+0 (secure 0 0 0) 0x7f75cc06df90 con 0x7f75e40aa680 2020-11-20T12:21:47.694 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.682+0000 7f75e259c700 1 --2- 172.21.15.189:0/2294579139 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f75b40557c0 0x7f75b4057c80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f75d4009fa0 tx=0x7f75d40091d0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:47.694 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.686+0000 7f75da7fc700 1 -- 172.21.15.189:0/2294579139 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f75e40a8f20 con 0x7f75e40aa680 2020-11-20T12:21:47.906 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.898+0000 7f75e88ff700 1 -- 172.21.15.189:0/2294579139 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75e40a8f20 con 0x7f75e40aa680 2020-11-20T12:21:48.079 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.898+0000 7f75da7fc700 1 -- 172.21.15.189:0/2294579139 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+1784 (secure 0 0 0) 0x7f75e40a8f20 con 0x7f75e40aa680 2020-11-20T12:21:48.079 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.902+0000 7f75e88ff700 1 -- 172.21.15.189:0/2294579139 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f75b40557c0 msgr2=0x7f75b4057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:48.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.902+0000 7f75e88ff700 1 --2- 172.21.15.189:0/2294579139 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f75b40557c0 0x7f75b4057c80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f75d4009fa0 tx=0x7f75d40091d0).stop 2020-11-20T12:21:48.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.902+0000 7f75e88ff700 1 -- 172.21.15.189:0/2294579139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75e40aa680 msgr2=0x7f75e4137b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:48.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.902+0000 7f75e88ff700 1 --2- 172.21.15.189:0/2294579139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75e40aa680 0x7f75e4137b50 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f75cc00c6e0 tx=0x7f75cc007a20).stop 2020-11-20T12:21:48.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.902+0000 7f75e88ff700 1 -- 172.21.15.189:0/2294579139 shutdown_connections 2020-11-20T12:21:48.080 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.902+0000 7f75e88ff700 1 --2- 172.21.15.189:0/2294579139 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f75b40557c0 0x7f75b4057c80 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:48.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.902+0000 7f75e88ff700 1 --2- 172.21.15.189:0/2294579139 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f75e40a75c0 0x7f75e4139bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:48.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.902+0000 7f75e88ff700 1 --2- 172.21.15.189:0/2294579139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75e40aa680 0x7f75e4137b50 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:48.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.902+0000 7f75e88ff700 1 --2- 172.21.15.189:0/2294579139 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f75e40a9b60 0x7f75e4147d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:48.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.902+0000 7f75e88ff700 1 -- 172.21.15.189:0/2294579139 >> 172.21.15.189:0/2294579139 conn(0x7f75e4061f60 msgr2=0x7f75e405ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:48.081 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.902+0000 7f75e88ff700 1 -- 172.21.15.189:0/2294579139 shutdown_connections 2020-11-20T12:21:48.082 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:47.902+0000 7f75e88ff700 1 -- 172.21.15.189:0/2294579139 wait complete. 2020-11-20T12:21:48.082 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 67 2020-11-20T12:21:48.082 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:21:48.082 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:21:47.833199+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5297":{"gid":5297,"name":"b","rank":0,"incarnation":67,"state":"up:replay","state_seq":2,"addr":"172.21.15.47:6833/3025803398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3025803398},{"type":"v1","addr":"172.21.15.47:6833","nonce":3025803398}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:21:48.083 INFO:tasks.mds_thrash.fs.[None]:mds.a down, removed from mdsmap 2020-11-20T12:21:48.083 INFO:tasks.mds_thrash.fs.[None]:waiting for mds cluster to stabilize... 2020-11-20T12:21:48.083 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:48.213 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.206+0000 7fdf23fff700 1 Processor -- start 2020-11-20T12:21:48.213 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.206+0000 7fdf23fff700 1 -- start start 2020-11-20T12:21:48.215 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.206+0000 7fdf23fff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdf24060bc0 0x7fdf24061040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:48.216 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.206+0000 7fdf23fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf24061530 con 0x7fdf240623c0 2020-11-20T12:21:48.216 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.206+0000 7fdf23fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf24065ca0 con 0x7fdf24134de0 2020-11-20T12:21:48.216 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.206+0000 7fdf23fff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf240687e0 con 0x7fdf24060bc0 2020-11-20T12:21:48.216 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.206+0000 7fdf237fe700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fdf24134de0 0x7fdf24135240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53726/0 (socket says 172.21.15.189:53726) 2020-11-20T12:21:48.216 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.206+0000 7fdf237fe700 1 -- 172.21.15.189:0/1755363976 learned_addr learned my addr 172.21.15.189:0/1755363976 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:21:48.217 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.206+0000 7fdf227fc700 1 --2- 172.21.15.189:0/1755363976 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdf24060bc0 0x7fdf24061040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:48.217 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.206+0000 7fdf21ffb700 1 -- 172.21.15.189:0/1755363976 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079163280 0 0) 0x7fdf24065ca0 con 0x7fdf24134de0 2020-11-20T12:21:48.217 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.206+0000 7fdf21ffb700 1 -- 172.21.15.189:0/1755363976 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf10003240 con 0x7fdf24134de0 2020-11-20T12:21:48.217 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.210+0000 7fdf21ffb700 1 -- 172.21.15.189:0/1755363976 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202454706 0 0) 0x7fdf24061530 con 0x7fdf240623c0 2020-11-20T12:21:48.218 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.210+0000 7fdf21ffb700 1 -- 172.21.15.189:0/1755363976 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf24065ca0 con 0x7fdf240623c0 2020-11-20T12:21:48.218 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.210+0000 7fdf227fc700 1 -- 172.21.15.189:0/1755363976 >> v1:172.21.15.189:6789/0 conn(0x7fdf24134de0 legacy=0x7fdf24135240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:48.218 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.210+0000 7fdf227fc700 1 -- 172.21.15.189:0/1755363976 >> v1:172.21.15.47:6789/0 conn(0x7fdf240623c0 legacy=0x7fdf24138290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:48.218 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.210+0000 7fdf227fc700 1 -- 172.21.15.189:0/1755363976 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf24138ba0 con 0x7fdf24060bc0 2020-11-20T12:21:48.218 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.210+0000 7fdf227fc700 1 --2- 172.21.15.189:0/1755363976 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdf24060bc0 0x7fdf24061040 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fdf1800b9d0 tx=0x7fdf18006df0).ready entity=mon.2 client_cookie=ea0e5e3b826371eb server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:48.220 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.210+0000 7fdf21ffb700 1 -- 172.21.15.189:0/1755363976 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf18009710 con 0x7fdf24060bc0 2020-11-20T12:21:48.220 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.210+0000 7fdf21ffb700 1 -- 172.21.15.189:0/1755363976 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf18009870 con 0x7fdf24060bc0 2020-11-20T12:21:48.220 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.210+0000 7fdf21ffb700 1 -- 172.21.15.189:0/1755363976 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf18009bf0 con 0x7fdf24060bc0 2020-11-20T12:21:48.221 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.210+0000 7fdf23fff700 1 -- 172.21.15.189:0/1755363976 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdf24060bc0 msgr2=0x7fdf24061040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:48.221 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.210+0000 7fdf23fff700 1 --2- 172.21.15.189:0/1755363976 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdf24060bc0 0x7fdf24061040 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fdf1800b9d0 tx=0x7fdf18006df0).stop 2020-11-20T12:21:48.221 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.210+0000 7fdf23fff700 1 -- 172.21.15.189:0/1755363976 shutdown_connections 2020-11-20T12:21:48.221 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.210+0000 7fdf23fff700 1 --2- 172.21.15.189:0/1755363976 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdf24060bc0 0x7fdf24061040 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:48.221 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.210+0000 7fdf23fff700 1 -- 172.21.15.189:0/1755363976 >> 172.21.15.189:0/1755363976 conn(0x7fdf24065fe0 msgr2=0x7fdf24061b10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:48.222 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.210+0000 7fdf23fff700 1 -- 172.21.15.189:0/1755363976 shutdown_connections 2020-11-20T12:21:48.222 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.210+0000 7fdf23fff700 1 -- 172.21.15.189:0/1755363976 wait complete. 2020-11-20T12:21:48.222 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.210+0000 7fdf23fff700 1 Processor -- start 2020-11-20T12:21:48.222 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.210+0000 7fdf23fff700 1 -- start start 2020-11-20T12:21:48.222 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.214+0000 7fdf23fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf24060bc0 0x7fdf2413e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:48.223 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.214+0000 7fdf23fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf240623c0 0x7fdf2409c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:48.223 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.214+0000 7fdf23fff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdf24134de0 0x7fdf2414ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:48.223 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.214+0000 7fdf23fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2413d060 con 0x7fdf240623c0 2020-11-20T12:21:48.223 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.214+0000 7fdf23fff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2413d300 con 0x7fdf24060bc0 2020-11-20T12:21:48.223 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.214+0000 7fdf23fff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2413d5a0 con 0x7fdf24134de0 2020-11-20T12:21:48.225 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.214+0000 7fdf22ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf24060bc0 0x7fdf2413e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:48.226 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.214+0000 7fdf22ffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf24060bc0 0x7fdf2413e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52106/0 (socket says 172.21.15.189:52106) 2020-11-20T12:21:48.226 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.214+0000 7fdf22ffd700 1 -- 172.21.15.189:0/2456343197 learned_addr learned my addr 172.21.15.189:0/2456343197 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:21:48.226 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.214+0000 7fdf227fc700 1 --2- 172.21.15.189:0/2456343197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf240623c0 0x7fdf2409c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:48.226 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.214+0000 7fdf237fe700 1 --2- 172.21.15.189:0/2456343197 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdf24134de0 0x7fdf2414ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:48.227 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.214+0000 7fdf22ffd700 1 -- 172.21.15.189:0/2456343197 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdf24134de0 msgr2=0x7fdf2414ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:48.227 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.214+0000 7fdf22ffd700 1 --2- 172.21.15.189:0/2456343197 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdf24134de0 0x7fdf2414ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:48.227 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.214+0000 7fdf22ffd700 1 -- 172.21.15.189:0/2456343197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf240623c0 msgr2=0x7fdf2409c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:48.227 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.214+0000 7fdf22ffd700 1 --2- 172.21.15.189:0/2456343197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf240623c0 0x7fdf2409c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:48.227 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.214+0000 7fdf22ffd700 1 -- 172.21.15.189:0/2456343197 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2413c390 con 0x7fdf24060bc0 2020-11-20T12:21:48.228 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.214+0000 7fdf22ffd700 1 --2- 172.21.15.189:0/2456343197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf24060bc0 0x7fdf2413e550 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7fdf1400c710 tx=0x7fdf14007a90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:48.228 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.214+0000 7fdf0affd700 1 -- 172.21.15.189:0/2456343197 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1400e490 con 0x7fdf24060bc0 2020-11-20T12:21:48.228 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.214+0000 7fdf0affd700 1 -- 172.21.15.189:0/2456343197 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf1400e5f0 con 0x7fdf24060bc0 2020-11-20T12:21:48.228 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.214+0000 7fdf23fff700 1 -- 172.21.15.189:0/2456343197 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf2413ea40 con 0x7fdf24060bc0 2020-11-20T12:21:48.228 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.214+0000 7fdf0affd700 1 -- 172.21.15.189:0/2456343197 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1400e990 con 0x7fdf24060bc0 2020-11-20T12:21:48.229 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.214+0000 7fdf23fff700 1 -- 172.21.15.189:0/2456343197 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf2413da40 con 0x7fdf24060bc0 2020-11-20T12:21:48.229 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.218+0000 7fdf23fff700 1 -- 172.21.15.189:0/2456343197 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf24133e10 con 0x7fdf24060bc0 2020-11-20T12:21:48.229 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.218+0000 7fdf0affd700 1 -- 172.21.15.189:0/2456343197 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fdf14019030 con 0x7fdf24060bc0 2020-11-20T12:21:48.229 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.218+0000 7fdf0affd700 1 --2- 172.21.15.189:0/2456343197 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fdf000557c0 0x7fdf00057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:48.229 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.218+0000 7fdf0affd700 1 -- 172.21.15.189:0/2456343197 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7191+0+0 (secure 0 0 0) 0x7fdf1406df50 con 0x7fdf24060bc0 2020-11-20T12:21:48.230 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.218+0000 7fdf227fc700 1 --2- 172.21.15.189:0/2456343197 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fdf000557c0 0x7fdf00057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:48.230 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.218+0000 7fdf227fc700 1 --2- 172.21.15.189:0/2456343197 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fdf000557c0 0x7fdf00057c80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fdf18009fd0 tx=0x7fdf18015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:48.231 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.226+0000 7fdf0affd700 1 -- 172.21.15.189:0/2456343197 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fdf24133e10 con 0x7fdf24060bc0 2020-11-20T12:21:48.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.426+0000 7fdf23fff700 1 -- 172.21.15.189:0/2456343197 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf24133e10 con 0x7fdf24060bc0 2020-11-20T12:21:48.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.426+0000 7fdf0affd700 1 -- 172.21.15.189:0/2456343197 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+1784 (secure 0 0 0) 0x7fdf24133e10 con 0x7fdf24060bc0 2020-11-20T12:21:48.436 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:21:48.436 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:21:47.833199+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5297":{"gid":5297,"name":"b","rank":0,"incarnation":67,"state":"up:replay","state_seq":2,"addr":"172.21.15.47:6833/3025803398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3025803398},{"type":"v1","addr":"172.21.15.47:6833","nonce":3025803398}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:21:48.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.430+0000 7fdf23fff700 1 -- 172.21.15.189:0/2456343197 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fdf000557c0 msgr2=0x7fdf00057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:48.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.430+0000 7fdf23fff700 1 --2- 172.21.15.189:0/2456343197 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fdf000557c0 0x7fdf00057c80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fdf18009fd0 tx=0x7fdf18015000).stop 2020-11-20T12:21:48.438 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.430+0000 7fdf23fff700 1 -- 172.21.15.189:0/2456343197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf24060bc0 msgr2=0x7fdf2413e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:48.439 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.430+0000 7fdf23fff700 1 --2- 172.21.15.189:0/2456343197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf24060bc0 0x7fdf2413e550 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7fdf1400c710 tx=0x7fdf14007a90).stop 2020-11-20T12:21:48.439 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.430+0000 7fdf23fff700 1 -- 172.21.15.189:0/2456343197 shutdown_connections 2020-11-20T12:21:48.439 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.430+0000 7fdf23fff700 1 --2- 172.21.15.189:0/2456343197 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fdf24134de0 0x7fdf2414ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:48.440 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.430+0000 7fdf23fff700 1 --2- 172.21.15.189:0/2456343197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf24060bc0 0x7fdf2413e550 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:48.440 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.430+0000 7fdf23fff700 1 --2- 172.21.15.189:0/2456343197 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fdf000557c0 0x7fdf00057c80 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:48.440 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.430+0000 7fdf23fff700 1 --2- 172.21.15.189:0/2456343197 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fdf240623c0 0x7fdf2409c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:48.441 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.430+0000 7fdf23fff700 1 -- 172.21.15.189:0/2456343197 >> 172.21.15.189:0/2456343197 conn(0x7fdf24065fe0 msgr2=0x7fdf240616e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:48.441 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.430+0000 7fdf23fff700 1 -- 172.21.15.189:0/2456343197 shutdown_connections 2020-11-20T12:21:48.441 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:48.430+0000 7fdf23fff700 1 -- 172.21.15.189:0/2456343197 wait complete. 2020-11-20T12:21:48.442 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 67 2020-11-20T12:21:48.450 INFO:tasks.mds_thrash.fs.[None]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 67, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2020-11-20T12:04:54.012110+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 67, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5297": { "addr": "172.21.15.47:6833/3025803398", "addrs": { "addrvec": [ { "addr": "172.21.15.47:6832", "nonce": 3025803398, "type": "v2" }, { "addr": "172.21.15.47:6833", "nonce": 3025803398, "type": "v1" } ] }, "export_targets": [], "features": 4540138297136906239, "flags": 0, "gid": 5297, "incarnation": 67, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 35, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2020-11-20T12:21:47.833199+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5297 } } } ], "standbys": [] } 2020-11-20T12:21:50.452 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:50.599 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.590+0000 7f270a691700 1 Processor -- start 2020-11-20T12:21:50.600 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.594+0000 7f270a691700 1 -- start start 2020-11-20T12:21:50.601 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.594+0000 7f270a691700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f26fc0378a0 0x7f26fc0c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:50.601 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.594+0000 7f270a691700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f26fc037d50 0x7f26fc0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:50.601 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.594+0000 7f270a691700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26fc037440 con 0x7f26fc007e00 2020-11-20T12:21:50.601 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.594+0000 7f270a691700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26fc037250 con 0x7f26fc037d50 2020-11-20T12:21:50.602 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.594+0000 7f270a691700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26fc038510 con 0x7f26fc0378a0 2020-11-20T12:21:50.602 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.594+0000 7f27037fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f26fc0378a0 0x7f26fc0c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:50.602 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.594+0000 7f27037fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f26fc0378a0 0x7f26fc0c44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52116/0 (socket says 172.21.15.189:52116) 2020-11-20T12:21:50.602 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.594+0000 7f27037fe700 1 -- 172.21.15.189:0/2922380186 learned_addr learned my addr 172.21.15.189:0/2922380186 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:21:50.602 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.594+0000 7f2708c2e700 1 --2- 172.21.15.189:0/2922380186 >> v2:172.21.15.47:3300/0 conn(0x7f26fc037d50 0x7f26fc0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:50.605 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.594+0000 7f27037fe700 1 -- 172.21.15.189:0/2922380186 >> v2:172.21.15.47:3300/0 conn(0x7f26fc037d50 msgr2=0x7f26fc0c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:50.605 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.594+0000 7f27037fe700 1 --2- 172.21.15.189:0/2922380186 >> v2:172.21.15.47:3300/0 conn(0x7f26fc037d50 0x7f26fc0c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:50.605 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.594+0000 7f27037fe700 1 -- 172.21.15.189:0/2922380186 >> v1:172.21.15.47:6789/0 conn(0x7f26fc007e00 legacy=0x7f26fc0caf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:50.605 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.594+0000 7f27037fe700 1 -- 172.21.15.189:0/2922380186 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26fc036e20 con 0x7f26fc0378a0 2020-11-20T12:21:50.606 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.594+0000 7f27037fe700 1 --2- 172.21.15.189:0/2922380186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26fc0378a0 0x7f26fc0c44b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7f26fc0398a0 tx=0x7f26f4006eb0).ready entity=mon.1 client_cookie=959d48a23ba48b7c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:50.606 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.594+0000 7f2702ffd700 1 -- 172.21.15.189:0/2922380186 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f40098d0 con 0x7f26fc0378a0 2020-11-20T12:21:50.606 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.594+0000 7f2702ffd700 1 -- 172.21.15.189:0/2922380186 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f4009a30 con 0x7f26fc0378a0 2020-11-20T12:21:50.606 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.594+0000 7f2702ffd700 1 -- 172.21.15.189:0/2922380186 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f4009db0 con 0x7f26fc0378a0 2020-11-20T12:21:50.606 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.594+0000 7f270a691700 1 -- 172.21.15.189:0/2922380186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26fc0378a0 msgr2=0x7f26fc0c44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:50.607 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.594+0000 7f270a691700 1 --2- 172.21.15.189:0/2922380186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26fc0378a0 0x7f26fc0c44b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7f26fc0398a0 tx=0x7f26f4006eb0).stop 2020-11-20T12:21:50.607 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.594+0000 7f270a691700 1 -- 172.21.15.189:0/2922380186 shutdown_connections 2020-11-20T12:21:50.607 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.594+0000 7f270a691700 1 --2- 172.21.15.189:0/2922380186 >> v2:172.21.15.47:3300/0 conn(0x7f26fc037d50 0x7f26fc0c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:50.607 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.594+0000 7f270a691700 1 --2- 172.21.15.189:0/2922380186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26fc0378a0 0x7f26fc0c44b0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:50.607 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.594+0000 7f270a691700 1 -- 172.21.15.189:0/2922380186 >> 172.21.15.189:0/2922380186 conn(0x7f26fc00cd10 msgr2=0x7f26fc007550 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:50.608 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.594+0000 7f270a691700 1 -- 172.21.15.189:0/2922380186 shutdown_connections 2020-11-20T12:21:50.608 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.594+0000 7f270a691700 1 -- 172.21.15.189:0/2922380186 wait complete. 2020-11-20T12:21:50.610 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.598+0000 7f270a691700 1 Processor -- start 2020-11-20T12:21:50.610 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.598+0000 7f270a691700 1 -- start start 2020-11-20T12:21:50.610 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.598+0000 7f270a691700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26fc007e00 0x7f26fc0c9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:50.611 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.598+0000 7f270a691700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26fc0378a0 0x7f26fc0da270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:50.611 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.598+0000 7f270a691700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26fc037d50 0x7f26fc0da760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:50.611 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.598+0000 7f270a691700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc0dac50 con 0x7f26fc007e00 2020-11-20T12:21:50.611 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.598+0000 7f270a691700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc0ce650 con 0x7f26fc037d50 2020-11-20T12:21:50.611 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.598+0000 7f270a691700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc0ce8f0 con 0x7f26fc0378a0 2020-11-20T12:21:50.613 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.602+0000 7f2708c2e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26fc037d50 0x7f26fc0da760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:50.613 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.602+0000 7f2708c2e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26fc037d50 0x7f26fc0da760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52126/0 (socket says 172.21.15.189:52126) 2020-11-20T12:21:50.613 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.602+0000 7f2708c2e700 1 -- 172.21.15.189:0/4287429290 learned_addr learned my addr 172.21.15.189:0/4287429290 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:21:50.613 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.602+0000 7f27037fe700 1 --2- 172.21.15.189:0/4287429290 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26fc0378a0 0x7f26fc0da270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:50.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.602+0000 7f2703fff700 1 --2- 172.21.15.189:0/4287429290 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26fc007e00 0x7f26fc0c9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:50.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.602+0000 7f2708c2e700 1 -- 172.21.15.189:0/4287429290 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26fc0378a0 msgr2=0x7f26fc0da270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:50.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.602+0000 7f2708c2e700 1 --2- 172.21.15.189:0/4287429290 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26fc0378a0 0x7f26fc0da270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:50.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.602+0000 7f2708c2e700 1 -- 172.21.15.189:0/4287429290 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26fc007e00 msgr2=0x7f26fc0c9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:50.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.602+0000 7f2708c2e700 1 --2- 172.21.15.189:0/4287429290 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26fc007e00 0x7f26fc0c9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:50.615 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.602+0000 7f2708c2e700 1 -- 172.21.15.189:0/4287429290 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26fc0ca6a0 con 0x7f26fc037d50 2020-11-20T12:21:50.615 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.602+0000 7f2708c2e700 1 --2- 172.21.15.189:0/4287429290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26fc037d50 0x7f26fc0da760 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7f2704045400 tx=0x7f270405d940).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:50.615 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.602+0000 7f26e3fff700 1 -- 172.21.15.189:0/4287429290 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2704068780 con 0x7f26fc037d50 2020-11-20T12:21:50.615 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.602+0000 7f26e3fff700 1 -- 172.21.15.189:0/4287429290 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f270405fb40 con 0x7f26fc037d50 2020-11-20T12:21:50.615 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.602+0000 7f270a691700 1 -- 172.21.15.189:0/4287429290 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26fc0ce140 con 0x7f26fc037d50 2020-11-20T12:21:50.616 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.602+0000 7f26e3fff700 1 -- 172.21.15.189:0/4287429290 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f270405fec0 con 0x7f26fc037d50 2020-11-20T12:21:50.616 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.602+0000 7f270a691700 1 -- 172.21.15.189:0/4287429290 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26fc02be80 con 0x7f26fc037d50 2020-11-20T12:21:50.616 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.602+0000 7f270a691700 1 -- 172.21.15.189:0/4287429290 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26fc0da070 con 0x7f26fc037d50 2020-11-20T12:21:50.617 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.602+0000 7f26e3fff700 1 -- 172.21.15.189:0/4287429290 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f2704091070 con 0x7f26fc037d50 2020-11-20T12:21:50.617 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.602+0000 7f26e3fff700 1 --2- 172.21.15.189:0/4287429290 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f26dc0557c0 0x7f26dc057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:50.617 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.606+0000 7f26e3fff700 1 -- 172.21.15.189:0/4287429290 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7191+0+0 (secure 0 0 0) 0x7f27040de290 con 0x7f26fc037d50 2020-11-20T12:21:50.618 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.606+0000 7f2703fff700 1 --2- 172.21.15.189:0/4287429290 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f26dc0557c0 0x7f26dc057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:50.618 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.606+0000 7f2703fff700 1 --2- 172.21.15.189:0/4287429290 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f26dc0557c0 0x7f26dc057c80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f26f8003610 tx=0x7f26f8000b80).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:50.618 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.610+0000 7f26e3fff700 1 -- 172.21.15.189:0/4287429290 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f27040b3d80 con 0x7f26fc037d50 2020-11-20T12:21:50.820 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.814+0000 7f270a691700 1 -- 172.21.15.189:0/4287429290 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26fc005360 con 0x7f26fc037d50 2020-11-20T12:21:50.821 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.814+0000 7f26e3fff700 1 -- 172.21.15.189:0/4287429290 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+1785 (secure 0 0 0) 0x7f27040b73a0 con 0x7f26fc037d50 2020-11-20T12:21:50.821 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:21:50.821 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:21:49.846454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5297":{"gid":5297,"name":"b","rank":0,"incarnation":67,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.47:6833/3025803398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3025803398},{"type":"v1","addr":"172.21.15.47:6833","nonce":3025803398}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:21:50.827 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.814+0000 7f270a691700 1 -- 172.21.15.189:0/4287429290 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f26dc0557c0 msgr2=0x7f26dc057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:50.828 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.814+0000 7f270a691700 1 --2- 172.21.15.189:0/4287429290 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f26dc0557c0 0x7f26dc057c80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f26f8003610 tx=0x7f26f8000b80).stop 2020-11-20T12:21:50.828 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.814+0000 7f270a691700 1 -- 172.21.15.189:0/4287429290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26fc037d50 msgr2=0x7f26fc0da760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:50.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.814+0000 7f270a691700 1 --2- 172.21.15.189:0/4287429290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26fc037d50 0x7f26fc0da760 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7f2704045400 tx=0x7f270405d940).stop 2020-11-20T12:21:50.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.814+0000 7f270a691700 1 -- 172.21.15.189:0/4287429290 shutdown_connections 2020-11-20T12:21:50.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.814+0000 7f270a691700 1 --2- 172.21.15.189:0/4287429290 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f26dc0557c0 0x7f26dc057c80 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:50.830 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.814+0000 7f270a691700 1 --2- 172.21.15.189:0/4287429290 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f26fc007e00 0x7f26fc0c9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:50.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.814+0000 7f270a691700 1 --2- 172.21.15.189:0/4287429290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26fc037d50 0x7f26fc0da760 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:50.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.814+0000 7f270a691700 1 --2- 172.21.15.189:0/4287429290 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f26fc0378a0 0x7f26fc0da270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:50.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.814+0000 7f270a691700 1 -- 172.21.15.189:0/4287429290 >> 172.21.15.189:0/4287429290 conn(0x7f26fc00cd10 msgr2=0x7f26fc007550 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:50.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.814+0000 7f270a691700 1 -- 172.21.15.189:0/4287429290 shutdown_connections 2020-11-20T12:21:50.831 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:50.814+0000 7f270a691700 1 -- 172.21.15.189:0/4287429290 wait complete. 2020-11-20T12:21:50.832 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 69 2020-11-20T12:21:50.836 INFO:tasks.mds_thrash.fs.[None]:no change 2020-11-20T12:21:52.837 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:52.963 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.954+0000 7f606ffff700 1 Processor -- start 2020-11-20T12:21:52.963 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.954+0000 7f606ffff700 1 -- start start 2020-11-20T12:21:52.965 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606ffff700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f6070060310 0x7f60700a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:52.965 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606ffff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6070134460 0x7f60701348e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:52.965 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606ffff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f607013a900 0x7f607013ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:52.966 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606ffff700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6070067a00 con 0x7f6070060310 2020-11-20T12:21:52.966 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606ffff700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6070063b50 con 0x7f607013a900 2020-11-20T12:21:52.966 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606ffff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60700a09a0 con 0x7f6070134460 2020-11-20T12:21:52.966 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f607013a900 0x7f607013ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:52.966 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606f7fe700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f607013a900 0x7f607013ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52134/0 (socket says 172.21.15.189:52134) 2020-11-20T12:21:52.967 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606f7fe700 1 -- 172.21.15.189:0/543424553 learned_addr learned my addr 172.21.15.189:0/543424553 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:21:52.967 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606effd700 1 --2- 172.21.15.189:0/543424553 >> v2:172.21.15.47:3300/0 conn(0x7f6070060310 0x7f60700a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:52.967 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606e7fc700 1 --2- 172.21.15.189:0/543424553 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6070134460 0x7f60701348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:52.967 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606f7fe700 1 -- 172.21.15.189:0/543424553 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6070134460 msgr2=0x7f60701348e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:52.967 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606f7fe700 1 --2- 172.21.15.189:0/543424553 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6070134460 0x7f60701348e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:52.968 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606f7fe700 1 -- 172.21.15.189:0/543424553 >> v2:172.21.15.47:3300/0 conn(0x7f6070060310 msgr2=0x7f60700a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:52.968 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606f7fe700 1 --2- 172.21.15.189:0/543424553 >> v2:172.21.15.47:3300/0 conn(0x7f6070060310 0x7f60700a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:52.968 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606f7fe700 1 -- 172.21.15.189:0/543424553 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6070066500 con 0x7f607013a900 2020-11-20T12:21:52.968 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606effd700 1 --2- 172.21.15.189:0/543424553 >> v2:172.21.15.47:3300/0 conn(0x7f6070060310 0x7f60700a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:21:52.968 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606e7fc700 1 --2- 172.21.15.189:0/543424553 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6070134460 0x7f60701348e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:21:52.969 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606f7fe700 1 --2- 172.21.15.189:0/543424553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f607013a900 0x7f607013ad60 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7f60700a3b00 tx=0x7f6068009290).ready entity=mon.1 client_cookie=157db2bd08ccb954 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:52.969 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606dffb700 1 -- 172.21.15.189:0/543424553 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606800bbc0 con 0x7f607013a900 2020-11-20T12:21:52.969 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606dffb700 1 -- 172.21.15.189:0/543424553 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f606800bd20 con 0x7f607013a900 2020-11-20T12:21:52.969 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606dffb700 1 -- 172.21.15.189:0/543424553 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6068011610 con 0x7f607013a900 2020-11-20T12:21:52.972 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606ffff700 1 -- 172.21.15.189:0/543424553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f607013a900 msgr2=0x7f607013ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:52.972 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606ffff700 1 --2- 172.21.15.189:0/543424553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f607013a900 0x7f607013ad60 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7f60700a3b00 tx=0x7f6068009290).stop 2020-11-20T12:21:52.972 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606ffff700 1 -- 172.21.15.189:0/543424553 shutdown_connections 2020-11-20T12:21:52.973 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606ffff700 1 --2- 172.21.15.189:0/543424553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f607013a900 0x7f607013ad60 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:52.973 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606ffff700 1 --2- 172.21.15.189:0/543424553 >> v2:172.21.15.47:3300/0 conn(0x7f6070060310 0x7f60700a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:52.973 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606ffff700 1 --2- 172.21.15.189:0/543424553 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f6070134460 0x7f60701348e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:52.973 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606ffff700 1 -- 172.21.15.189:0/543424553 >> 172.21.15.189:0/543424553 conn(0x7f6070063210 msgr2=0x7f6070060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:52.974 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606ffff700 1 -- 172.21.15.189:0/543424553 shutdown_connections 2020-11-20T12:21:52.974 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.958+0000 7f606ffff700 1 -- 172.21.15.189:0/543424553 wait complete. 2020-11-20T12:21:52.974 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.962+0000 7f606ffff700 1 Processor -- start 2020-11-20T12:21:52.974 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.962+0000 7f606ffff700 1 -- start start 2020-11-20T12:21:52.974 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.962+0000 7f606ffff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6070060310 0x7f60700a5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:52.975 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.962+0000 7f606ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6070134460 0x7f607012f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:52.975 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.962+0000 7f606ffff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f607013a900 0x7f607012fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:52.975 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.962+0000 7f606ffff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60700a4cd0 con 0x7f6070060310 2020-11-20T12:21:52.975 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.962+0000 7f606ffff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60700a4f70 con 0x7f6070134460 2020-11-20T12:21:52.975 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.962+0000 7f606ffff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60700a5210 con 0x7f607013a900 2020-11-20T12:21:52.976 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.962+0000 7f606e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6070134460 0x7f607012f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:52.976 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.962+0000 7f606e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6070134460 0x7f607012f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52138/0 (socket says 172.21.15.189:52138) 2020-11-20T12:21:52.976 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.962+0000 7f606e7fc700 1 -- 172.21.15.189:0/1889318323 learned_addr learned my addr 172.21.15.189:0/1889318323 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:21:52.976 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.962+0000 7f606effd700 1 --2- 172.21.15.189:0/1889318323 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6070060310 0x7f60700a5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:52.976 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.962+0000 7f606f7fe700 1 --2- 172.21.15.189:0/1889318323 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f607013a900 0x7f607012fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:52.977 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.962+0000 7f606e7fc700 1 -- 172.21.15.189:0/1889318323 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f607013a900 msgr2=0x7f607012fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:52.977 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.962+0000 7f606e7fc700 1 --2- 172.21.15.189:0/1889318323 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f607013a900 0x7f607012fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:52.977 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.962+0000 7f606e7fc700 1 -- 172.21.15.189:0/1889318323 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6070060310 msgr2=0x7f60700a5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:52.977 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.962+0000 7f606e7fc700 1 --2- 172.21.15.189:0/1889318323 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6070060310 0x7f60700a5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:52.977 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.962+0000 7f606e7fc700 1 -- 172.21.15.189:0/1889318323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60700a5a30 con 0x7f6070134460 2020-11-20T12:21:52.978 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.962+0000 7f606e7fc700 1 --2- 172.21.15.189:0/1889318323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6070134460 0x7f607012f640 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7f606400db60 tx=0x7f6064008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:52.978 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.962+0000 7f6056ffd700 1 -- 172.21.15.189:0/1889318323 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606400be30 con 0x7f6070134460 2020-11-20T12:21:52.978 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.962+0000 7f6056ffd700 1 -- 172.21.15.189:0/1889318323 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6064009790 con 0x7f6070134460 2020-11-20T12:21:52.978 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.962+0000 7f606ffff700 1 -- 172.21.15.189:0/1889318323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60700a4830 con 0x7f6070134460 2020-11-20T12:21:52.978 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.962+0000 7f6056ffd700 1 -- 172.21.15.189:0/1889318323 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6064009b30 con 0x7f6070134460 2020-11-20T12:21:52.979 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.962+0000 7f606ffff700 1 -- 172.21.15.189:0/1889318323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6070130940 con 0x7f6070134460 2020-11-20T12:21:52.979 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.962+0000 7f606ffff700 1 -- 172.21.15.189:0/1889318323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f607013b780 con 0x7f6070134460 2020-11-20T12:21:52.979 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.966+0000 7f6056ffd700 1 -- 172.21.15.189:0/1889318323 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f6064013480 con 0x7f6070134460 2020-11-20T12:21:52.979 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.966+0000 7f6056ffd700 1 --2- 172.21.15.189:0/1889318323 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f604c0557c0 0x7f604c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:52.979 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.966+0000 7f6056ffd700 1 -- 172.21.15.189:0/1889318323 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7191+0+0 (secure 0 0 0) 0x7f606406efd0 con 0x7f6070134460 2020-11-20T12:21:52.980 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.966+0000 7f606effd700 1 --2- 172.21.15.189:0/1889318323 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f604c0557c0 0x7f604c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:52.980 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.966+0000 7f606effd700 1 --2- 172.21.15.189:0/1889318323 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f604c0557c0 0x7f604c057c80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f606000ba80 tx=0x7f6060008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:52.980 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:52.970+0000 7f6056ffd700 1 -- 172.21.15.189:0/1889318323 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f60640449b0 con 0x7f6070134460 2020-11-20T12:21:53.211 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:53.202+0000 7f606ffff700 1 -- 172.21.15.189:0/1889318323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60700446d0 con 0x7f6070134460 2020-11-20T12:21:53.212 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:53.202+0000 7f6056ffd700 1 -- 172.21.15.189:0/1889318323 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+1785 (secure 0 0 0) 0x7f6064047fd0 con 0x7f6070134460 2020-11-20T12:21:53.213 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:21:53.213 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:21:50.854972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5297":{"gid":5297,"name":"b","rank":0,"incarnation":67,"state":"up:active","state_seq":14,"addr":"172.21.15.47:6833/3025803398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3025803398},{"type":"v1","addr":"172.21.15.47:6833","nonce":3025803398}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:21:53.219 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:53.206+0000 7f606ffff700 1 -- 172.21.15.189:0/1889318323 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f604c0557c0 msgr2=0x7f604c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:53.219 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:53.206+0000 7f606ffff700 1 --2- 172.21.15.189:0/1889318323 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f604c0557c0 0x7f604c057c80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f606000ba80 tx=0x7f6060008040).stop 2020-11-20T12:21:53.220 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:53.206+0000 7f606ffff700 1 -- 172.21.15.189:0/1889318323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6070134460 msgr2=0x7f607012f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:53.220 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:53.206+0000 7f606ffff700 1 --2- 172.21.15.189:0/1889318323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6070134460 0x7f607012f640 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7f606400db60 tx=0x7f6064008040).stop 2020-11-20T12:21:53.222 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:53.206+0000 7f606ffff700 1 -- 172.21.15.189:0/1889318323 shutdown_connections 2020-11-20T12:21:53.223 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:53.206+0000 7f606ffff700 1 --2- 172.21.15.189:0/1889318323 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f604c0557c0 0x7f604c057c80 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:53.223 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:53.206+0000 7f606ffff700 1 --2- 172.21.15.189:0/1889318323 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f6070060310 0x7f60700a5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:53.224 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:53.206+0000 7f606ffff700 1 --2- 172.21.15.189:0/1889318323 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f607013a900 0x7f607012fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:53.224 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:53.206+0000 7f606ffff700 1 --2- 172.21.15.189:0/1889318323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6070134460 0x7f607012f640 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:53.225 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:53.206+0000 7f606ffff700 1 -- 172.21.15.189:0/1889318323 >> 172.21.15.189:0/1889318323 conn(0x7f6070063210 msgr2=0x7f6070068030 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:53.225 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:53.206+0000 7f606ffff700 1 -- 172.21.15.189:0/1889318323 shutdown_connections 2020-11-20T12:21:53.226 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:53.206+0000 7f606ffff700 1 -- 172.21.15.189:0/1889318323 wait complete. 2020-11-20T12:21:53.226 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 70 2020-11-20T12:21:53.229 INFO:tasks.mds_thrash.fs.[None]:mds.b has gained rank=0, replacing gid=5746 2020-11-20T12:21:53.230 INFO:tasks.mds_thrash.fs.[None]:waiting for 2 secs before reviving mds.a 2020-11-20T12:21:54.111 INFO:tasks.ceph.mds.b.smithi047.stderr:2020-11-20T12:21:54.101+0000 7f1226007700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T12:21:55.231 INFO:tasks.mds_thrash.fs.[None]:reviving mds.a 2020-11-20T12:21:55.232 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T12:21:55.232 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T12:21:55.240 INFO:tasks.ceph.mds.a:Started 2020-11-20T12:21:55.241 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:55.319 INFO:tasks.ceph.mds.a.smithi189.stdout:starting mds.a at 2020-11-20T12:21:55.372 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4f59e700 1 Processor -- start 2020-11-20T12:21:55.372 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4f59e700 1 -- start start 2020-11-20T12:21:55.380 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4f59e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2f5013dc20 0x7f2f50061180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:55.381 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4f59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f50066ee0 con 0x7f2f500a0a90 2020-11-20T12:21:55.381 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4f59e700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f50067180 con 0x7f2f500a9cc0 2020-11-20T12:21:55.382 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4f59e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f500a96e0 con 0x7f2f5013dc20 2020-11-20T12:21:55.382 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4ed9d700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2f500a9cc0 0x7f2f500aa150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53782/0 (socket says 172.21.15.189:53782) 2020-11-20T12:21:55.382 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4ed9d700 1 -- 172.21.15.189:0/404514858 learned_addr learned my addr 172.21.15.189:0/404514858 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:21:55.383 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f47fff700 1 --2- 172.21.15.189:0/404514858 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2f5013dc20 0x7f2f50061180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:55.383 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4dd9b700 1 -- 172.21.15.189:0/404514858 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076866507 0 0) 0x7f2f50067180 con 0x7f2f500a9cc0 2020-11-20T12:21:55.383 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4dd9b700 1 -- 172.21.15.189:0/404514858 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f30003240 con 0x7f2f500a9cc0 2020-11-20T12:21:55.384 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4dd9b700 1 -- 172.21.15.189:0/404514858 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 451961764 0 0) 0x7f2f50066ee0 con 0x7f2f500a0a90 2020-11-20T12:21:55.384 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4dd9b700 1 -- 172.21.15.189:0/404514858 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f50067180 con 0x7f2f500a0a90 2020-11-20T12:21:55.384 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4dd9b700 1 -- 172.21.15.189:0/404514858 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1276523203 0 0) 0x7f2f30003240 con 0x7f2f500a9cc0 2020-11-20T12:21:55.385 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4dd9b700 1 -- 172.21.15.189:0/404514858 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2f5013dc20 msgr2=0x7f2f50061180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:55.385 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4dd9b700 1 --2- 172.21.15.189:0/404514858 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2f5013dc20 0x7f2f50061180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:55.385 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4dd9b700 1 -- 172.21.15.189:0/404514858 >> v1:172.21.15.47:6789/0 conn(0x7f2f500a0a90 legacy=0x7f2f500a8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:55.386 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4dd9b700 1 -- 172.21.15.189:0/404514858 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f500a0e70 con 0x7f2f500a9cc0 2020-11-20T12:21:55.386 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4dd9b700 1 -- 172.21.15.189:0/404514858 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f2f48001860 con 0x7f2f500a9cc0 2020-11-20T12:21:55.386 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4dd9b700 1 -- 172.21.15.189:0/404514858 >> v1:172.21.15.189:6789/0 conn(0x7f2f500a9cc0 legacy=0x7f2f500aa150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:55.387 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4dd9b700 1 --2- 172.21.15.189:0/404514858 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2f5013dc20 0x7f2f300038c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:55.387 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4dd9b700 1 --2- 172.21.15.189:0/404514858 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2f30003db0 0x7f2f300044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:55.387 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4dd9b700 1 --2- 172.21.15.189:0/404514858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f300049d0 0x7f2f30004e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:55.387 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4dd9b700 1 -- 172.21.15.189:0/404514858 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f500a96e0 con 0x7f2f30003db0 2020-11-20T12:21:55.388 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4dd9b700 1 -- 172.21.15.189:0/404514858 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f30005340 con 0x7f2f300049d0 2020-11-20T12:21:55.388 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4dd9b700 1 -- 172.21.15.189:0/404514858 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3000b760 con 0x7f2f5013dc20 2020-11-20T12:21:55.388 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f47fff700 1 --2- 172.21.15.189:0/404514858 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2f30003db0 0x7f2f300044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:55.389 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4ed9d700 1 --2- 172.21.15.189:0/404514858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f300049d0 0x7f2f30004e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:55.389 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4e59c700 1 --2- 172.21.15.189:0/404514858 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2f5013dc20 0x7f2f300038c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:55.389 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4ed9d700 1 -- 172.21.15.189:0/404514858 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2f5013dc20 msgr2=0x7f2f300038c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:55.390 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4ed9d700 1 --2- 172.21.15.189:0/404514858 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2f5013dc20 0x7f2f300038c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:55.390 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4ed9d700 1 -- 172.21.15.189:0/404514858 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2f30003db0 msgr2=0x7f2f300044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:55.390 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4ed9d700 1 --2- 172.21.15.189:0/404514858 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2f30003db0 0x7f2f300044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:55.390 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4ed9d700 1 -- 172.21.15.189:0/404514858 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f30003240 con 0x7f2f300049d0 2020-11-20T12:21:55.391 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4e59c700 1 --2- 172.21.15.189:0/404514858 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2f5013dc20 0x7f2f300038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:21:55.391 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4ed9d700 1 --2- 172.21.15.189:0/404514858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f300049d0 0x7f2f30004e50 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7f2f4800c5e0 tx=0x7f2f50067070).ready entity=mon.1 client_cookie=a74a7d61f9b41d1e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:55.391 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4dd9b700 1 -- 172.21.15.189:0/404514858 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f48005b80 con 0x7f2f300049d0 2020-11-20T12:21:55.392 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4dd9b700 1 -- 172.21.15.189:0/404514858 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f48005ce0 con 0x7f2f300049d0 2020-11-20T12:21:55.392 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.366+0000 7f2f4dd9b700 1 -- 172.21.15.189:0/404514858 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f48007300 con 0x7f2f300049d0 2020-11-20T12:21:55.392 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.370+0000 7f2f4f59e700 1 -- 172.21.15.189:0/404514858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f300049d0 msgr2=0x7f2f30004e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:55.393 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.370+0000 7f2f4f59e700 1 --2- 172.21.15.189:0/404514858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f300049d0 0x7f2f30004e50 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7f2f4800c5e0 tx=0x7f2f50067070).stop 2020-11-20T12:21:55.393 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.370+0000 7f2f4ed9d700 1 -- 172.21.15.189:0/404514858 reap_dead start 2020-11-20T12:21:55.393 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.370+0000 7f2f4f59e700 1 -- 172.21.15.189:0/404514858 shutdown_connections 2020-11-20T12:21:55.394 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.370+0000 7f2f4f59e700 1 -- 172.21.15.189:0/404514858 >> 172.21.15.189:0/404514858 conn(0x7f2f50063210 msgr2=0x7f2f50060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:55.394 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.370+0000 7f2f4f59e700 1 -- 172.21.15.189:0/404514858 shutdown_connections 2020-11-20T12:21:55.394 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.370+0000 7f2f4f59e700 1 -- 172.21.15.189:0/404514858 wait complete. 2020-11-20T12:21:55.395 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.370+0000 7f2f4f59e700 1 Processor -- start 2020-11-20T12:21:55.395 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.370+0000 7f2f4f59e700 1 -- start start 2020-11-20T12:21:55.395 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.370+0000 7f2f4f59e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2f500a3c40 0x7f2f5013ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:55.395 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.370+0000 7f2f4f59e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f500a10f0 0x7f2f50135850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:55.396 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.370+0000 7f2f4f59e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2f500a35f0 0x7f2f50135d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:55.396 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.370+0000 7f2f4f59e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f500a1c40 con 0x7f2f500a3c40 2020-11-20T12:21:55.396 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.370+0000 7f2f4f59e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5013cf10 con 0x7f2f500a10f0 2020-11-20T12:21:55.397 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.370+0000 7f2f4f59e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f500a19a0 con 0x7f2f500a35f0 2020-11-20T12:21:55.397 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.370+0000 7f2f47fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f500a10f0 0x7f2f50135850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:55.397 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.370+0000 7f2f47fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f500a10f0 0x7f2f50135850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52170/0 (socket says 172.21.15.189:52170) 2020-11-20T12:21:55.398 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.370+0000 7f2f47fff700 1 -- 172.21.15.189:0/853254011 learned_addr learned my addr 172.21.15.189:0/853254011 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:21:55.398 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.370+0000 7f2f4e59c700 1 --2- 172.21.15.189:0/853254011 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2f500a3c40 0x7f2f5013ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:55.398 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.370+0000 7f2f4ed9d700 1 --2- 172.21.15.189:0/853254011 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2f500a35f0 0x7f2f50135d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:55.399 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.370+0000 7f2f47fff700 1 -- 172.21.15.189:0/853254011 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2f500a35f0 msgr2=0x7f2f50135d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:55.399 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.370+0000 7f2f47fff700 1 --2- 172.21.15.189:0/853254011 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2f500a35f0 0x7f2f50135d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:55.399 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.374+0000 7f2f47fff700 1 -- 172.21.15.189:0/853254011 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2f500a3c40 msgr2=0x7f2f5013ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:55.400 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.374+0000 7f2f47fff700 1 --2- 172.21.15.189:0/853254011 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2f500a3c40 0x7f2f5013ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:55.400 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.374+0000 7f2f47fff700 1 -- 172.21.15.189:0/853254011 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f500a3150 con 0x7f2f500a10f0 2020-11-20T12:21:55.400 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.374+0000 7f2f4ed9d700 1 --2- 172.21.15.189:0/853254011 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2f500a35f0 0x7f2f50135d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:21:55.400 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.374+0000 7f2f47fff700 1 --2- 172.21.15.189:0/853254011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f500a10f0 0x7f2f50135850 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f2f38011cf0 tx=0x7f2f38006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:55.401 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.374+0000 7f2f467fc700 1 -- 172.21.15.189:0/853254011 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3800f9e0 con 0x7f2f500a10f0 2020-11-20T12:21:55.401 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.374+0000 7f2f467fc700 1 -- 172.21.15.189:0/853254011 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3800fb40 con 0x7f2f500a10f0 2020-11-20T12:21:55.401 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.374+0000 7f2f4f59e700 1 -- 172.21.15.189:0/853254011 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f500a4020 con 0x7f2f500a10f0 2020-11-20T12:21:55.402 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.374+0000 7f2f467fc700 1 -- 172.21.15.189:0/853254011 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3800fec0 con 0x7f2f500a10f0 2020-11-20T12:21:55.402 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.374+0000 7f2f4f59e700 1 -- 172.21.15.189:0/853254011 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f50135490 con 0x7f2f500a10f0 2020-11-20T12:21:55.404 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.374+0000 7f2f4f59e700 1 -- 172.21.15.189:0/853254011 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f500a32e0 con 0x7f2f500a10f0 2020-11-20T12:21:55.404 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.374+0000 7f2f467fc700 1 -- 172.21.15.189:0/853254011 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f2f38014650 con 0x7f2f500a10f0 2020-11-20T12:21:55.404 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.374+0000 7f2f467fc700 1 --2- 172.21.15.189:0/853254011 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2f2c0557c0 0x7f2f2c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:55.405 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.374+0000 7f2f467fc700 1 -- 172.21.15.189:0/853254011 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7191+0+0 (secure 0 0 0) 0x7f2f3806f5b0 con 0x7f2f500a10f0 2020-11-20T12:21:55.405 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.374+0000 7f2f4e59c700 1 --2- 172.21.15.189:0/853254011 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2f2c0557c0 0x7f2f2c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:55.405 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.378+0000 7f2f4e59c700 1 --2- 172.21.15.189:0/853254011 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2f2c0557c0 0x7f2f2c057c80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f2f500a28a0 tx=0x7f2f4000c040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:55.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.378+0000 7f2f467fc700 1 -- 172.21.15.189:0/853254011 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2f38044ff0 con 0x7f2f500a10f0 2020-11-20T12:21:55.609 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.601+0000 7f2f4f59e700 1 -- 172.21.15.189:0/853254011 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f50066af0 con 0x7f2f500a10f0 2020-11-20T12:21:55.609 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.601+0000 7f2f467fc700 1 -- 172.21.15.189:0/853254011 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+1785 (secure 0 0 0) 0x7f2f38048610 con 0x7f2f500a10f0 2020-11-20T12:21:55.610 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:21:55.610 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:21:50.854972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5297":{"gid":5297,"name":"b","rank":0,"incarnation":67,"state":"up:active","state_seq":14,"addr":"172.21.15.47:6833/3025803398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3025803398},{"type":"v1","addr":"172.21.15.47:6833","nonce":3025803398}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:21:55.613 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.605+0000 7f2f4f59e700 1 -- 172.21.15.189:0/853254011 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2f2c0557c0 msgr2=0x7f2f2c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:55.613 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.605+0000 7f2f4f59e700 1 --2- 172.21.15.189:0/853254011 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2f2c0557c0 0x7f2f2c057c80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f2f500a28a0 tx=0x7f2f4000c040).stop 2020-11-20T12:21:55.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.605+0000 7f2f4f59e700 1 -- 172.21.15.189:0/853254011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f500a10f0 msgr2=0x7f2f50135850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:55.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.605+0000 7f2f4f59e700 1 --2- 172.21.15.189:0/853254011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f500a10f0 0x7f2f50135850 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f2f38011cf0 tx=0x7f2f38006df0).stop 2020-11-20T12:21:55.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.605+0000 7f2f4f59e700 1 -- 172.21.15.189:0/853254011 shutdown_connections 2020-11-20T12:21:55.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.605+0000 7f2f4f59e700 1 --2- 172.21.15.189:0/853254011 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f2f2c0557c0 0x7f2f2c057c80 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:55.614 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.605+0000 7f2f4f59e700 1 --2- 172.21.15.189:0/853254011 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f2f500a3c40 0x7f2f5013ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:55.615 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.605+0000 7f2f4f59e700 1 --2- 172.21.15.189:0/853254011 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f2f500a35f0 0x7f2f50135d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:55.615 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.605+0000 7f2f4f59e700 1 --2- 172.21.15.189:0/853254011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f500a10f0 0x7f2f50135850 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:55.615 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.605+0000 7f2f4f59e700 1 -- 172.21.15.189:0/853254011 >> 172.21.15.189:0/853254011 conn(0x7f2f50063210 msgr2=0x7f2f5005ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:55.615 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.605+0000 7f2f4f59e700 1 -- 172.21.15.189:0/853254011 shutdown_connections 2020-11-20T12:21:55.615 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:55.605+0000 7f2f4f59e700 1 -- 172.21.15.189:0/853254011 wait complete. 2020-11-20T12:21:55.616 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 70 2020-11-20T12:21:55.631 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2020-11-20T12:21:56.313 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:21:56.316 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:21:57.632 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:21:57.770 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.765+0000 7f389938e700 1 Processor -- start 2020-11-20T12:21:57.771 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.765+0000 7f389938e700 1 -- start start 2020-11-20T12:21:57.773 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.765+0000 7f389938e700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f38940acd30 0x7f389413a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:57.773 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f389938e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f38940672f0 0x7f38940a9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:57.773 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f389938e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38940669f0 con 0x7f38940ac950 2020-11-20T12:21:57.773 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f389938e700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f389413ac00 con 0x7f38940acd30 2020-11-20T12:21:57.774 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f389938e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f389413aea0 con 0x7f38940672f0 2020-11-20T12:21:57.774 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f3898b8d700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f38940672f0 0x7f38940a9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:57.774 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f38937fe700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f38940acd30 0x7f389413a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:57.774 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f3893fff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f38940ac950 0x7f38940a8e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.189:48462/0 (socket says 172.21.15.189:48462) 2020-11-20T12:21:57.774 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f3898b8d700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f38940672f0 0x7f38940a9940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.189:45214/0 (socket says 172.21.15.189:45214) 2020-11-20T12:21:57.775 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f3893fff700 1 -- 172.21.15.189:0/1440096166 learned_addr learned my addr 172.21.15.189:0/1440096166 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:21:57.775 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f3892ffd700 1 -- 172.21.15.189:0/1440096166 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 151217925 0 0) 0x7f38940669f0 con 0x7f38940ac950 2020-11-20T12:21:57.775 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f3892ffd700 1 -- 172.21.15.189:0/1440096166 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3874003240 con 0x7f38940ac950 2020-11-20T12:21:57.775 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f3898b8d700 1 -- 172.21.15.189:0/1440096166 >> v2:172.21.15.47:3300/0 conn(0x7f38940acd30 msgr2=0x7f389413a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:21:57.775 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f3898b8d700 1 --2- 172.21.15.189:0/1440096166 >> v2:172.21.15.47:3300/0 conn(0x7f38940acd30 0x7f389413a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:57.776 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f3898b8d700 1 -- 172.21.15.189:0/1440096166 >> v1:172.21.15.47:6789/0 conn(0x7f38940ac950 legacy=0x7f38940a8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:57.776 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f3898b8d700 1 -- 172.21.15.189:0/1440096166 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389413b140 con 0x7f38940672f0 2020-11-20T12:21:57.776 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f38937fe700 1 --2- 172.21.15.189:0/1440096166 >> v2:172.21.15.47:3300/0 conn(0x7f38940acd30 0x7f389413a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:21:57.776 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f3898b8d700 1 --2- 172.21.15.189:0/1440096166 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f38940672f0 0x7f38940a9940 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f388c009c90 tx=0x7f388c0091d0).ready entity=mon.2 client_cookie=ce81e5e72a350e16 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:57.779 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f3892ffd700 1 -- 172.21.15.189:0/1440096166 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c00b9c0 con 0x7f38940672f0 2020-11-20T12:21:57.779 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f3892ffd700 1 -- 172.21.15.189:0/1440096166 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f388c00bb20 con 0x7f38940672f0 2020-11-20T12:21:57.779 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f3892ffd700 1 -- 172.21.15.189:0/1440096166 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c00bea0 con 0x7f38940672f0 2020-11-20T12:21:57.780 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f389938e700 1 -- 172.21.15.189:0/1440096166 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f38940672f0 msgr2=0x7f38940a9940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:57.780 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f389938e700 1 --2- 172.21.15.189:0/1440096166 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f38940672f0 0x7f38940a9940 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f388c009c90 tx=0x7f388c0091d0).stop 2020-11-20T12:21:57.780 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f389938e700 1 -- 172.21.15.189:0/1440096166 shutdown_connections 2020-11-20T12:21:57.780 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f389938e700 1 --2- 172.21.15.189:0/1440096166 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f38940672f0 0x7f38940a9940 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:57.781 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f389938e700 1 --2- 172.21.15.189:0/1440096166 >> v2:172.21.15.47:3300/0 conn(0x7f38940acd30 0x7f389413a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:57.781 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f389938e700 1 -- 172.21.15.189:0/1440096166 >> 172.21.15.189:0/1440096166 conn(0x7f3894062080 msgr2=0x7f3894062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:57.781 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f389938e700 1 -- 172.21.15.189:0/1440096166 shutdown_connections 2020-11-20T12:21:57.781 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.769+0000 7f389938e700 1 -- 172.21.15.189:0/1440096166 wait complete. 2020-11-20T12:21:57.781 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.773+0000 7f389938e700 1 Processor -- start 2020-11-20T12:21:57.782 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.773+0000 7f389938e700 1 -- start start 2020-11-20T12:21:57.782 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.773+0000 7f389938e700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38940672f0 0x7f389413ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:57.782 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.773+0000 7f389938e700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f38940ac950 0x7f38940a6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:57.782 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.773+0000 7f389938e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38940acd30 0x7f3894149930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:57.782 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.773+0000 7f389938e700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3894138420 con 0x7f38940672f0 2020-11-20T12:21:57.783 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.773+0000 7f389938e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38940a5d20 con 0x7f38940acd30 2020-11-20T12:21:57.783 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.773+0000 7f389938e700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38940a5340 con 0x7f38940ac950 2020-11-20T12:21:57.783 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.773+0000 7f3898b8d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38940acd30 0x7f3894149930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:57.783 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.773+0000 7f3893fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38940672f0 0x7f389413ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:57.783 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.773+0000 7f3898b8d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38940acd30 0x7f3894149930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52186/0 (socket says 172.21.15.189:52186) 2020-11-20T12:21:57.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.773+0000 7f3898b8d700 1 -- 172.21.15.189:0/2556801571 learned_addr learned my addr 172.21.15.189:0/2556801571 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:21:57.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.773+0000 7f38937fe700 1 --2- 172.21.15.189:0/2556801571 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f38940ac950 0x7f38940a6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:57.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.773+0000 7f3898b8d700 1 -- 172.21.15.189:0/2556801571 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f38940ac950 msgr2=0x7f38940a6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:57.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.773+0000 7f3898b8d700 1 --2- 172.21.15.189:0/2556801571 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f38940ac950 0x7f38940a6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:57.784 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.773+0000 7f3898b8d700 1 -- 172.21.15.189:0/2556801571 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38940672f0 msgr2=0x7f389413ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:57.785 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.773+0000 7f3898b8d700 1 --2- 172.21.15.189:0/2556801571 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38940672f0 0x7f389413ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:57.785 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.773+0000 7f3898b8d700 1 -- 172.21.15.189:0/2556801571 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38940a55e0 con 0x7f38940acd30 2020-11-20T12:21:57.785 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.773+0000 7f3898b8d700 1 --2- 172.21.15.189:0/2556801571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38940acd30 0x7f3894149930 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7f388c003340 tx=0x7f388c016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:57.785 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.773+0000 7f388bfff700 1 -- 172.21.15.189:0/2556801571 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c01c6c0 con 0x7f38940acd30 2020-11-20T12:21:57.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.773+0000 7f388bfff700 1 -- 172.21.15.189:0/2556801571 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f388c01c820 con 0x7f38940acd30 2020-11-20T12:21:57.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.773+0000 7f389938e700 1 -- 172.21.15.189:0/2556801571 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38940a5880 con 0x7f38940acd30 2020-11-20T12:21:57.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.773+0000 7f388bfff700 1 -- 172.21.15.189:0/2556801571 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c01cba0 con 0x7f38940acd30 2020-11-20T12:21:57.786 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.773+0000 7f389938e700 1 -- 172.21.15.189:0/2556801571 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38940a6b50 con 0x7f38940acd30 2020-11-20T12:21:57.787 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.777+0000 7f389938e700 1 -- 172.21.15.189:0/2556801571 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38940a5a10 con 0x7f38940acd30 2020-11-20T12:21:57.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.777+0000 7f388bfff700 1 -- 172.21.15.189:0/2556801571 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f388c017070 con 0x7f38940acd30 2020-11-20T12:21:57.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.777+0000 7f388bfff700 1 --2- 172.21.15.189:0/2556801571 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f38700557c0 0x7f3870057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:21:57.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.777+0000 7f388bfff700 1 -- 172.21.15.189:0/2556801571 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7191+0+0 (secure 0 0 0) 0x7f388c06fe80 con 0x7f38940acd30 2020-11-20T12:21:57.788 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.777+0000 7f3893fff700 1 --2- 172.21.15.189:0/2556801571 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f38700557c0 0x7f3870057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:21:57.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.777+0000 7f3893fff700 1 --2- 172.21.15.189:0/2556801571 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f38700557c0 0x7f3870057c80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f38840055a0 tx=0x7f3884009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:21:57.789 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:57.781+0000 7f388bfff700 1 -- 172.21.15.189:0/2556801571 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f388c0459f0 con 0x7f38940acd30 2020-11-20T12:21:58.016 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:58.013+0000 7f389938e700 1 -- 172.21.15.189:0/2556801571 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f389413a8e0 con 0x7f38940acd30 2020-11-20T12:21:58.017 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:58.013+0000 7f388bfff700 1 -- 172.21.15.189:0/2556801571 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+1785 (secure 0 0 0) 0x7f388c022070 con 0x7f38940acd30 2020-11-20T12:21:58.017 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:21:58.017 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:21:50.854972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5297":{"gid":5297,"name":"b","rank":0,"incarnation":67,"state":"up:active","state_seq":14,"addr":"172.21.15.47:6833/3025803398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3025803398},{"type":"v1","addr":"172.21.15.47:6833","nonce":3025803398}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:21:58.018 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:58.013+0000 7f389938e700 1 -- 172.21.15.189:0/2556801571 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f38700557c0 msgr2=0x7f3870057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:58.019 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:58.013+0000 7f389938e700 1 --2- 172.21.15.189:0/2556801571 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f38700557c0 0x7f3870057c80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f38840055a0 tx=0x7f3884009040).stop 2020-11-20T12:21:58.019 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:58.013+0000 7f389938e700 1 -- 172.21.15.189:0/2556801571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38940acd30 msgr2=0x7f3894149930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:21:58.019 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:58.013+0000 7f389938e700 1 --2- 172.21.15.189:0/2556801571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38940acd30 0x7f3894149930 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7f388c003340 tx=0x7f388c016000).stop 2020-11-20T12:21:58.019 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:58.013+0000 7f389938e700 1 -- 172.21.15.189:0/2556801571 shutdown_connections 2020-11-20T12:21:58.020 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:58.013+0000 7f389938e700 1 --2- 172.21.15.189:0/2556801571 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f38700557c0 0x7f3870057c80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:58.020 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:58.013+0000 7f389938e700 1 --2- 172.21.15.189:0/2556801571 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f38940672f0 0x7f389413ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:58.020 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:58.013+0000 7f389938e700 1 --2- 172.21.15.189:0/2556801571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38940acd30 0x7f3894149930 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:58.020 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:58.013+0000 7f389938e700 1 --2- 172.21.15.189:0/2556801571 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f38940ac950 0x7f38940a6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:21:58.020 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:58.013+0000 7f389938e700 1 -- 172.21.15.189:0/2556801571 >> 172.21.15.189:0/2556801571 conn(0x7f3894062080 msgr2=0x7f3894062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:21:58.021 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:58.013+0000 7f389938e700 1 -- 172.21.15.189:0/2556801571 shutdown_connections 2020-11-20T12:21:58.021 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:21:58.013+0000 7f389938e700 1 -- 172.21.15.189:0/2556801571 wait complete. 2020-11-20T12:21:58.021 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 70 2020-11-20T12:21:58.035 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2020-11-20T12:22:00.036 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:22:00.171 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.165+0000 7f0663fff700 1 Processor -- start 2020-11-20T12:22:00.171 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.165+0000 7f0663fff700 1 -- start start 2020-11-20T12:22:00.176 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f0663fff700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f06640ad000 0x7f066413e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:00.176 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f0663fff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f066413a6e0 0x7f066413ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:00.177 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f0663fff700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0664067730 con 0x7f06640ad3e0 2020-11-20T12:22:00.177 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f0663fff700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0664066050 con 0x7f06640ad000 2020-11-20T12:22:00.177 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f0663fff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0664138190 con 0x7f066413a6e0 2020-11-20T12:22:00.178 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f06627fc700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f06640ad3e0 0x7f0664066d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53816/0 (socket says 172.21.15.189:53816) 2020-11-20T12:22:00.178 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f06627fc700 1 -- 172.21.15.189:0/3756167657 learned_addr learned my addr 172.21.15.189:0/3756167657 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:22:00.178 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f0662ffd700 1 --2- 172.21.15.189:0/3756167657 >> v2:172.21.15.47:3300/0 conn(0x7f06640ad000 0x7f066413e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:00.179 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f06637fe700 1 --2- 172.21.15.189:0/3756167657 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f066413a6e0 0x7f066413ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:00.179 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f0661ffb700 1 -- 172.21.15.189:0/3756167657 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462022649 0 0) 0x7f0664067730 con 0x7f06640ad3e0 2020-11-20T12:22:00.179 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f0661ffb700 1 -- 172.21.15.189:0/3756167657 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0650003240 con 0x7f06640ad3e0 2020-11-20T12:22:00.183 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f06637fe700 1 -- 172.21.15.189:0/3756167657 >> v2:172.21.15.47:3300/0 conn(0x7f06640ad000 msgr2=0x7f066413e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:00.183 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f06637fe700 1 --2- 172.21.15.189:0/3756167657 >> v2:172.21.15.47:3300/0 conn(0x7f06640ad000 0x7f066413e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:00.183 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f06637fe700 1 -- 172.21.15.189:0/3756167657 >> v1:172.21.15.189:6789/0 conn(0x7f06640ad3e0 legacy=0x7f0664066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:00.184 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f0661ffb700 1 -- 172.21.15.189:0/3756167657 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4122821422 0 0) 0x7f0650003240 con 0x7f06640ad3e0 2020-11-20T12:22:00.184 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f06637fe700 1 -- 172.21.15.189:0/3756167657 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066413dd40 con 0x7f066413a6e0 2020-11-20T12:22:00.184 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f0662ffd700 1 --2- 172.21.15.189:0/3756167657 >> v2:172.21.15.47:3300/0 conn(0x7f06640ad000 0x7f066413e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:22:00.185 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f06637fe700 1 --2- 172.21.15.189:0/3756167657 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f066413a6e0 0x7f066413ad00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f065c009c90 tx=0x7f065c0091d0).ready entity=mon.2 client_cookie=59ccb920b7eac940 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:00.185 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f0661ffb700 1 -- 172.21.15.189:0/3756167657 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06580014e0 con 0x7f066413a6e0 2020-11-20T12:22:00.185 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f0661ffb700 1 -- 172.21.15.189:0/3756167657 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f065c00b740 con 0x7f066413a6e0 2020-11-20T12:22:00.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f0661ffb700 1 -- 172.21.15.189:0/3756167657 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065c00bac0 con 0x7f066413a6e0 2020-11-20T12:22:00.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f0663fff700 1 -- 172.21.15.189:0/3756167657 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f066413a6e0 msgr2=0x7f066413ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:00.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f0663fff700 1 --2- 172.21.15.189:0/3756167657 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f066413a6e0 0x7f066413ad00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f065c009c90 tx=0x7f065c0091d0).stop 2020-11-20T12:22:00.186 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f0663fff700 1 -- 172.21.15.189:0/3756167657 shutdown_connections 2020-11-20T12:22:00.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f0663fff700 1 --2- 172.21.15.189:0/3756167657 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f066413a6e0 0x7f066413ad00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:00.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f0663fff700 1 --2- 172.21.15.189:0/3756167657 >> v2:172.21.15.47:3300/0 conn(0x7f06640ad000 0x7f066413e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:00.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f0663fff700 1 -- 172.21.15.189:0/3756167657 >> 172.21.15.189:0/3756167657 conn(0x7f0664062080 msgr2=0x7f0664062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:00.187 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f0663fff700 1 -- 172.21.15.189:0/3756167657 shutdown_connections 2020-11-20T12:22:00.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f0663fff700 1 -- 172.21.15.189:0/3756167657 wait complete. 2020-11-20T12:22:00.188 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f0663fff700 1 Processor -- start 2020-11-20T12:22:00.189 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.169+0000 7f0663fff700 1 -- start start 2020-11-20T12:22:00.192 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.173+0000 7f0663fff700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06640ad000 0x7f0664140790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:00.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.173+0000 7f0663fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06640ad3e0 0x7f06640aa530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:00.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.173+0000 7f0663fff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f06640a6290 0x7f06640a98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:00.193 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.173+0000 7f0663fff700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06640aaa20 con 0x7f06640ad000 2020-11-20T12:22:00.194 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.173+0000 7f0663fff700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06640aacb0 con 0x7f06640ad3e0 2020-11-20T12:22:00.194 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.173+0000 7f0663fff700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06640aaf50 con 0x7f06640a6290 2020-11-20T12:22:00.194 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.173+0000 7f06627fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06640ad3e0 0x7f06640aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:00.195 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.173+0000 7f06627fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06640ad3e0 0x7f06640aa530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52200/0 (socket says 172.21.15.189:52200) 2020-11-20T12:22:00.195 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.173+0000 7f06627fc700 1 -- 172.21.15.189:0/852144706 learned_addr learned my addr 172.21.15.189:0/852144706 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:22:00.195 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.173+0000 7f0662ffd700 1 --2- 172.21.15.189:0/852144706 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06640ad000 0x7f0664140790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:00.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.173+0000 7f06637fe700 1 --2- 172.21.15.189:0/852144706 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f06640a6290 0x7f06640a98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:00.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.173+0000 7f06627fc700 1 -- 172.21.15.189:0/852144706 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f06640a6290 msgr2=0x7f06640a98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:00.196 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.173+0000 7f06627fc700 1 --2- 172.21.15.189:0/852144706 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f06640a6290 0x7f06640a98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:00.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.173+0000 7f06627fc700 1 -- 172.21.15.189:0/852144706 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06640ad000 msgr2=0x7f0664140790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:00.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.173+0000 7f06627fc700 1 --2- 172.21.15.189:0/852144706 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06640ad000 0x7f0664140790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:00.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.173+0000 7f06627fc700 1 -- 172.21.15.189:0/852144706 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06640a9410 con 0x7f06640ad3e0 2020-11-20T12:22:00.197 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.173+0000 7f06637fe700 1 --2- 172.21.15.189:0/852144706 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f06640a6290 0x7f06640a98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:22:00.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.173+0000 7f06627fc700 1 --2- 172.21.15.189:0/852144706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06640ad3e0 0x7f06640aa530 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7f065800cab0 tx=0x7f0658001330).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:00.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.173+0000 7f064affd700 1 -- 172.21.15.189:0/852144706 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0658007170 con 0x7f06640ad3e0 2020-11-20T12:22:00.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.173+0000 7f064affd700 1 -- 172.21.15.189:0/852144706 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06580072d0 con 0x7f06640ad3e0 2020-11-20T12:22:00.198 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.173+0000 7f0663fff700 1 -- 172.21.15.189:0/852144706 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06640a9da0 con 0x7f06640ad3e0 2020-11-20T12:22:00.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.173+0000 7f064affd700 1 -- 172.21.15.189:0/852144706 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065800aa10 con 0x7f06640ad3e0 2020-11-20T12:22:00.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.173+0000 7f0663fff700 1 -- 172.21.15.189:0/852144706 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f066414d1b0 con 0x7f06640ad3e0 2020-11-20T12:22:00.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.173+0000 7f0663fff700 1 -- 172.21.15.189:0/852144706 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0664138630 con 0x7f06640ad3e0 2020-11-20T12:22:00.199 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.173+0000 7f064affd700 1 -- 172.21.15.189:0/852144706 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f0658010070 con 0x7f06640ad3e0 2020-11-20T12:22:00.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.177+0000 7f064affd700 1 --2- 172.21.15.189:0/852144706 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f06400557c0 0x7f0640057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:00.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.177+0000 7f064affd700 1 -- 172.21.15.189:0/852144706 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7191+0+0 (secure 0 0 0) 0x7f065806e7c0 con 0x7f06640ad3e0 2020-11-20T12:22:00.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.177+0000 7f0662ffd700 1 --2- 172.21.15.189:0/852144706 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f06400557c0 0x7f0640057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:00.200 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.177+0000 7f0662ffd700 1 --2- 172.21.15.189:0/852144706 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f06400557c0 0x7f0640057c80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f0654006fd0 tx=0x7f0654006df0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:00.201 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.181+0000 7f064affd700 1 -- 172.21.15.189:0/852144706 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0658044280 con 0x7f06640ad3e0 2020-11-20T12:22:00.420 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.417+0000 7f0663fff700 1 -- 172.21.15.189:0/852144706 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06640a95a0 con 0x7f06640ad3e0 2020-11-20T12:22:00.422 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.417+0000 7f064affd700 1 -- 172.21.15.189:0/852144706 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f06580478a0 con 0x7f06640ad3e0 2020-11-20T12:22:00.422 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:22:00.422 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5938,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.189:6835/3162779725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3162779725},{"type":"v1","addr":"172.21.15.189:6835","nonce":3162779725}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:21:50.854972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5297":{"gid":5297,"name":"b","rank":0,"incarnation":67,"state":"up:active","state_seq":14,"addr":"172.21.15.47:6833/3025803398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3025803398},{"type":"v1","addr":"172.21.15.47:6833","nonce":3025803398}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:22:00.424 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.417+0000 7f0663fff700 1 -- 172.21.15.189:0/852144706 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f06400557c0 msgr2=0x7f0640057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:00.424 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.417+0000 7f0663fff700 1 --2- 172.21.15.189:0/852144706 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f06400557c0 0x7f0640057c80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f0654006fd0 tx=0x7f0654006df0).stop 2020-11-20T12:22:00.424 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.417+0000 7f0663fff700 1 -- 172.21.15.189:0/852144706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06640ad3e0 msgr2=0x7f06640aa530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:00.425 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.417+0000 7f0663fff700 1 --2- 172.21.15.189:0/852144706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06640ad3e0 0x7f06640aa530 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7f065800cab0 tx=0x7f0658001330).stop 2020-11-20T12:22:00.425 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.417+0000 7f0663fff700 1 -- 172.21.15.189:0/852144706 shutdown_connections 2020-11-20T12:22:00.425 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.417+0000 7f0663fff700 1 --2- 172.21.15.189:0/852144706 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f06400557c0 0x7f0640057c80 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:00.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.417+0000 7f0663fff700 1 --2- 172.21.15.189:0/852144706 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f06640ad000 0x7f0664140790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:00.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.417+0000 7f0663fff700 1 --2- 172.21.15.189:0/852144706 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f06640a6290 0x7f06640a98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:00.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.417+0000 7f0663fff700 1 --2- 172.21.15.189:0/852144706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06640ad3e0 0x7f06640aa530 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:00.426 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.417+0000 7f0663fff700 1 -- 172.21.15.189:0/852144706 >> 172.21.15.189:0/852144706 conn(0x7f0664062080 msgr2=0x7f06640ac3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:00.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.417+0000 7f0663fff700 1 -- 172.21.15.189:0/852144706 shutdown_connections 2020-11-20T12:22:00.427 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:00.417+0000 7f0663fff700 1 -- 172.21.15.189:0/852144706 wait complete. 2020-11-20T12:22:00.427 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 71 2020-11-20T12:22:00.437 INFO:tasks.mds_thrash.fs.[None]:mds.a reported in up:standby state 2020-11-20T12:22:00.438 INFO:tasks.mds_thrash.fs.[None]:waiting for 55 secs before thrashing 2020-11-20T12:22:26.369 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:22:26.372 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:22:55.440 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:22:55.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.568+0000 7fd57b9d1700 1 Processor -- start 2020-11-20T12:22:55.574 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.568+0000 7fd57b9d1700 1 -- start start 2020-11-20T12:22:55.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.568+0000 7fd57b9d1700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd574060310 0x7fd5740667d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:55.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.568+0000 7fd57b9d1700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd57413b4d0 0x7fd574141ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:55.576 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.568+0000 7fd57b9d1700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd574066370 con 0x7fd57413a960 2020-11-20T12:22:55.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.568+0000 7fd57b9d1700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5740a9870 con 0x7fd57413b4d0 2020-11-20T12:22:55.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.568+0000 7fd57b9d1700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5740a95d0 con 0x7fd574060310 2020-11-20T12:22:55.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.568+0000 7fd578f6c700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd57413a960 0x7fd57413adc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53830/0 (socket says 172.21.15.189:53830) 2020-11-20T12:22:55.577 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.572+0000 7fd578f6c700 1 -- 172.21.15.189:0/1287465933 learned_addr learned my addr 172.21.15.189:0/1287465933 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:22:55.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.568+0000 7fd57976d700 1 --2- 172.21.15.189:0/1287465933 >> v2:172.21.15.189:3300/0 conn(0x7fd574060310 0x7fd5740667d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:55.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.572+0000 7fd579f6e700 1 --2- 172.21.15.189:0/1287465933 >> v2:172.21.15.47:3300/0 conn(0x7fd57413b4d0 0x7fd574141ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:55.578 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.572+0000 7fd563fff700 1 -- 172.21.15.189:0/1287465933 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1475824038 0 0) 0x7fd574066370 con 0x7fd57413a960 2020-11-20T12:22:55.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.572+0000 7fd563fff700 1 -- 172.21.15.189:0/1287465933 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd55c003240 con 0x7fd57413a960 2020-11-20T12:22:55.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.572+0000 7fd57976d700 1 -- 172.21.15.189:0/1287465933 >> v2:172.21.15.47:3300/0 conn(0x7fd57413b4d0 msgr2=0x7fd574141ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:55.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.572+0000 7fd57976d700 1 --2- 172.21.15.189:0/1287465933 >> v2:172.21.15.47:3300/0 conn(0x7fd57413b4d0 0x7fd574141ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:55.582 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.572+0000 7fd57976d700 1 -- 172.21.15.189:0/1287465933 >> v1:172.21.15.189:6789/0 conn(0x7fd57413a960 legacy=0x7fd57413adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:55.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.572+0000 7fd57976d700 1 -- 172.21.15.189:0/1287465933 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5740606f0 con 0x7fd574060310 2020-11-20T12:22:55.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.572+0000 7fd57976d700 1 --2- 172.21.15.189:0/1287465933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd574060310 0x7fd5740667d0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7fd5740ad1b0 tx=0x7fd570006df0).ready entity=mon.1 client_cookie=657132493f759466 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:55.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.572+0000 7fd563fff700 1 -- 172.21.15.189:0/1287465933 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5700096d0 con 0x7fd574060310 2020-11-20T12:22:55.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.572+0000 7fd563fff700 1 -- 172.21.15.189:0/1287465933 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd570009830 con 0x7fd574060310 2020-11-20T12:22:55.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.572+0000 7fd563fff700 1 -- 172.21.15.189:0/1287465933 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd570009bb0 con 0x7fd574060310 2020-11-20T12:22:55.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.572+0000 7fd57b9d1700 1 -- 172.21.15.189:0/1287465933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd574060310 msgr2=0x7fd5740667d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:55.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.572+0000 7fd57b9d1700 1 --2- 172.21.15.189:0/1287465933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd574060310 0x7fd5740667d0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7fd5740ad1b0 tx=0x7fd570006df0).stop 2020-11-20T12:22:55.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.572+0000 7fd57b9d1700 1 -- 172.21.15.189:0/1287465933 shutdown_connections 2020-11-20T12:22:55.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.572+0000 7fd57b9d1700 1 --2- 172.21.15.189:0/1287465933 >> v2:172.21.15.47:3300/0 conn(0x7fd57413b4d0 0x7fd574141ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:55.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.572+0000 7fd57b9d1700 1 --2- 172.21.15.189:0/1287465933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd574060310 0x7fd5740667d0 secure :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 rx=0x7fd5740ad1b0 tx=0x7fd570006df0).stop 2020-11-20T12:22:55.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.572+0000 7fd57b9d1700 1 -- 172.21.15.189:0/1287465933 >> 172.21.15.189:0/1287465933 conn(0x7fd5740627f0 msgr2=0x7fd574065870 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:55.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.572+0000 7fd57b9d1700 1 -- 172.21.15.189:0/1287465933 shutdown_connections 2020-11-20T12:22:55.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.572+0000 7fd57b9d1700 1 -- 172.21.15.189:0/1287465933 wait complete. 2020-11-20T12:22:55.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.572+0000 7fd57b9d1700 1 Processor -- start 2020-11-20T12:22:55.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.572+0000 7fd57b9d1700 1 -- start start 2020-11-20T12:22:55.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.576+0000 7fd57b9d1700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd57413a960 0x7fd5740ac3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:55.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.576+0000 7fd57b9d1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd57413b4d0 0x7fd5740a5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:55.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.576+0000 7fd57b9d1700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd57413d110 0x7fd574131ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:55.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.576+0000 7fd57b9d1700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5740a5510 con 0x7fd57413a960 2020-11-20T12:22:55.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.576+0000 7fd57b9d1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57413de10 con 0x7fd57413b4d0 2020-11-20T12:22:55.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.576+0000 7fd57b9d1700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd57413e480 con 0x7fd57413d110 2020-11-20T12:22:55.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.576+0000 7fd578f6c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd57413b4d0 0x7fd5740a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:55.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.576+0000 7fd578f6c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd57413b4d0 0x7fd5740a5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52214/0 (socket says 172.21.15.189:52214) 2020-11-20T12:22:55.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.576+0000 7fd578f6c700 1 -- 172.21.15.189:0/1735028315 learned_addr learned my addr 172.21.15.189:0/1735028315 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:22:55.588 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.576+0000 7fd57976d700 1 --2- 172.21.15.189:0/1735028315 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd57413a960 0x7fd5740ac3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:55.588 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.576+0000 7fd579f6e700 1 --2- 172.21.15.189:0/1735028315 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd57413d110 0x7fd574131ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:55.588 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.576+0000 7fd578f6c700 1 -- 172.21.15.189:0/1735028315 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd57413d110 msgr2=0x7fd574131ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:55.588 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.576+0000 7fd578f6c700 1 --2- 172.21.15.189:0/1735028315 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd57413d110 0x7fd574131ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:55.588 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.576+0000 7fd578f6c700 1 -- 172.21.15.189:0/1735028315 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd57413a960 msgr2=0x7fd5740ac3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:55.589 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.576+0000 7fd578f6c700 1 --2- 172.21.15.189:0/1735028315 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd57413a960 0x7fd5740ac3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:55.589 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.576+0000 7fd578f6c700 1 -- 172.21.15.189:0/1735028315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57413db10 con 0x7fd57413b4d0 2020-11-20T12:22:55.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.576+0000 7fd579f6e700 1 --2- 172.21.15.189:0/1735028315 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd57413d110 0x7fd574131ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:22:55.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.576+0000 7fd578f6c700 1 --2- 172.21.15.189:0/1735028315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd57413b4d0 0x7fd5740a5020 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7fd56400c6e0 tx=0x7fd564002870).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:55.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.576+0000 7fd5617fa700 1 -- 172.21.15.189:0/1735028315 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5640072e0 con 0x7fd57413b4d0 2020-11-20T12:22:55.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.576+0000 7fd5617fa700 1 -- 172.21.15.189:0/1735028315 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5640054c0 con 0x7fd57413b4d0 2020-11-20T12:22:55.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.576+0000 7fd5617fa700 1 -- 172.21.15.189:0/1735028315 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd564005840 con 0x7fd57413b4d0 2020-11-20T12:22:55.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.576+0000 7fd57b9d1700 1 -- 172.21.15.189:0/1735028315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5740acf00 con 0x7fd57413b4d0 2020-11-20T12:22:55.592 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.576+0000 7fd57b9d1700 1 -- 172.21.15.189:0/1735028315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5740a6140 con 0x7fd57413b4d0 2020-11-20T12:22:55.593 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.576+0000 7fd57b9d1700 1 -- 172.21.15.189:0/1735028315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd57413e2e0 con 0x7fd57413b4d0 2020-11-20T12:22:55.593 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.576+0000 7fd5617fa700 1 -- 172.21.15.189:0/1735028315 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fd564010070 con 0x7fd57413b4d0 2020-11-20T12:22:55.593 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.580+0000 7fd5617fa700 1 --2- 172.21.15.189:0/1735028315 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd54c0557c0 0x7fd54c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:55.594 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.580+0000 7fd5617fa700 1 -- 172.21.15.189:0/1735028315 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7191+0+0 (secure 0 0 0) 0x7fd56406e270 con 0x7fd57413b4d0 2020-11-20T12:22:55.594 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.580+0000 7fd57976d700 1 --2- 172.21.15.189:0/1735028315 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd54c0557c0 0x7fd54c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:55.594 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.580+0000 7fd57976d700 1 --2- 172.21.15.189:0/1735028315 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd54c0557c0 0x7fd54c057c80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fd5740a7da0 tx=0x7fd570015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:55.594 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.584+0000 7fd5617fa700 1 -- 172.21.15.189:0/1735028315 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd57413e2e0 con 0x7fd57413b4d0 2020-11-20T12:22:55.813 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.808+0000 7fd57b9d1700 1 -- 172.21.15.189:0/1735028315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd57413e2e0 con 0x7fd57413b4d0 2020-11-20T12:22:55.814 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.808+0000 7fd5617fa700 1 -- 172.21.15.189:0/1735028315 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fd57413e2e0 con 0x7fd57413b4d0 2020-11-20T12:22:55.814 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:22:55.814 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5938,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.189:6835/3162779725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3162779725},{"type":"v1","addr":"172.21.15.189:6835","nonce":3162779725}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:21:50.854972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5297":{"gid":5297,"name":"b","rank":0,"incarnation":67,"state":"up:active","state_seq":14,"addr":"172.21.15.47:6833/3025803398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3025803398},{"type":"v1","addr":"172.21.15.47:6833","nonce":3025803398}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:22:55.815 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.808+0000 7fd57b9d1700 1 -- 172.21.15.189:0/1735028315 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd54c0557c0 msgr2=0x7fd54c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:55.816 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.812+0000 7fd57b9d1700 1 --2- 172.21.15.189:0/1735028315 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd54c0557c0 0x7fd54c057c80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fd5740a7da0 tx=0x7fd570015000).stop 2020-11-20T12:22:55.816 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.812+0000 7fd57b9d1700 1 -- 172.21.15.189:0/1735028315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd57413b4d0 msgr2=0x7fd5740a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:55.816 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.812+0000 7fd57b9d1700 1 --2- 172.21.15.189:0/1735028315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd57413b4d0 0x7fd5740a5020 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7fd56400c6e0 tx=0x7fd564002870).stop 2020-11-20T12:22:55.816 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.812+0000 7fd57b9d1700 1 -- 172.21.15.189:0/1735028315 shutdown_connections 2020-11-20T12:22:55.817 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.812+0000 7fd57b9d1700 1 --2- 172.21.15.189:0/1735028315 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd54c0557c0 0x7fd54c057c80 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:55.817 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.812+0000 7fd57b9d1700 1 --2- 172.21.15.189:0/1735028315 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd57413a960 0x7fd5740ac3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:55.817 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.812+0000 7fd57b9d1700 1 --2- 172.21.15.189:0/1735028315 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd57413d110 0x7fd574131ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:55.817 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.812+0000 7fd57b9d1700 1 --2- 172.21.15.189:0/1735028315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd57413b4d0 0x7fd5740a5020 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:55.818 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.812+0000 7fd57b9d1700 1 -- 172.21.15.189:0/1735028315 >> 172.21.15.189:0/1735028315 conn(0x7fd5740627f0 msgr2=0x7fd5740a8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:55.818 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.812+0000 7fd57b9d1700 1 -- 172.21.15.189:0/1735028315 shutdown_connections 2020-11-20T12:22:55.818 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.812+0000 7fd57b9d1700 1 -- 172.21.15.189:0/1735028315 wait complete. 2020-11-20T12:22:55.819 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 71 2020-11-20T12:22:55.831 INFO:tasks.mds_thrash.fs.[None]:kill mds.b (rank=0) 2020-11-20T12:22:55.832 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T12:22:55.832 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T12:22:55.847 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T12:22:55.847 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:22:55.987 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.980+0000 7f831dc00700 1 Processor -- start 2020-11-20T12:22:55.987 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.980+0000 7f831dc00700 1 -- start start 2020-11-20T12:22:55.989 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f831dc00700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f8318060310 0x7f83180a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:55.990 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f831dc00700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f831813fc70 0x7f8318142060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:55.990 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f831dc00700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8318063b50 con 0x7f8318134460 2020-11-20T12:22:55.991 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f831dc00700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8318067a00 con 0x7f8318060310 2020-11-20T12:22:55.991 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f831dc00700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8318066500 con 0x7f831813fc70 2020-11-20T12:22:55.991 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f8317fff700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8318134460 0x7f83181348e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53846/0 (socket says 172.21.15.189:53846) 2020-11-20T12:22:55.992 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f8317fff700 1 -- 172.21.15.189:0/970988219 learned_addr learned my addr 172.21.15.189:0/970988219 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:22:55.992 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f831d3ff700 1 --2- 172.21.15.189:0/970988219 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f831813fc70 0x7f8318142060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:55.992 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f831cbfe700 1 --2- 172.21.15.189:0/970988219 >> v2:172.21.15.47:3300/0 conn(0x7f8318060310 0x7f83180a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:55.992 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f83177fe700 1 -- 172.21.15.189:0/970988219 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620514843 0 0) 0x7f8318063b50 con 0x7f8318134460 2020-11-20T12:22:55.992 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f83177fe700 1 -- 172.21.15.189:0/970988219 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82fc003240 con 0x7f8318134460 2020-11-20T12:22:55.993 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f83177fe700 1 -- 172.21.15.189:0/970988219 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2323966081 0 0) 0x7f82fc003240 con 0x7f8318134460 2020-11-20T12:22:55.996 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f831d3ff700 1 -- 172.21.15.189:0/970988219 >> v2:172.21.15.47:3300/0 conn(0x7f8318060310 msgr2=0x7f83180a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:55.996 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f831d3ff700 1 --2- 172.21.15.189:0/970988219 >> v2:172.21.15.47:3300/0 conn(0x7f8318060310 0x7f83180a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:55.996 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f831d3ff700 1 -- 172.21.15.189:0/970988219 >> v1:172.21.15.189:6789/0 conn(0x7f8318134460 legacy=0x7f83181348e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:55.997 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f831d3ff700 1 -- 172.21.15.189:0/970988219 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83180a09a0 con 0x7f831813fc70 2020-11-20T12:22:55.997 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f831d3ff700 1 --2- 172.21.15.189:0/970988219 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f831813fc70 0x7f8318142060 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f8310009c90 tx=0x7f83100091d0).ready entity=mon.2 client_cookie=746a9b5bb452bc41 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:55.997 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f83177fe700 1 -- 172.21.15.189:0/970988219 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83040014e0 con 0x7f831813fc70 2020-11-20T12:22:55.997 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f83177fe700 1 -- 172.21.15.189:0/970988219 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f831000b6b0 con 0x7f831813fc70 2020-11-20T12:22:55.997 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f83177fe700 1 -- 172.21.15.189:0/970988219 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831000ba30 con 0x7f831813fc70 2020-11-20T12:22:55.998 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f831dc00700 1 -- 172.21.15.189:0/970988219 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f831813fc70 msgr2=0x7f8318142060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:55.998 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f831dc00700 1 --2- 172.21.15.189:0/970988219 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f831813fc70 0x7f8318142060 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f8310009c90 tx=0x7f83100091d0).stop 2020-11-20T12:22:55.998 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f831dc00700 1 -- 172.21.15.189:0/970988219 shutdown_connections 2020-11-20T12:22:55.998 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f831dc00700 1 --2- 172.21.15.189:0/970988219 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f831813fc70 0x7f8318142060 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:55.999 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f831dc00700 1 --2- 172.21.15.189:0/970988219 >> v2:172.21.15.47:3300/0 conn(0x7f8318060310 0x7f83180a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:55.999 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f831dc00700 1 -- 172.21.15.189:0/970988219 >> 172.21.15.189:0/970988219 conn(0x7f8318063210 msgr2=0x7f8318060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:55.999 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f831dc00700 1 -- 172.21.15.189:0/970988219 shutdown_connections 2020-11-20T12:22:55.999 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.984+0000 7f831dc00700 1 -- 172.21.15.189:0/970988219 wait complete. 2020-11-20T12:22:55.999 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.988+0000 7f831dc00700 1 Processor -- start 2020-11-20T12:22:56.000 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.988+0000 7f831dc00700 1 -- start start 2020-11-20T12:22:56.000 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.988+0000 7f831dc00700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8318060310 0x7f831813b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:56.000 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.988+0000 7f831dc00700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8318134460 0x7f8318135ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:56.000 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.988+0000 7f831dc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f831813fc70 0x7f8318145b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:56.001 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.988+0000 7f831dc00700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83180acad0 con 0x7f8318060310 2020-11-20T12:22:56.001 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.988+0000 7f831dc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8318136430 con 0x7f831813fc70 2020-11-20T12:22:56.001 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.988+0000 7f831dc00700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8318136660 con 0x7f8318134460 2020-11-20T12:22:56.001 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.988+0000 7f831d3ff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f831813fc70 0x7f8318145b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:56.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.988+0000 7f8317fff700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8318134460 0x7f8318135ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:56.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.988+0000 7f831d3ff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f831813fc70 0x7f8318145b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52232/0 (socket says 172.21.15.189:52232) 2020-11-20T12:22:56.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.988+0000 7f831d3ff700 1 -- 172.21.15.189:0/3240602786 learned_addr learned my addr 172.21.15.189:0/3240602786 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:22:56.002 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.988+0000 7f831cbfe700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8318060310 0x7f831813b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:56.003 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.988+0000 7f831d3ff700 1 -- 172.21.15.189:0/3240602786 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8318134460 msgr2=0x7f8318135ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:56.003 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.988+0000 7f831d3ff700 1 --2- 172.21.15.189:0/3240602786 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8318134460 0x7f8318135ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:56.003 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.988+0000 7f831d3ff700 1 -- 172.21.15.189:0/3240602786 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8318060310 msgr2=0x7f831813b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:56.003 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.988+0000 7f831d3ff700 1 --2- 172.21.15.189:0/3240602786 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8318060310 0x7f831813b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:56.003 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.988+0000 7f831d3ff700 1 -- 172.21.15.189:0/3240602786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8318135630 con 0x7f831813fc70 2020-11-20T12:22:56.004 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.988+0000 7f831d3ff700 1 --2- 172.21.15.189:0/3240602786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f831813fc70 0x7f8318145b60 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7f8310011ba0 tx=0x7f8310016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:56.004 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.988+0000 7f8314ff9700 1 -- 172.21.15.189:0/3240602786 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8310011e20 con 0x7f831813fc70 2020-11-20T12:22:56.004 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.988+0000 7f8314ff9700 1 -- 172.21.15.189:0/3240602786 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f831001d660 con 0x7f831813fc70 2020-11-20T12:22:56.004 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.988+0000 7f8314ff9700 1 -- 172.21.15.189:0/3240602786 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831001d9e0 con 0x7f831813fc70 2020-11-20T12:22:56.004 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.988+0000 7f831dc00700 1 -- 172.21.15.189:0/3240602786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83180a0730 con 0x7f831813fc70 2020-11-20T12:22:56.005 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.988+0000 7f831dc00700 1 -- 172.21.15.189:0/3240602786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8318136870 con 0x7f831813fc70 2020-11-20T12:22:56.005 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.992+0000 7f831dc00700 1 -- 172.21.15.189:0/3240602786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83181357c0 con 0x7f831813fc70 2020-11-20T12:22:56.006 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.992+0000 7f8314ff9700 1 -- 172.21.15.189:0/3240602786 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f831002d070 con 0x7f831813fc70 2020-11-20T12:22:56.006 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.992+0000 7f8314ff9700 1 --2- 172.21.15.189:0/3240602786 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f82f00557c0 0x7f82f0057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:56.006 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.992+0000 7f8314ff9700 1 -- 172.21.15.189:0/3240602786 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7191+0+0 (secure 0 0 0) 0x7f8310071270 con 0x7f831813fc70 2020-11-20T12:22:56.007 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.992+0000 7f831cbfe700 1 --2- 172.21.15.189:0/3240602786 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f82f00557c0 0x7f82f0057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:56.007 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.992+0000 7f831cbfe700 1 --2- 172.21.15.189:0/3240602786 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f82f00557c0 0x7f82f0057c80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f830c00ed90 tx=0x7f830c006e80).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:56.007 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:55.996+0000 7f8314ff9700 1 -- 172.21.15.189:0/3240602786 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8310046cb0 con 0x7f831813fc70 2020-11-20T12:22:56.235 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:56.232+0000 7f831dc00700 1 -- 172.21.15.189:0/3240602786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83180a10f0 con 0x7f831813fc70 2020-11-20T12:22:56.236 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:56.232+0000 7f8314ff9700 1 -- 172.21.15.189:0/3240602786 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f8310015090 con 0x7f831813fc70 2020-11-20T12:22:56.237 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:22:56.237 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5938,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.189:6835/3162779725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3162779725},{"type":"v1","addr":"172.21.15.189:6835","nonce":3162779725}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:21:50.854972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5297":{"gid":5297,"name":"b","rank":0,"incarnation":67,"state":"up:active","state_seq":14,"addr":"172.21.15.47:6833/3025803398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3025803398},{"type":"v1","addr":"172.21.15.47:6833","nonce":3025803398}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:22:56.238 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:56.232+0000 7f831dc00700 1 -- 172.21.15.189:0/3240602786 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f82f00557c0 msgr2=0x7f82f0057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:56.239 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:56.232+0000 7f831dc00700 1 --2- 172.21.15.189:0/3240602786 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f82f00557c0 0x7f82f0057c80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f830c00ed90 tx=0x7f830c006e80).stop 2020-11-20T12:22:56.239 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:56.232+0000 7f831dc00700 1 -- 172.21.15.189:0/3240602786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f831813fc70 msgr2=0x7f8318145b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:56.239 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:56.232+0000 7f831dc00700 1 --2- 172.21.15.189:0/3240602786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f831813fc70 0x7f8318145b60 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7f8310011ba0 tx=0x7f8310016000).stop 2020-11-20T12:22:56.239 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:56.232+0000 7f831dc00700 1 -- 172.21.15.189:0/3240602786 shutdown_connections 2020-11-20T12:22:56.239 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:56.232+0000 7f831dc00700 1 --2- 172.21.15.189:0/3240602786 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f82f00557c0 0x7f82f0057c80 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:56.240 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:56.232+0000 7f831dc00700 1 --2- 172.21.15.189:0/3240602786 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f8318060310 0x7f831813b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:56.240 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:56.232+0000 7f831dc00700 1 --2- 172.21.15.189:0/3240602786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f831813fc70 0x7f8318145b60 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:56.240 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:56.232+0000 7f831dc00700 1 --2- 172.21.15.189:0/3240602786 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f8318134460 0x7f8318135ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:56.240 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:56.232+0000 7f831dc00700 1 -- 172.21.15.189:0/3240602786 >> 172.21.15.189:0/3240602786 conn(0x7f8318063210 msgr2=0x7f83180a8330 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:56.241 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:56.232+0000 7f831dc00700 1 -- 172.21.15.189:0/3240602786 shutdown_connections 2020-11-20T12:22:56.241 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:56.232+0000 7f831dc00700 1 -- 172.21.15.189:0/3240602786 wait complete. 2020-11-20T12:22:56.241 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 71 2020-11-20T12:22:56.254 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:22:56.453 INFO:teuthology.orchestra.run.smithi047:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:22:56.458 INFO:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T12:22:58.254 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:22:58.395 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.384+0000 7f0c3ac43700 1 Processor -- start 2020-11-20T12:22:58.396 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.384+0000 7f0c3ac43700 1 -- start start 2020-11-20T12:22:58.396 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.384+0000 7f0c3ac43700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7f0c34137440 0x7f0c341423f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:58.396 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.384+0000 7f0c3ac43700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c34067c30 con 0x7f0c340a8fe0 2020-11-20T12:22:58.405 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.384+0000 7f0c3ac43700 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c34062b50 con 0x7f0c340a9f10 2020-11-20T12:22:58.405 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.384+0000 7f0c3ac43700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c34065c60 con 0x7f0c34137440 2020-11-20T12:22:58.406 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.384+0000 7f0c39c41700 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0c340a9f10 0x7f0c340a6300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53864/0 (socket says 172.21.15.189:53864) 2020-11-20T12:22:58.407 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.384+0000 7f0c39c41700 1 -- 172.21.15.189:0/643910623 learned_addr learned my addr 172.21.15.189:0/643910623 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:22:58.408 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.384+0000 7f0c3a442700 1 --2- 172.21.15.189:0/643910623 >> v2:172.21.15.47:3300/0 conn(0x7f0c34137440 0x7f0c341423f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:58.409 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.384+0000 7f0c38c3f700 1 -- 172.21.15.189:0/643910623 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2148821802 0 0) 0x7f0c34062b50 con 0x7f0c340a9f10 2020-11-20T12:22:58.410 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.384+0000 7f0c38c3f700 1 -- 172.21.15.189:0/643910623 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c1c003240 con 0x7f0c340a9f10 2020-11-20T12:22:58.451 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c38c3f700 1 -- 172.21.15.189:0/643910623 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3224758308 0 0) 0x7f0c34067c30 con 0x7f0c340a8fe0 2020-11-20T12:22:58.460 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c38c3f700 1 -- 172.21.15.189:0/643910623 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c34062b50 con 0x7f0c340a8fe0 2020-11-20T12:22:58.468 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c38c3f700 1 -- 172.21.15.189:0/643910623 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2589546900 0 0) 0x7f0c1c003240 con 0x7f0c340a9f10 2020-11-20T12:22:58.469 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c38c3f700 1 -- 172.21.15.189:0/643910623 >> v2:172.21.15.47:3300/0 conn(0x7f0c34137440 msgr2=0x7f0c341423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:58.469 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c38c3f700 1 --2- 172.21.15.189:0/643910623 >> v2:172.21.15.47:3300/0 conn(0x7f0c34137440 0x7f0c341423f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:58.469 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c38c3f700 1 -- 172.21.15.189:0/643910623 >> v1:172.21.15.47:6789/0 conn(0x7f0c340a8fe0 legacy=0x7f0c34136d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:58.469 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c38c3f700 1 -- 172.21.15.189:0/643910623 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c34065f00 con 0x7f0c340a9f10 2020-11-20T12:22:58.469 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c38c3f700 1 -- 172.21.15.189:0/643910623 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 142667399 0 0) 0x7f0c240014e0 con 0x7f0c340a9f10 2020-11-20T12:22:58.470 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c38c3f700 1 -- 172.21.15.189:0/643910623 >> v1:172.21.15.189:6789/0 conn(0x7f0c340a9f10 legacy=0x7f0c340a6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:58.470 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c38c3f700 1 --2- 172.21.15.189:0/643910623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c1c003910 0x7f0c1c003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:58.470 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c38c3f700 1 --2- 172.21.15.189:0/643910623 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c1c0044e0 0x7f0c1c004960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:58.470 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c38c3f700 1 --2- 172.21.15.189:0/643910623 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0c1c004e50 0x7f0c1c00b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:58.470 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c38c3f700 1 -- 172.21.15.189:0/643910623 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c34065c60 con 0x7f0c1c0044e0 2020-11-20T12:22:58.471 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c38c3f700 1 -- 172.21.15.189:0/643910623 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1c00bc20 con 0x7f0c1c003910 2020-11-20T12:22:58.471 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c38c3f700 1 -- 172.21.15.189:0/643910623 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1c00ce70 con 0x7f0c1c004e50 2020-11-20T12:22:58.471 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c39c41700 1 --2- 172.21.15.189:0/643910623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c1c003910 0x7f0c1c003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:58.471 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c39440700 1 --2- 172.21.15.189:0/643910623 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c1c0044e0 0x7f0c1c004960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:58.471 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c3a442700 1 --2- 172.21.15.189:0/643910623 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0c1c004e50 0x7f0c1c00b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:58.472 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c39440700 1 -- 172.21.15.189:0/643910623 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0c1c004e50 msgr2=0x7f0c1c00b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:58.472 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c39440700 1 --2- 172.21.15.189:0/643910623 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0c1c004e50 0x7f0c1c00b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:58.472 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c39440700 1 -- 172.21.15.189:0/643910623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c1c003910 msgr2=0x7f0c1c003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:22:58.472 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c39440700 1 --2- 172.21.15.189:0/643910623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c1c003910 0x7f0c1c003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:58.472 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c39440700 1 -- 172.21.15.189:0/643910623 --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c1c003240 con 0x7f0c1c0044e0 2020-11-20T12:22:58.473 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c3a442700 1 -- 172.21.15.189:0/643910623 reap_dead start 2020-11-20T12:22:58.473 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c39440700 1 --2- 172.21.15.189:0/643910623 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c1c0044e0 0x7f0c1c004960 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f0c30001a80 tx=0x7f0c30007b90).ready entity=mon.0 client_cookie=2145a42a41239791 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:58.473 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c38c3f700 1 -- 172.21.15.189:0/643910623 <== mon.0 v2:172.21.15.47:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c30007020 con 0x7f0c1c0044e0 2020-11-20T12:22:58.473 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c38c3f700 1 -- 172.21.15.189:0/643910623 <== mon.0 v2:172.21.15.47:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c30007180 con 0x7f0c1c0044e0 2020-11-20T12:22:58.473 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c38c3f700 1 -- 172.21.15.189:0/643910623 <== mon.0 v2:172.21.15.47:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3000c620 con 0x7f0c1c0044e0 2020-11-20T12:22:58.474 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c3ac43700 1 -- 172.21.15.189:0/643910623 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c1c0044e0 msgr2=0x7f0c1c004960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:58.474 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c3ac43700 1 --2- 172.21.15.189:0/643910623 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c1c0044e0 0x7f0c1c004960 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f0c30001a80 tx=0x7f0c30007b90).stop 2020-11-20T12:22:58.474 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c3ac43700 1 -- 172.21.15.189:0/643910623 shutdown_connections 2020-11-20T12:22:58.474 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c3ac43700 1 --2- 172.21.15.189:0/643910623 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c1c0044e0 0x7f0c1c004960 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:58.475 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c3ac43700 1 -- 172.21.15.189:0/643910623 >> 172.21.15.189:0/643910623 conn(0x7f0c34060170 msgr2=0x7f0c34060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:58.475 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c3ac43700 1 -- 172.21.15.189:0/643910623 shutdown_connections 2020-11-20T12:22:58.475 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c3ac43700 1 -- 172.21.15.189:0/643910623 wait complete. 2020-11-20T12:22:58.475 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.388+0000 7f0c3ac43700 1 Processor -- start 2020-11-20T12:22:58.475 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.392+0000 7f0c3ac43700 1 -- start start 2020-11-20T12:22:58.476 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.392+0000 7f0c3ac43700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0c1c0044e0 0x7f0c341148a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:58.476 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.392+0000 7f0c3ac43700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c341169b0 0x7f0c341479f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:58.476 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.392+0000 7f0c3ac43700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c34147ee0 0x7f0c34133050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:58.476 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.392+0000 7f0c3ac43700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c341159e0 con 0x7f0c341169b0 2020-11-20T12:22:58.476 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.392+0000 7f0c3ac43700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3409fc20 con 0x7f0c34147ee0 2020-11-20T12:22:58.477 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.392+0000 7f0c3ac43700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3409fec0 con 0x7f0c1c0044e0 2020-11-20T12:22:58.477 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.392+0000 7f0c3a442700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c34147ee0 0x7f0c34133050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:58.477 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.392+0000 7f0c3a442700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c34147ee0 0x7f0c34133050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52256/0 (socket says 172.21.15.189:52256) 2020-11-20T12:22:58.478 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.392+0000 7f0c3a442700 1 -- 172.21.15.189:0/2626560715 learned_addr learned my addr 172.21.15.189:0/2626560715 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:22:58.478 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.392+0000 7f0c39c41700 1 --2- 172.21.15.189:0/2626560715 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0c1c0044e0 0x7f0c341148a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:58.478 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.392+0000 7f0c39440700 1 --2- 172.21.15.189:0/2626560715 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c341169b0 0x7f0c341479f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:58.478 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.392+0000 7f0c3a442700 1 -- 172.21.15.189:0/2626560715 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0c1c0044e0 msgr2=0x7f0c341148a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:58.478 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.392+0000 7f0c3a442700 1 --2- 172.21.15.189:0/2626560715 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0c1c0044e0 0x7f0c341148a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:58.479 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.392+0000 7f0c3a442700 1 -- 172.21.15.189:0/2626560715 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c341169b0 msgr2=0x7f0c341479f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:58.479 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.392+0000 7f0c3a442700 1 --2- 172.21.15.189:0/2626560715 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c341169b0 0x7f0c341479f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:58.479 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.392+0000 7f0c3a442700 1 -- 172.21.15.189:0/2626560715 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c340a0160 con 0x7f0c34147ee0 2020-11-20T12:22:58.479 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.392+0000 7f0c39c41700 1 --2- 172.21.15.189:0/2626560715 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0c1c0044e0 0x7f0c341148a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T12:22:58.479 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.392+0000 7f0c3a442700 1 --2- 172.21.15.189:0/2626560715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c34147ee0 0x7f0c34133050 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7f0c34137820 tx=0x7f0c2c0091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:58.480 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.392+0000 7f0c21ffb700 1 -- 172.21.15.189:0/2626560715 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c00fbb0 con 0x7f0c34147ee0 2020-11-20T12:22:58.480 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.392+0000 7f0c21ffb700 1 -- 172.21.15.189:0/2626560715 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2c00fd10 con 0x7f0c34147ee0 2020-11-20T12:22:58.480 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.392+0000 7f0c3ac43700 1 -- 172.21.15.189:0/2626560715 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c340a0600 con 0x7f0c34147ee0 2020-11-20T12:22:58.480 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.392+0000 7f0c21ffb700 1 -- 172.21.15.189:0/2626560715 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c014690 con 0x7f0c34147ee0 2020-11-20T12:22:58.480 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.392+0000 7f0c3ac43700 1 -- 172.21.15.189:0/2626560715 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c34114d90 con 0x7f0c34147ee0 2020-11-20T12:22:58.481 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.392+0000 7f0c21ffb700 1 -- 172.21.15.189:0/2626560715 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f0c2c014a80 con 0x7f0c34147ee0 2020-11-20T12:22:58.481 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.396+0000 7f0c3ac43700 1 -- 172.21.15.189:0/2626560715 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c340a02f0 con 0x7f0c34147ee0 2020-11-20T12:22:58.481 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.396+0000 7f0c21ffb700 1 --2- 172.21.15.189:0/2626560715 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0c0c0557f0 0x7f0c0c057cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:22:58.481 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.396+0000 7f0c39c41700 1 --2- 172.21.15.189:0/2626560715 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0c0c0557f0 0x7f0c0c057cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:22:58.482 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.396+0000 7f0c21ffb700 1 -- 172.21.15.189:0/2626560715 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7191+0+0 (secure 0 0 0) 0x7f0c2c06f7a0 con 0x7f0c34147ee0 2020-11-20T12:22:58.482 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.396+0000 7f0c39c41700 1 --2- 172.21.15.189:0/2626560715 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0c0c0557f0 0x7f0c0c057cb0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f0c240013f0 tx=0x7f0c24009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:22:58.482 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.404+0000 7f0c21ffb700 1 -- 172.21.15.189:0/2626560715 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0c2c0450d0 con 0x7f0c34147ee0 2020-11-20T12:22:58.609 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.604+0000 7f0c3ac43700 1 -- 172.21.15.189:0/2626560715 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c340446d0 con 0x7f0c34147ee0 2020-11-20T12:22:58.610 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.604+0000 7f0c21ffb700 1 -- 172.21.15.189:0/2626560715 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f0c2c0486f0 con 0x7f0c34147ee0 2020-11-20T12:22:58.611 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:22:58.611 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5938,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.189:6835/3162779725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3162779725},{"type":"v1","addr":"172.21.15.189:6835","nonce":3162779725}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:21:50.854972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5297":{"gid":5297,"name":"b","rank":0,"incarnation":67,"state":"up:active","state_seq":14,"addr":"172.21.15.47:6833/3025803398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3025803398},{"type":"v1","addr":"172.21.15.47:6833","nonce":3025803398}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:22:58.637 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.608+0000 7f0c3ac43700 1 -- 172.21.15.189:0/2626560715 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0c0c0557f0 msgr2=0x7f0c0c057cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:58.638 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.608+0000 7f0c3ac43700 1 --2- 172.21.15.189:0/2626560715 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0c0c0557f0 0x7f0c0c057cb0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f0c240013f0 tx=0x7f0c24009040).stop 2020-11-20T12:22:58.644 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.608+0000 7f0c3ac43700 1 -- 172.21.15.189:0/2626560715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c34147ee0 msgr2=0x7f0c34133050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:22:58.648 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.608+0000 7f0c3ac43700 1 --2- 172.21.15.189:0/2626560715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c34147ee0 0x7f0c34133050 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7f0c34137820 tx=0x7f0c2c0091d0).stop 2020-11-20T12:22:58.648 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.608+0000 7f0c3ac43700 1 -- 172.21.15.189:0/2626560715 shutdown_connections 2020-11-20T12:22:58.648 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.608+0000 7f0c3ac43700 1 --2- 172.21.15.189:0/2626560715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c34147ee0 0x7f0c34133050 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:58.649 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.608+0000 7f0c3ac43700 1 --2- 172.21.15.189:0/2626560715 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f0c1c0044e0 0x7f0c341148a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:58.649 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.608+0000 7f0c3ac43700 1 --2- 172.21.15.189:0/2626560715 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f0c0c0557f0 0x7f0c0c057cb0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:58.649 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.608+0000 7f0c3ac43700 1 --2- 172.21.15.189:0/2626560715 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f0c341169b0 0x7f0c341479f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:22:58.649 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.608+0000 7f0c3ac43700 1 -- 172.21.15.189:0/2626560715 >> 172.21.15.189:0/2626560715 conn(0x7f0c34060170 msgr2=0x7f0c340a94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:22:58.649 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.608+0000 7f0c3ac43700 1 -- 172.21.15.189:0/2626560715 shutdown_connections 2020-11-20T12:22:58.650 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:22:58.608+0000 7f0c3ac43700 1 -- 172.21.15.189:0/2626560715 wait complete. 2020-11-20T12:22:58.650 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 71 2020-11-20T12:22:58.650 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:23:00.651 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:23:00.850 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.836+0000 7fd4edef0700 1 Processor -- start 2020-11-20T12:23:00.850 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.836+0000 7fd4edef0700 1 -- start start 2020-11-20T12:23:00.851 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4edef0700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd4e80612b0 0x7fd4e80a0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:23:00.851 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4edef0700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd4e8067b50 0x7fd4e8136fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:23:00.851 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4edef0700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fd4e81374a0 0x7fd4e8133a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:23:00.851 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4edef0700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4e8068020 con 0x7fd4e81374a0 2020-11-20T12:23:00.851 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4edef0700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4e80682c0 con 0x7fd4e8067b50 2020-11-20T12:23:00.852 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4edef0700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e81378f0 con 0x7fd4e80612b0 2020-11-20T12:23:00.852 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4e7fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd4e8067b50 0x7fd4e8136fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:23:00.852 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4e7fff700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd4e8067b50 0x7fd4e8136fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52262/0 (socket says 172.21.15.189:52262) 2020-11-20T12:23:00.852 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4e7fff700 1 -- 172.21.15.189:0/2791183793 learned_addr learned my addr 172.21.15.189:0/2791183793 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:23:00.852 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4ed6ef700 1 --2- 172.21.15.189:0/2791183793 >> v2:172.21.15.47:3300/0 conn(0x7fd4e81374a0 0x7fd4e8133a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:23:00.853 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4eceee700 1 --2- 172.21.15.189:0/2791183793 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd4e80612b0 0x7fd4e80a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:23:00.853 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4e7fff700 1 -- 172.21.15.189:0/2791183793 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd4e80612b0 msgr2=0x7fd4e80a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:23:00.853 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4e7fff700 1 --2- 172.21.15.189:0/2791183793 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd4e80612b0 0x7fd4e80a0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:00.853 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4e7fff700 1 -- 172.21.15.189:0/2791183793 >> v2:172.21.15.47:3300/0 conn(0x7fd4e81374a0 msgr2=0x7fd4e8133a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:23:00.853 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4e7fff700 1 --2- 172.21.15.189:0/2791183793 >> v2:172.21.15.47:3300/0 conn(0x7fd4e81374a0 0x7fd4e8133a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:00.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4e7fff700 1 -- 172.21.15.189:0/2791183793 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e8061690 con 0x7fd4e8067b50 2020-11-20T12:23:00.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4e7fff700 1 --2- 172.21.15.189:0/2791183793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4e8067b50 0x7fd4e8136fb0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7fd4e80656c0 tx=0x7fd4dc009290).ready entity=mon.1 client_cookie=f1764da455f168de server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:23:00.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4e77fe700 1 -- 172.21.15.189:0/2791183793 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc00bbc0 con 0x7fd4e8067b50 2020-11-20T12:23:00.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4e77fe700 1 -- 172.21.15.189:0/2791183793 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4dc00bd20 con 0x7fd4e8067b50 2020-11-20T12:23:00.854 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4e77fe700 1 -- 172.21.15.189:0/2791183793 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc00f680 con 0x7fd4e8067b50 2020-11-20T12:23:00.855 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4edef0700 1 -- 172.21.15.189:0/2791183793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4e8067b50 msgr2=0x7fd4e8136fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:23:00.855 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4edef0700 1 --2- 172.21.15.189:0/2791183793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4e8067b50 0x7fd4e8136fb0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7fd4e80656c0 tx=0x7fd4dc009290).stop 2020-11-20T12:23:00.855 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4edef0700 1 -- 172.21.15.189:0/2791183793 shutdown_connections 2020-11-20T12:23:00.855 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4edef0700 1 --2- 172.21.15.189:0/2791183793 >> v2:172.21.15.47:3300/0 conn(0x7fd4e81374a0 0x7fd4e8133a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:00.855 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4edef0700 1 --2- 172.21.15.189:0/2791183793 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd4e80612b0 0x7fd4e80a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:00.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4edef0700 1 --2- 172.21.15.189:0/2791183793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4e8067b50 0x7fd4e8136fb0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:00.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4edef0700 1 -- 172.21.15.189:0/2791183793 >> 172.21.15.189:0/2791183793 conn(0x7fd4e8063210 msgr2=0x7fd4e8060a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:23:00.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4edef0700 1 -- 172.21.15.189:0/2791183793 shutdown_connections 2020-11-20T12:23:00.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4edef0700 1 -- 172.21.15.189:0/2791183793 wait complete. 2020-11-20T12:23:00.856 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4edef0700 1 Processor -- start 2020-11-20T12:23:00.857 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.840+0000 7fd4edef0700 1 -- start start 2020-11-20T12:23:00.857 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.844+0000 7fd4edef0700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd4e80612b0 0x7fd4e81499e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:23:00.857 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.844+0000 7fd4edef0700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4e8067b50 0x7fd4e80ad900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:23:00.857 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.844+0000 7fd4edef0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4e81374a0 0x7fd4e80a6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:23:00.857 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.844+0000 7fd4edef0700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e812c240 con 0x7fd4e8067b50 2020-11-20T12:23:00.858 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.844+0000 7fd4edef0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e80adfa0 con 0x7fd4e81374a0 2020-11-20T12:23:00.858 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.844+0000 7fd4edef0700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e80a4960 con 0x7fd4e80612b0 2020-11-20T12:23:00.858 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.844+0000 7fd4ed6ef700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4e81374a0 0x7fd4e80a6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:23:00.858 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.844+0000 7fd4ed6ef700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4e81374a0 0x7fd4e80a6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52272/0 (socket says 172.21.15.189:52272) 2020-11-20T12:23:00.858 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.844+0000 7fd4ed6ef700 1 -- 172.21.15.189:0/1940598057 learned_addr learned my addr 172.21.15.189:0/1940598057 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:23:00.859 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.844+0000 7fd4eceee700 1 --2- 172.21.15.189:0/1940598057 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd4e80612b0 0x7fd4e81499e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:23:00.859 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.844+0000 7fd4e7fff700 1 --2- 172.21.15.189:0/1940598057 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4e8067b50 0x7fd4e80ad900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:23:00.859 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.844+0000 7fd4ed6ef700 1 -- 172.21.15.189:0/1940598057 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd4e80612b0 msgr2=0x7fd4e81499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:23:00.859 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.844+0000 7fd4ed6ef700 1 --2- 172.21.15.189:0/1940598057 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd4e80612b0 0x7fd4e81499e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:00.859 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.844+0000 7fd4ed6ef700 1 -- 172.21.15.189:0/1940598057 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4e8067b50 msgr2=0x7fd4e80ad900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:23:00.860 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.844+0000 7fd4ed6ef700 1 --2- 172.21.15.189:0/1940598057 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4e8067b50 0x7fd4e80ad900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:00.860 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.844+0000 7fd4ed6ef700 1 -- 172.21.15.189:0/1940598057 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e80a4c00 con 0x7fd4e81374a0 2020-11-20T12:23:00.860 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.844+0000 7fd4ed6ef700 1 --2- 172.21.15.189:0/1940598057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4e81374a0 0x7fd4e80a6b00 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7fd4e000eba0 tx=0x7fd4e0006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:23:00.860 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.844+0000 7fd4e4ff9700 1 -- 172.21.15.189:0/1940598057 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e000b810 con 0x7fd4e81374a0 2020-11-20T12:23:00.860 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.844+0000 7fd4e4ff9700 1 -- 172.21.15.189:0/1940598057 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e000b970 con 0x7fd4e81374a0 2020-11-20T12:23:00.861 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.844+0000 7fd4edef0700 1 -- 172.21.15.189:0/1940598057 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e80a7330 con 0x7fd4e81374a0 2020-11-20T12:23:00.861 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.844+0000 7fd4e4ff9700 1 -- 172.21.15.189:0/1940598057 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e000bd10 con 0x7fd4e81374a0 2020-11-20T12:23:00.861 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.844+0000 7fd4edef0700 1 -- 172.21.15.189:0/1940598057 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e80a5240 con 0x7fd4e81374a0 2020-11-20T12:23:00.862 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.844+0000 7fd4edef0700 1 -- 172.21.15.189:0/1940598057 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e80addf0 con 0x7fd4e81374a0 2020-11-20T12:23:00.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.844+0000 7fd4e4ff9700 1 -- 172.21.15.189:0/1940598057 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fd4e0012600 con 0x7fd4e81374a0 2020-11-20T12:23:00.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.848+0000 7fd4e4ff9700 1 --2- 172.21.15.189:0/1940598057 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd4c00557c0 0x7fd4c0057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:23:00.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.848+0000 7fd4e4ff9700 1 -- 172.21.15.189:0/1940598057 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7191+0+0 (secure 0 0 0) 0x7fd4e006ecf0 con 0x7fd4e81374a0 2020-11-20T12:23:00.863 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.848+0000 7fd4eceee700 1 --2- 172.21.15.189:0/1940598057 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd4c00557c0 0x7fd4c0057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:23:00.864 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.848+0000 7fd4eceee700 1 --2- 172.21.15.189:0/1940598057 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd4c00557c0 0x7fd4c0057c80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fd4e80ad380 tx=0x7fd4d8008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:23:00.864 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:00.852+0000 7fd4e4ff9700 1 -- 172.21.15.189:0/1940598057 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd4e0044730 con 0x7fd4e81374a0 2020-11-20T12:23:01.061 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:01.056+0000 7fd4edef0700 1 -- 172.21.15.189:0/1940598057 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4e80446d0 con 0x7fd4e81374a0 2020-11-20T12:23:01.061 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:01.056+0000 7fd4e4ff9700 1 -- 172.21.15.189:0/1940598057 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fd4e0047d50 con 0x7fd4e81374a0 2020-11-20T12:23:01.062 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:23:01.062 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5938,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.189:6835/3162779725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3162779725},{"type":"v1","addr":"172.21.15.189:6835","nonce":3162779725}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:21:50.854972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5297":{"gid":5297,"name":"b","rank":0,"incarnation":67,"state":"up:active","state_seq":14,"addr":"172.21.15.47:6833/3025803398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3025803398},{"type":"v1","addr":"172.21.15.47:6833","nonce":3025803398}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:23:01.064 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:01.056+0000 7fd4edef0700 1 -- 172.21.15.189:0/1940598057 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd4c00557c0 msgr2=0x7fd4c0057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:23:01.064 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:01.056+0000 7fd4edef0700 1 --2- 172.21.15.189:0/1940598057 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd4c00557c0 0x7fd4c0057c80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fd4e80ad380 tx=0x7fd4d8008040).stop 2020-11-20T12:23:01.064 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:01.056+0000 7fd4edef0700 1 -- 172.21.15.189:0/1940598057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4e81374a0 msgr2=0x7fd4e80a6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:23:01.065 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:01.056+0000 7fd4edef0700 1 --2- 172.21.15.189:0/1940598057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4e81374a0 0x7fd4e80a6b00 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7fd4e000eba0 tx=0x7fd4e0006df0).stop 2020-11-20T12:23:01.065 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:01.060+0000 7fd4edef0700 1 -- 172.21.15.189:0/1940598057 shutdown_connections 2020-11-20T12:23:01.065 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:01.060+0000 7fd4edef0700 1 --2- 172.21.15.189:0/1940598057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd4e81374a0 0x7fd4e80a6b00 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:01.065 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:01.060+0000 7fd4edef0700 1 --2- 172.21.15.189:0/1940598057 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fd4e80612b0 0x7fd4e81499e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:01.065 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:01.060+0000 7fd4edef0700 1 --2- 172.21.15.189:0/1940598057 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fd4c00557c0 0x7fd4c0057c80 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:01.066 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:01.060+0000 7fd4edef0700 1 --2- 172.21.15.189:0/1940598057 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fd4e8067b50 0x7fd4e80ad900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:01.066 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:01.060+0000 7fd4edef0700 1 -- 172.21.15.189:0/1940598057 >> 172.21.15.189:0/1940598057 conn(0x7fd4e8063210 msgr2=0x7fd4e8133f30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:23:01.066 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:01.060+0000 7fd4edef0700 1 -- 172.21.15.189:0/1940598057 shutdown_connections 2020-11-20T12:23:01.066 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:01.060+0000 7fd4edef0700 1 -- 172.21.15.189:0/1940598057 wait complete. 2020-11-20T12:23:01.067 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 71 2020-11-20T12:23:01.077 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:23:03.078 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:23:03.214 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.208+0000 7f59ae17f700 1 Processor -- start 2020-11-20T12:23:03.214 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.208+0000 7f59ae17f700 1 -- start start 2020-11-20T12:23:03.217 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.208+0000 7f59ae17f700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f59a81386b0 0x7f59a80a9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:23:03.217 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.208+0000 7f59ae17f700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59a8135b60 0x7f59a80a5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:23:03.217 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.208+0000 7f59ae17f700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a80a9a60 con 0x7f59a80a50e0 2020-11-20T12:23:03.218 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.208+0000 7f59ae17f700 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59a8135ff0 con 0x7f59a81386b0 2020-11-20T12:23:03.218 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.208+0000 7f59ae17f700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a80a5a50 con 0x7f59a8135b60 2020-11-20T12:23:03.218 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.208+0000 7f59ad17d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f59a81386b0 0x7f59a80a9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:23:03.218 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.208+0000 7f59ad17d700 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f59a81386b0 0x7f59a80a9410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52278/0 (socket says 172.21.15.189:52278) 2020-11-20T12:23:03.219 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.208+0000 7f59ad17d700 1 -- 172.21.15.189:0/2542118658 learned_addr learned my addr 172.21.15.189:0/2542118658 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:23:03.219 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.212+0000 7f59ad97e700 1 --2- 172.21.15.189:0/2542118658 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59a8135b60 0x7f59a80a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:23:03.219 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.212+0000 7f59a7fff700 1 -- 172.21.15.189:0/2542118658 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322613244 0 0) 0x7f59a80a9a60 con 0x7f59a80a50e0 2020-11-20T12:23:03.219 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.212+0000 7f59a7fff700 1 -- 172.21.15.189:0/2542118658 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5994003240 con 0x7f59a80a50e0 2020-11-20T12:23:03.220 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.212+0000 7f59ad17d700 1 -- 172.21.15.189:0/2542118658 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59a8135b60 msgr2=0x7f59a80a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:23:03.220 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.212+0000 7f59ad17d700 1 --2- 172.21.15.189:0/2542118658 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59a8135b60 0x7f59a80a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:03.220 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.212+0000 7f59ad17d700 1 -- 172.21.15.189:0/2542118658 >> v1:172.21.15.47:6789/0 conn(0x7f59a80a50e0 legacy=0x7f59a8135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:23:03.220 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.212+0000 7f59ad17d700 1 -- 172.21.15.189:0/2542118658 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a80a9d00 con 0x7f59a81386b0 2020-11-20T12:23:03.221 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.212+0000 7f59ad17d700 1 --2- 172.21.15.189:0/2542118658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59a81386b0 0x7f59a80a9410 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0x7f599c006fd0 tx=0x7f599c006eb0).ready entity=mon.1 client_cookie=59dfa1baea68d3d7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:23:03.221 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.212+0000 7f59a7fff700 1 -- 172.21.15.189:0/2542118658 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c0097b0 con 0x7f59a81386b0 2020-11-20T12:23:03.221 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.212+0000 7f59a7fff700 1 -- 172.21.15.189:0/2542118658 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f599c009910 con 0x7f59a81386b0 2020-11-20T12:23:03.221 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.212+0000 7f59a7fff700 1 -- 172.21.15.189:0/2542118658 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c009c90 con 0x7f59a81386b0 2020-11-20T12:23:03.225 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.212+0000 7f59ae17f700 1 -- 172.21.15.189:0/2542118658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59a81386b0 msgr2=0x7f59a80a9410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:23:03.225 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.212+0000 7f59ae17f700 1 --2- 172.21.15.189:0/2542118658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59a81386b0 0x7f59a80a9410 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0x7f599c006fd0 tx=0x7f599c006eb0).stop 2020-11-20T12:23:03.225 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.212+0000 7f59ae17f700 1 -- 172.21.15.189:0/2542118658 shutdown_connections 2020-11-20T12:23:03.226 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.212+0000 7f59ae17f700 1 --2- 172.21.15.189:0/2542118658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59a81386b0 0x7f59a80a9410 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:03.226 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.212+0000 7f59ae17f700 1 --2- 172.21.15.189:0/2542118658 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59a8135b60 0x7f59a80a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:03.226 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.212+0000 7f59ae17f700 1 -- 172.21.15.189:0/2542118658 >> 172.21.15.189:0/2542118658 conn(0x7f59a8060170 msgr2=0x7f59a8060580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:23:03.227 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.212+0000 7f59ae17f700 1 -- 172.21.15.189:0/2542118658 shutdown_connections 2020-11-20T12:23:03.227 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.212+0000 7f59ae17f700 1 -- 172.21.15.189:0/2542118658 wait complete. 2020-11-20T12:23:03.227 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.216+0000 7f59ae17f700 1 Processor -- start 2020-11-20T12:23:03.227 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.216+0000 7f59ae17f700 1 -- start start 2020-11-20T12:23:03.228 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.216+0000 7f59ae17f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59a80a50e0 0x7f59a80ac9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:23:03.228 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.216+0000 7f59ae17f700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59a8135b60 0x7f59a80a86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:23:03.228 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.216+0000 7f59ae17f700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59a81386b0 0x7f59a80a8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:23:03.228 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.216+0000 7f59ae17f700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a8139fe0 con 0x7f59a81386b0 2020-11-20T12:23:03.229 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.216+0000 7f59ae17f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a813a280 con 0x7f59a80a50e0 2020-11-20T12:23:03.229 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.216+0000 7f59ae17f700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a8139d20 con 0x7f59a8135b60 2020-11-20T12:23:03.229 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.216+0000 7f59ad17d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59a80a50e0 0x7f59a80ac9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:23:03.229 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.216+0000 7f59ad17d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59a80a50e0 0x7f59a80ac9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52284/0 (socket says 172.21.15.189:52284) 2020-11-20T12:23:03.230 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.216+0000 7f59ad17d700 1 -- 172.21.15.189:0/2796057329 learned_addr learned my addr 172.21.15.189:0/2796057329 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:23:03.230 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.216+0000 7f59ad97e700 1 --2- 172.21.15.189:0/2796057329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59a81386b0 0x7f59a80a8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:23:03.230 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.216+0000 7f59ac97c700 1 --2- 172.21.15.189:0/2796057329 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59a8135b60 0x7f59a80a86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:23:03.230 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.216+0000 7f59ad17d700 1 -- 172.21.15.189:0/2796057329 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59a8135b60 msgr2=0x7f59a80a86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:23:03.231 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.216+0000 7f59ad17d700 1 --2- 172.21.15.189:0/2796057329 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59a8135b60 0x7f59a80a86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:03.231 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.216+0000 7f59ad17d700 1 -- 172.21.15.189:0/2796057329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59a81386b0 msgr2=0x7f59a80a8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:23:03.231 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.216+0000 7f59ad17d700 1 --2- 172.21.15.189:0/2796057329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59a81386b0 0x7f59a80a8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:03.231 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.216+0000 7f59ad17d700 1 -- 172.21.15.189:0/2796057329 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a80ad3b0 con 0x7f59a80a50e0 2020-11-20T12:23:03.232 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.216+0000 7f59ad97e700 1 --2- 172.21.15.189:0/2796057329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59a81386b0 0x7f59a80a8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:23:03.232 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.216+0000 7f59ac97c700 1 --2- 172.21.15.189:0/2796057329 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59a8135b60 0x7f59a80a86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T12:23:03.232 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.216+0000 7f59ad17d700 1 --2- 172.21.15.189:0/2796057329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59a80a50e0 0x7f59a80ac9a0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7f59a8067db0 tx=0x7f599c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:23:03.232 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.216+0000 7f59a57fa700 1 -- 172.21.15.189:0/2796057329 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c011d50 con 0x7f59a80a50e0 2020-11-20T12:23:03.233 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.220+0000 7f59a57fa700 1 -- 172.21.15.189:0/2796057329 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f599c01e460 con 0x7f59a80a50e0 2020-11-20T12:23:03.233 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.220+0000 7f59ae17f700 1 -- 172.21.15.189:0/2796057329 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a8139880 con 0x7f59a80a50e0 2020-11-20T12:23:03.233 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.220+0000 7f59a57fa700 1 -- 172.21.15.189:0/2796057329 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c01e7e0 con 0x7f59a80a50e0 2020-11-20T12:23:03.234 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.220+0000 7f59ae17f700 1 -- 172.21.15.189:0/2796057329 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a813c170 con 0x7f59a80a50e0 2020-11-20T12:23:03.234 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.220+0000 7f59ae17f700 1 -- 172.21.15.189:0/2796057329 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59a80a96e0 con 0x7f59a80a50e0 2020-11-20T12:23:03.235 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.220+0000 7f59a57fa700 1 -- 172.21.15.189:0/2796057329 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7f599c025030 con 0x7f59a80a50e0 2020-11-20T12:23:03.235 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.220+0000 7f59a57fa700 1 --2- 172.21.15.189:0/2796057329 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f59800557c0 0x7f5980057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:23:03.235 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.220+0000 7f59a57fa700 1 -- 172.21.15.189:0/2796057329 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7191+0+0 (secure 0 0 0) 0x7f599c070230 con 0x7f59a80a50e0 2020-11-20T12:23:03.236 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.220+0000 7f59ac97c700 1 --2- 172.21.15.189:0/2796057329 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f59800557c0 0x7f5980057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:23:03.236 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.220+0000 7f59ac97c700 1 --2- 172.21.15.189:0/2796057329 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f59800557c0 0x7f5980057c80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f59a813c590 tx=0x7f59a0009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:23:03.236 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.224+0000 7f59a57fa700 1 -- 172.21.15.189:0/2796057329 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f599c045d20 con 0x7f59a80a50e0 2020-11-20T12:23:03.430 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.424+0000 7f59ae17f700 1 -- 172.21.15.189:0/2796057329 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59a80446d0 con 0x7f59a80a50e0 2020-11-20T12:23:03.430 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.424+0000 7f59a57fa700 1 -- 172.21.15.189:0/2796057329 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f599c023070 con 0x7f59a80a50e0 2020-11-20T12:23:03.431 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:23:03.431 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5938,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.189:6835/3162779725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3162779725},{"type":"v1","addr":"172.21.15.189:6835","nonce":3162779725}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:21:50.854972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5297":{"gid":5297,"name":"b","rank":0,"incarnation":67,"state":"up:active","state_seq":14,"addr":"172.21.15.47:6833/3025803398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3025803398},{"type":"v1","addr":"172.21.15.47:6833","nonce":3025803398}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:23:03.432 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.428+0000 7f59ae17f700 1 -- 172.21.15.189:0/2796057329 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f59800557c0 msgr2=0x7f5980057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:23:03.433 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.428+0000 7f59ae17f700 1 --2- 172.21.15.189:0/2796057329 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f59800557c0 0x7f5980057c80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f59a813c590 tx=0x7f59a0009040).stop 2020-11-20T12:23:03.433 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.428+0000 7f59ae17f700 1 -- 172.21.15.189:0/2796057329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59a80a50e0 msgr2=0x7f59a80ac9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:23:03.433 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.428+0000 7f59ae17f700 1 --2- 172.21.15.189:0/2796057329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59a80a50e0 0x7f59a80ac9a0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7f59a8067db0 tx=0x7f599c015000).stop 2020-11-20T12:23:03.433 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.428+0000 7f59ae17f700 1 -- 172.21.15.189:0/2796057329 shutdown_connections 2020-11-20T12:23:03.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.428+0000 7f59ae17f700 1 --2- 172.21.15.189:0/2796057329 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7f59800557c0 0x7f5980057c80 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:03.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.428+0000 7f59ae17f700 1 --2- 172.21.15.189:0/2796057329 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7f59a81386b0 0x7f59a80a8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:03.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.428+0000 7f59ae17f700 1 --2- 172.21.15.189:0/2796057329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59a80a50e0 0x7f59a80ac9a0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:03.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.428+0000 7f59ae17f700 1 --2- 172.21.15.189:0/2796057329 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7f59a8135b60 0x7f59a80a86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:03.434 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.428+0000 7f59ae17f700 1 -- 172.21.15.189:0/2796057329 >> 172.21.15.189:0/2796057329 conn(0x7f59a8060170 msgr2=0x7f59a8060550 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:23:03.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.428+0000 7f59ae17f700 1 -- 172.21.15.189:0/2796057329 shutdown_connections 2020-11-20T12:23:03.435 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:03.428+0000 7f59ae17f700 1 -- 172.21.15.189:0/2796057329 wait complete. 2020-11-20T12:23:03.435 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 71 2020-11-20T12:23:03.445 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2020-11-20T12:23:05.447 INFO:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T12:23:05.580 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.576+0000 7fa89b0b9700 1 Processor -- start 2020-11-20T12:23:05.581 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.576+0000 7fa89b0b9700 1 -- start start 2020-11-20T12:23:05.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.576+0000 7fa89b0b9700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa8940ad3e0 0x7fa894066d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:23:05.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.576+0000 7fa89b0b9700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa89413a6e0 0x7fa89413ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T12:23:05.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.576+0000 7fa89b0b9700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa894067730 con 0x7fa8940ad000 2020-11-20T12:23:05.583 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.576+0000 7fa89b0b9700 1 -- --> v2:172.21.15.47:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa894066050 con 0x7fa8940ad3e0 2020-11-20T12:23:05.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.576+0000 7fa89b0b9700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa894138190 con 0x7fa89413a6e0 2020-11-20T12:23:05.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.576+0000 7fa89a8b8700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa89413a6e0 0x7fa89413ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:23:05.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.576+0000 7fa89a0b7700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa8940ad000 0x7fa89413e480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.189:48578/0 (socket says 172.21.15.189:48578) 2020-11-20T12:23:05.584 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.576+0000 7fa89a0b7700 1 -- 172.21.15.189:0/1643660039 learned_addr learned my addr 172.21.15.189:0/1643660039 (peer_addr_for_me v1:172.21.15.189:0/0) 2020-11-20T12:23:05.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.576+0000 7fa8998b6700 1 --2- >> v2:172.21.15.47:3300/0 conn(0x7fa8940ad3e0 0x7fa894066d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:23:05.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.576+0000 7fa8990b5700 1 -- 172.21.15.189:0/1643660039 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500244573 0 0) 0x7fa894067730 con 0x7fa8940ad000 2020-11-20T12:23:05.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.576+0000 7fa8990b5700 1 -- 172.21.15.189:0/1643660039 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa87c003240 con 0x7fa8940ad000 2020-11-20T12:23:05.585 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.576+0000 7fa89a8b8700 1 -- 172.21.15.189:0/1643660039 >> v2:172.21.15.47:3300/0 conn(0x7fa8940ad3e0 msgr2=0x7fa894066d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T12:23:05.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.576+0000 7fa89a8b8700 1 --2- 172.21.15.189:0/1643660039 >> v2:172.21.15.47:3300/0 conn(0x7fa8940ad3e0 0x7fa894066d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:05.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.576+0000 7fa89a8b8700 1 -- 172.21.15.189:0/1643660039 >> v1:172.21.15.47:6789/0 conn(0x7fa8940ad000 legacy=0x7fa89413e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:23:05.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.576+0000 7fa89a8b8700 1 -- 172.21.15.189:0/1643660039 --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa89413dd40 con 0x7fa89413a6e0 2020-11-20T12:23:05.586 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.576+0000 7fa8998b6700 1 --2- 172.21.15.189:0/1643660039 >> v2:172.21.15.47:3300/0 conn(0x7fa8940ad3e0 0x7fa894066d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:23:05.587 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.576+0000 7fa89a8b8700 1 --2- 172.21.15.189:0/1643660039 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa89413a6e0 0x7fa89413ad00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fa88c009c90 tx=0x7fa88c0091d0).ready entity=mon.2 client_cookie=aedbce86063b0549 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:23:05.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.576+0000 7fa8990b5700 1 -- 172.21.15.189:0/1643660039 <== mon.2 v2:172.21.15.47:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c00ba50 con 0x7fa89413a6e0 2020-11-20T12:23:05.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa8990b5700 1 -- 172.21.15.189:0/1643660039 <== mon.2 v2:172.21.15.47:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa88c00bbb0 con 0x7fa89413a6e0 2020-11-20T12:23:05.590 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa8990b5700 1 -- 172.21.15.189:0/1643660039 <== mon.2 v2:172.21.15.47:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c011400 con 0x7fa89413a6e0 2020-11-20T12:23:05.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa89b0b9700 1 -- 172.21.15.189:0/1643660039 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa89413a6e0 msgr2=0x7fa89413ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:23:05.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa89b0b9700 1 --2- 172.21.15.189:0/1643660039 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa89413a6e0 0x7fa89413ad00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fa88c009c90 tx=0x7fa88c0091d0).stop 2020-11-20T12:23:05.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa89b0b9700 1 -- 172.21.15.189:0/1643660039 shutdown_connections 2020-11-20T12:23:05.591 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa89b0b9700 1 --2- 172.21.15.189:0/1643660039 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa89413a6e0 0x7fa89413ad00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:05.592 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa89b0b9700 1 --2- 172.21.15.189:0/1643660039 >> v2:172.21.15.47:3300/0 conn(0x7fa8940ad3e0 0x7fa894066d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:05.592 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa89b0b9700 1 -- 172.21.15.189:0/1643660039 >> 172.21.15.189:0/1643660039 conn(0x7fa894062080 msgr2=0x7fa894062460 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:23:05.592 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa89b0b9700 1 -- 172.21.15.189:0/1643660039 shutdown_connections 2020-11-20T12:23:05.593 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa89b0b9700 1 -- 172.21.15.189:0/1643660039 wait complete. 2020-11-20T12:23:05.593 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa89b0b9700 1 Processor -- start 2020-11-20T12:23:05.593 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa89b0b9700 1 -- start start 2020-11-20T12:23:05.594 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa89b0b9700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8940ad000 0x7fa894140610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:23:05.594 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa89b0b9700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa8940ad3e0 0x7fa8940aac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:23:05.594 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa89b0b9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa89414cc30 0x7fa89414d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:23:05.595 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa89b0b9700 1 -- --> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8940ab0f0 con 0x7fa8940ad000 2020-11-20T12:23:05.595 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa89b0b9700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8941414e0 con 0x7fa89414cc30 2020-11-20T12:23:05.595 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa89b0b9700 1 -- --> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8940a98b0 con 0x7fa8940ad3e0 2020-11-20T12:23:05.596 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa89a8b8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa89414cc30 0x7fa89414d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:23:05.596 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa8998b6700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa8940ad3e0 0x7fa8940aac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:23:05.597 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa89a0b7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8940ad000 0x7fa894140610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:23:05.597 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa8998b6700 1 --2- >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa8940ad3e0 0x7fa8940aac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3301/0 says I am v2:172.21.15.189:45332/0 (socket says 172.21.15.189:45332) 2020-11-20T12:23:05.597 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa8998b6700 1 -- 172.21.15.189:0/77389594 learned_addr learned my addr 172.21.15.189:0/77389594 (peer_addr_for_me v2:172.21.15.189:0/0) 2020-11-20T12:23:05.598 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa89a0b7700 1 --2- >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8940ad000 0x7fa894140610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.47:3300/0 says I am v2:172.21.15.189:58524/0 (socket says 172.21.15.189:58524) 2020-11-20T12:23:05.598 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa89a8b8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa89414cc30 0x7fa89414d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:52302/0 (socket says 172.21.15.189:52302) 2020-11-20T12:23:05.598 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa89a8b8700 1 -- 172.21.15.189:0/77389594 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa8940ad3e0 msgr2=0x7fa8940aac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:23:05.598 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa89a8b8700 1 --2- 172.21.15.189:0/77389594 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa8940ad3e0 0x7fa8940aac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:05.598 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa89a8b8700 1 -- 172.21.15.189:0/77389594 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8940ad000 msgr2=0x7fa894140610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:23:05.599 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa89a8b8700 1 --2- 172.21.15.189:0/77389594 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8940ad000 0x7fa894140610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:05.599 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa89a8b8700 1 -- 172.21.15.189:0/77389594 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8940a9b50 con 0x7fa89414cc30 2020-11-20T12:23:05.599 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa89a0b7700 1 --2- 172.21.15.189:0/77389594 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8940ad000 0x7fa894140610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:23:05.599 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.580+0000 7fa8998b6700 1 --2- 172.21.15.189:0/77389594 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa8940ad3e0 0x7fa8940aac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T12:23:05.600 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.584+0000 7fa89a8b8700 1 --2- 172.21.15.189:0/77389594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa89414cc30 0x7fa89414d090 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7fa88c013000 tx=0x7fa88c016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:23:05.600 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.584+0000 7fa8867fc700 1 -- 172.21.15.189:0/77389594 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c006ea0 con 0x7fa89414cc30 2020-11-20T12:23:05.600 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.584+0000 7fa8867fc700 1 -- 172.21.15.189:0/77389594 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa88c01c5f0 con 0x7fa89414cc30 2020-11-20T12:23:05.600 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.584+0000 7fa8867fc700 1 -- 172.21.15.189:0/77389594 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c01c990 con 0x7fa89414cc30 2020-11-20T12:23:05.600 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.584+0000 7fa89b0b9700 1 -- 172.21.15.189:0/77389594 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8940a9df0 con 0x7fa89414cc30 2020-11-20T12:23:05.601 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.584+0000 7fa89b0b9700 1 -- 172.21.15.189:0/77389594 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8940a9480 con 0x7fa89414cc30 2020-11-20T12:23:05.601 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.584+0000 7fa89b0b9700 1 -- 172.21.15.189:0/77389594 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8940a9f80 con 0x7fa89414cc30 2020-11-20T12:23:05.602 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.584+0000 7fa8867fc700 1 -- 172.21.15.189:0/77389594 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 4) v1 ==== 66655+0+0 (secure 0 0 0) 0x7fa88c024030 con 0x7fa89414cc30 2020-11-20T12:23:05.602 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.584+0000 7fa8867fc700 1 --2- 172.21.15.189:0/77389594 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa86c0557c0 0x7fa86c057c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T12:23:05.602 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.584+0000 7fa8867fc700 1 -- 172.21.15.189:0/77389594 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7191+0+0 (secure 0 0 0) 0x7fa88c070420 con 0x7fa89414cc30 2020-11-20T12:23:05.603 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.584+0000 7fa89a0b7700 1 --2- 172.21.15.189:0/77389594 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa86c0557c0 0x7fa86c057c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T12:23:05.603 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.588+0000 7fa89a0b7700 1 --2- 172.21.15.189:0/77389594 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa86c0557c0 0x7fa86c057c80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fa890007fd0 tx=0x7fa890009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T12:23:05.603 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.592+0000 7fa8867fc700 1 -- 172.21.15.189:0/77389594 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa88c045f10 con 0x7fa89414cc30 2020-11-20T12:23:05.798 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.792+0000 7fa89b0b9700 1 -- 172.21.15.189:0/77389594 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8940b13b0 con 0x7fa89414cc30 2020-11-20T12:23:05.799 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.792+0000 7fa8867fc700 1 -- 172.21.15.189:0/77389594 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fa88c022020 con 0x7fa89414cc30 2020-11-20T12:23:05.799 INFO:teuthology.orchestra.run.smithi189.stdout: 2020-11-20T12:23:05.799 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5938,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.189:6835/3162779725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3162779725},{"type":"v1","addr":"172.21.15.189:6835","nonce":3162779725}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":71}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T12:04:54.012110+0000","modified":"2020-11-20T12:21:50.854972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5297":{"gid":5297,"name":"b","rank":0,"incarnation":67,"state":"up:active","state_seq":14,"addr":"172.21.15.47:6833/3025803398","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.47:6832","nonce":3025803398},{"type":"v1","addr":"172.21.15.47:6833","nonce":3025803398}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T12:23:05.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.796+0000 7fa89b0b9700 1 -- 172.21.15.189:0/77389594 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa86c0557c0 msgr2=0x7fa86c057c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:23:05.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.796+0000 7fa89b0b9700 1 --2- 172.21.15.189:0/77389594 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa86c0557c0 0x7fa86c057c80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fa890007fd0 tx=0x7fa890009040).stop 2020-11-20T12:23:05.802 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.796+0000 7fa89b0b9700 1 -- 172.21.15.189:0/77389594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa89414cc30 msgr2=0x7fa89414d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T12:23:05.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.796+0000 7fa89b0b9700 1 --2- 172.21.15.189:0/77389594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa89414cc30 0x7fa89414d090 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7fa88c013000 tx=0x7fa88c016000).stop 2020-11-20T12:23:05.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.796+0000 7fa89b0b9700 1 -- 172.21.15.189:0/77389594 shutdown_connections 2020-11-20T12:23:05.803 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.796+0000 7fa89b0b9700 1 --2- 172.21.15.189:0/77389594 >> [v2:172.21.15.189:6832/11671,v1:172.21.15.189:6833/11671] conn(0x7fa86c0557c0 0x7fa86c057c80 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:05.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.796+0000 7fa89b0b9700 1 --2- 172.21.15.189:0/77389594 >> [v2:172.21.15.47:3300/0,v1:172.21.15.47:6789/0] conn(0x7fa8940ad000 0x7fa894140610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:05.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.796+0000 7fa89b0b9700 1 --2- 172.21.15.189:0/77389594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa89414cc30 0x7fa89414d090 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:05.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.796+0000 7fa89b0b9700 1 --2- 172.21.15.189:0/77389594 >> [v2:172.21.15.47:3301/0,v1:172.21.15.47:6790/0] conn(0x7fa8940ad3e0 0x7fa8940aac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T12:23:05.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.796+0000 7fa89b0b9700 1 -- 172.21.15.189:0/77389594 >> 172.21.15.189:0/77389594 conn(0x7fa894062080 msgr2=0x7fa8940ac2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T12:23:05.804 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.796+0000 7fa89b0b9700 1 -- 172.21.15.189:0/77389594 shutdown_connections 2020-11-20T12:23:05.805 INFO:teuthology.orchestra.run.smithi189.stderr:2020-11-20T12:23:05.796+0000 7fa89b0b9700 1 -- 172.21.15.189:0/77389594 wait complete. 2020-11-20T12:23:05.805 INFO:teuthology.orchestra.run.smithi189.stderr:dumped fsmap epoch 71 2020-11-20T12:23:05.820 INFO:tasks.mds_thrash.fs.[None]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap